See rdar://79378762, SR-14802, SR-14841, SR-14875.
This doesn't resolve all hangs, such as those occurring
due to explicit usage of priorities and certain other
situations where priorities seem to be causing issues
(rdar://79823345), but it does resolve some cases.
The original async ABI made callees deallocate the context,
which allows tail calls (at the async-function level) but
interferes with callers' ability to optimize callee frame
allocation. The purpose of this bit was to allow callers
to do that optimization, but we've since just made callers
responsible for deallocating the context, which is overall
just a lot simpler. So this has been dead for quite some
time.
Tracking this as a single bit is actually largely uninteresting
to the runtime. To handle priority escalation properly, we really
need to track this at a finer grain of detail: recording that the
task is running on a specific thread, enqueued on a specific actor,
or so on. But starting by tracking a single bit is important for
two reasons:
- First, it's more realistic about the performance overheads of
tasks: we're going to be doing this tracking eventually, and
the cost of that tracking will be dominated by the atomic
access, so doing that access now sets the baseline about right.
- Second, it ensures that we've actually got runtime involvement
in all the right places to do this tracking.
A propos of the latter: there was no runtime involvement with
awaiting a continuation, which is a point at which the task
potentially transitions from running to suspended. We must do
the tracking as part of this transition, rather than recognizing
in the run-loops that a task is still active and treating it as
having suspended, because the latter point potentially races with
the resumption of the task. To do this, I've had to introduce
a runtime function, swift_continuation_await, to do this awaiting
rather than inlining the atomic operation on the continuation.
As part of doing this work, I've also fixed a bug where we failed
to load-acquire in swift_task_escalate before walking the task
status records to invoke escalation actions.
I've also fixed several places where the handling of task statuses
may have accidentally allowed the task to revert to uncancelled.
The symbol is swift_async_extendedFramePointerFlags. Since the value doesn't need to be dynamically computed, we save a level of indirection by emitting a fake global variable whose address is the value we want, similar to objc_absolute_packed_isa_class_mask.
This bit is mixed in to the frame pointer address stored on the stack to signal that a frame is an async frame. The compiler can emit code that ORs in the address of this symbol to apply the appropriate flag when it doesn't know the flag statically.
rdar://80277146
* Synchronize both versions of actor_counters.swift test
* Synchronize on Job address
Make sure to synchronize on Job address (AsyncTasks are Jobs, but not
all Jobs are AsyncTasks).
* Add fprintf debug output for TSan acquire/release
* Add tsan_release edge on task creation
without this, we are getting false data races between when a task
is created and immediately scheduled on a different thread.
False positive for `Sanitizers/tsan/actor_counters.swift` test:
```
WARNING: ThreadSanitizer: data race (pid=81452)
Read of size 8 at 0x7b2000000560 by thread T5:
#0 Counter.next() <null>:2 (a.out:x86_64+0x1000047f8)
#1 (1) suspend resume partial function for worker(identity:counters:numIterations:) <null>:2 (a.out:x86_64+0x100005961)
#2 swift::runJobInEstablishedExecutorContext(swift::Job*) <null>:2 (libswift_Concurrency.dylib:x86_64+0x280ef)
Previous write of size 8 at 0x7b2000000560 by main thread:
#0 Counter.init(maxCount:) <null>:2 (a.out:x86_64+0x1000046af)
#1 Counter.__allocating_init(maxCount:) <null>:2 (a.out:x86_64+0x100004619)
#2 runTest(numCounters:numWorkers:numIterations:) <null>:2 (a.out:x86_64+0x100006d2e)
#3 swift::runJobInEstablishedExecutorContext(swift::Job*) <null>:2 (libswift_Concurrency.dylib:x86_64+0x280ef)
#4 main <null>:2 (a.out:x86_64+0x10000a175)
```
New edge with this change:
```
[4357150208] allocate task 0x7b3800000000, parent = 0x0
[4357150208] creating task 0x7b3800000000 with parent 0x0
[4357150208] tsan_release on 0x7b3800000000 <<< new release edge
[139088221442048] tsan_acquire on 0x7b3800000000
[139088221442048] trying to switch from executor 0x0 to 0x7ff85e2d9a00
[139088221442048] switch failed, task 0x7b3800000000 enqueued on executor 0x7ff85e2d9a00
[139088221442048] enqueue job 0x7b3800000000 on executor 0x7ff85e2d9a00
[139088221442048] tsan_release on 0x7b3800000000
[139088221442048] tsan_release on 0x7b3800000000
[4357150208] tsan_acquire on 0x7b3800000000
counters: 1, workers: 1, iterations: 1
[4357150208] allocate task 0x7b3c00000000, parent = 0x0
[4357150208] creating task 0x7b3c00000000 with parent 0x0
[4357150208] tsan_release on 0x7b3c00000000 <<< new release edge
[139088221442048] tsan_acquire on 0x7b3c00000000
[4357150208] task 0x7b3800000000 waiting on task 0x7b3c00000000, going to sleep
[4357150208] tsan_release on 0x7b3800000000
[4357150208] tsan_release on 0x7b3800000000
[139088221442048] getting current executor 0x0
[139088221442048] tsan_release on 0x7b3c00000000
...
```
rdar://78932849
* Add static_cast<Job *>()
* Move TSan release edge to swift_task_enqueueGlobal()
Move the TSan release edge from `swift_task_create_commonImpl()` to
`swift_task_enqueueGlobalImpl()`. Task creation itself is not an event
that needs synchronization, but rather that task creation "happens
before" execution of that task on another thread.
This edge is usually added when the task is scheduled via
`swift_task_enqueue()` (which then usually calls
`swift_task_enqueueGlobal()`). However, not all task scheduling goes
through the `swift_task_enqueue()` funnel as some places call the more
specific `swift_task_enqueueGlobal()` directly. So let's annotate this
function (duplicate edges aren't harmful) to ensure we cover all
schedule events, including newly-created tasks (our original problem
here).
rdar://78932849
Co-authored-by: Julian Lettner <julian.lettner@apple.com>
This is to workaround a bug in llvm's codegen when emitting the
callee-pop stack adjustment on a regular return from a swiftasync
function (vs. a tail call).
Without the workaround we fail to emit the callee-pop stack adjustment
leading to a mis-aligned stack on return.
```
pop {r7, pc}
add sp, #16
```
Workaround for rdar://79726989
Introduce a builtin `createAsyncTask` that maps to `swift_task_create`,
and use that for the non-group task creation operations based on the
task-creation flags. `swift_task_create` and the thin function version
`swift_task_create_f` go through the dynamically-replaceable
`swift_task_create_common`, where all of the task creation logic is
present.
While here, move copying of task locals and the initial scheduling of
the task into `swift_task_create_common`, enabling by separate flags.
The flags that are useful for task creation are a bit different from
the flags that go on a job. Create a separate flag set for task
creation and use that in the API for `swift_task_create`. For now,
have the callers do the remapping.
Collapse the `group` parameter of this API into the task options, and
have existing callers set up the options appropriately. The goal for
this function is to become the centralized entry point for all task
creation, with an extensible interface.
introduce new options parameter to all task spawning
[Concurrency] ABI for asynclet start to accept options
[Concurrency] fix unittest usages of changed task creation ABI
[Concurrency] introduce constants for parameter indexes in ownership
[Concurrency] fix test/SILOptimizer/closure_lifetime_fixup_concurrency.swift
I added Builtin.buildMainActorExecutor before, but because I never
implemented it correctly in IRGen, it's not okay to use it on old
versions, so I had to introduce a new feature only for it.
The shim dispatch queue class in the Concurrency runtime is rather
awful, but I couldn't think of a reasonable alternative without
just entirely hard-coding the witness table in the runtime.
It's not ABI, at least.
Changes the task, taskGroup, asyncLet wait funtion call ABIs.
To reduce code size pass the context parameters and resumption function
as arguments to the wait function.
This means that the suspend point does not need to store parent context
and resumption to the suspend point's context.
```
void swift_task_future_wait_throwing(
OpaqueValue * result,
SWIFT_ASYNC_CONTEXT AsyncContext *callerContext,
AsyncTask *task,
ThrowingTaskFutureWaitContinuationFunction *resume,
AsyncContext *callContext);
```
The runtime passes the caller context to the resume entry point saving
the load of the parent context in the resumption function.
This patch adds a `Metadata *` field to `GroupImpl`. The await entry
pointer no longer pass the metadata pointer and there is a path through
the runtime where the task future is no longer available.
`pthread_self` is not portable to all platforms. Introduce a
`_swift_get_current_thread_id` to abstract over accessing the current
thread ID. On Windows, the thread ID and thread handle are two separate
entities, unlike POSIX threads which treats them the same.
This commit changes JobFlags storage to be 32bits, but leaves the runtime
API expressed in terms of size_t. This allows us to pack an Id in the
32bits we freed up.
The offset of this Id in the AsyncTask is an ABI constant. This way
introspection tools can extract the currently running task identifier
without any need for special APIs.
* [Concurrency] Reduce overhead of Task.yield and Task.sleep
Instead of creating a new task, we create a simple job that wraps a Builtin.RawUnsafeContinuation and resumes the continuation when it is executed. The job instance is allocated on the task local allocator, meaning we don't malloc anything.
* Update stdlib/public/Concurrency/Task.swift
Co-authored-by: Konrad `ktoso` Malawski <konrad.malawski@project13.pl>
Co-authored-by: Konrad `ktoso` Malawski <konrad.malawski@project13.pl>
Rename duplicated `swift::fatalError` in `swiftRuntime` and `swift_Concurrency`.
Both `swiftRuntime` and `swift_Concurrency` had `swift::fatalError` implementation, but it causes symbol conflict with the `-static-stdlib` flag.
This patch removes one of the implementations in `swift_Concurrency` to avoid conflicts. Also added a test case to ensure that linking the Concurrency module with `-static-stdlib` works.
This issue was found by SwiftWasm test cases.
- Introduce an UnownedSerialExecutor type into the concurrency library.
- Create a SerialExecutor protocol which allows an executor type to
change how it executes jobs.
- Add an unownedExecutor requirement to the Actor protocol.
- Change the ABI for ExecutorRef so that it stores a SerialExecutor
witness table pointer in the implementation field. This effectively
makes ExecutorRef an `unowned(unsafe) SerialExecutor`, except that
default actors are represented without a witness table pointer (just
a bit-pattern).
- Synthesize the unownedExecutor method for default actors (i.e. actors
that don't provide an unownedExecutor property).
- Make synthesized unownedExecutor properties `final`, and give them
a semantics attribute specifying that they're for default actors.
- Split `Builtin.buildSerialExecutorRef` into a few more precise
builtins. We're not using the main-actor one yet, though.
Pitch thread:
https://forums.swift.org/t/support-custom-executors-in-swift-concurrency/44425
Both swiftRuntime and swift_Concurrency had swift::fatalError
implementation, but it causes symbol conflict when -static-stdlib.
This patch renames one of the impl in swift_Concurrency to avoid
conflict.
Also, do this for the initial slab for the task's allocator itself.
This avoids memory allocations for async-lets.
In case the async-task's memory demand does not exceed the initial slab size, it is now completely malloc-free.
The refcount bits of an async-let task are initialized to "immortal" so that ARC operations don't have an effect on the task.
The closure does not escape the startAsyncLet - endAsyncLet scope. Even though it's (potentially) running on a different thread.
The substantial change in the runtime is to not call swift_release on the closure context if it's a non-escaping closure.
