A task can be in one of 4 states over its lifetime:
(a) suspended
(b) enqueued
(c) running
(d) completed
This change provides priority inversion avoidance support if a task gets
escalated when it is in state (a), (c), (d).
Radar-Id: rdar://problem/76127624
thread has the task status record lock.
Today, if a thread is holding the StatusRecordLock, then no other
modification of the task status is possible - including a thread
starting to execute the task or stopping execution of the task.
However, the TaskStatusRecordLock is really about protecting the linked
list that is maintained in the ActiveTaskStatus. As such, other
operations which don't need to look at that linked list of us records
really shouldn't have to block on the StatusRecordLock.
This change allows for concurrent modification of the veTaskStatus while
the TaskStatusRecordLock is held. In particular, a task can cancelled,
escalated, start and stop running, all while another ad is holding onto
the task's StatusRecordLock. In the event of cancellation and
escalation, the task's StatusRecordLock must be n in order to propagate
cancellation and escalation to its child tasks is not needed to cancel
or escalate the task itself.
Radar-Id: rdar://problem/76127624
A slab capacity of 1000 bytes was overflowing the 1024 byte malloc bucket when adding in the slab header. Adjust it down to 984 bytes. Calculate this by subtracting the slab header size from 1024, plus a little slop for malloc stack logging, to ensure we don't overflow the bucket again.
rdar://87612288
Each trace point is declared as a function in the new `Tracing.h` header. These functions are called from the appropriate places in the concurrency runtime.
On Darwin, an implementation of these functions is provided which uses the `os/signpost.h` API to emit signpost events/intervals.
When the signpost API is not available, no-op stub implementations are provided. Implementations for other OSes can be provided by providing implementations of the trace functions for that OS.
rdar://81858487
This change has two parts to it:
1. Add in a new interface (addStatusRecordWithChecks) for adding task
status records that also takes in a function ref. This function ref will
be used to evaluate if current state of the parent task has any changes
that need to be propagated to the child task that has been created.
This is necessary to prevent the following race between task creation
and concurrent cancellation and escalation:
a. Parent task create child task. It does lazy relaxed loads on its own
state while doing so and propagates this state to the child.
b. Child task is created but has not been attached to the parent
task/task group.
c. Parent task gets cancelled by another thread.
d. Child task gets linked into the parent’s task status records but no
reevaluation has happened to account for changes that might have happened to
the parent after (a).
2. Move status record management functions from the
Runtime/Concurrency.h to TaskPrivate.h. Remove any corresponding
overrides that are no longer needed. Remove unused tryAddStatusRecord
method whose functionality is provided by addStatusRecordWithChecks.
Radar-Id: rdar://problem/86347801
The 32-bit identifier in Job is locked down at this point, so we expand the ID by storing the top 32 bits separately inside AsyncTask::PrivateStorage.
rdar://85167409
We remove the existing `swift_reflection_iterateAsyncTaskAllocations` API that attempts to provide all necessary information about a tasks's allocations starting from the task. Instead, we split it into two pieces: `swift_reflection_asyncTaskSlabPointer` to get the first slab for a task, and `+swift_reflection_asyncTaskSlabAllocations` to get the allocations in a slab, and a pointer to the next slab.
We also add a dummy metadata pointer to the beginning of each slab. This allows tools to identify slab allocations on the heap without needing to locate every single async task object. They can then use `swift_reflection_asyncTaskSlabAllocations` on such allocations to find out about the contents.
rdar://82549631
We remove the existing `swift_reflection_iterateAsyncTaskAllocations` API that attempts to provide all necessary information about a tasks's allocations starting from the task. Instead, we split it into two pieces: `swift_reflection_asyncTaskSlabPointer` to get the first slab for a task, and `+swift_reflection_asyncTaskSlabAllocations` to get the allocations in a slab, and a pointer to the next slab.
We also add a dummy metadata pointer to the beginning of each slab. This allows tools to identify slab allocations on the heap without needing to locate every single async task object. They can then use `swift_reflection_asyncTaskSlabAllocations` on such allocations to find out about the contents.
rdar://82549631
The goal here is not to eventually implement a concurrent thread
pool ourselves. We're just making it easier for integrators who
have their own pool and don't want to use Dispatch to build the
Swift concurrency runtime. Just hook the right functions and
you should be fine.
The necessary functions to hook are:
- swift_task_enqueueGlobal
- swift_task_enqueueGlobalAfterDelay
The following functions *would* be necessary to hook:
- swift_task_enqueueMainExecutor
- swift_task_asyncMainDrainQueue (only if you have an async main?)
However, this configuration does not currently properly support
the main executor, and so `@MainActor` should be avoided for now.
rdar://83513751
The goal here is not to eventually implement a concurrent thread
pool ourselves. We're just making it easier for integrators who
have their own pool and don't want to use Dispatch to build the
Swift concurrency runtime. Just hook the right functions and
you should be fine.
The necessary functions to hook are:
- swift_task_enqueueGlobal
- swift_task_enqueueGlobalAfterDelay
The following functions *would* be necessary to hook:
- swift_task_enqueueMainExecutor
- swift_task_asyncMainDrainQueue (only if you have an async main?)
However, this configuration does not currently properly support
the main executor, and so `@MainActor` should be avoided for now.
rdar://83513751
If we didn't do this (and we didn't), the tasks get released as we
perform the next() impl, and move the value from the ready task to the
waiting task. Then, the ready task gets destroyed.
But as the task group exists, it performs a cancelAll() and that
iterates over all records. Those records were not removed previously
(!!!) which meant we were pointing at now deallocated tasks.
Previously this worked because we didn't deallocate the tasks, so they
leaked, but we didn't crash. With the memory leak fixed, this began to
crash since we'd attempt to cancel already destroyed tasks.
Solution:
- Remove task records whenever they complete a waiting task.
- This can ONLY be done by the "group owning task" itself, becuause
the contract of ONLY this task being allowed to modify records. o
It MUST NOT be done by the completing tasks as they complete, as it
would race with the owning task modifying this linked list of child
tasks in the group record.
The proper handling of task group child tasks is that:
- if it completes a waiting task immediately, we don't need to retain it
- we just move the value to the waiting task and can destroy the task
- if we need to store the ready task and wait for a waiting task (for a
task that hits `await group.next()`) then we need to retain the ready
task.
- as the waiting task arrives, we move the value from the ready task
to the waiting task, and swift_release the ready task -- it will now
be destroyed safely.
Darwin OSes support vouchers, which are key/value sets that can be adopted on a thread to influence its execution, or sent to another process. APIs like Dispatch propagate vouchers to worker threads when running async code. This change makes Swift Concurrency do the same.
The change consists of a few different parts:
1. A set of shims (in VoucherShims.h) which provides declarations for the necessary calls when they're not available from the SDK, and stub implementations for non-Darwin platforms.
2. One of Job's reserved fields is now used to store the voucher associated with a job.
3. Jobs grab the current thread's voucher when they're created.
4. A VoucherManager class manages adoption of vouchers when running a Job, and replacing vouchers in suspended tasks.
5. A VoucherManager instance is maintained in ExecutionTrackingInfo, and is updated as necessary throughout a Job/Task's lifecycle.
rdar://76080222
`SWIFT_STDLIB_SINGLE_THREADED_RUNTIME` mode has been broken for a long time.
This patch guards some includes and use of libdispatch headers so that platforms
that doesn't support libdispatch can build cooperative executor runtime.
And fixed missing implementations for cooperative mode.
This macro takes the string and parameters directly, and is conditionally defined to either call fprintf or ignore its arguments. This makes the call sites a little more pleasant (no #if scattered about) and ensures every log includes the thread ID and a newline automatically.
Due to malloc quanta rounding, 1024-byte async task allocation slabs
actually end up allocating 1536 bytes on Darwin. Instead, use 1000-byte
slabs.
Fixes rdar://81181856.
Due to a missing `~` when trying to mask in a new priority + the
`IsEscalated` flag, we were instead getting an incorrect priority as
well as dropping other useful bits. This led to assertions about the
running state of a task not being set.
The implemented semantics are that:
1. Tasks have separate exclusivity access sets.
2. Any synchronous context that creates tasks will have its exclusive access set
merged into the Tasks while the Task is running.
rdar://80492364
Change the code generation patterns for `async let` bindings to use an ABI based on the following
functions:
- `swift_asyncLet_begin`, which starts an `async let` child task, but which additionally
now associates the `async let` with a caller-owned buffer to receive the result of the task.
This is intended to allow the task to emplace its result in caller-owned memory, allowing the
child task to be deallocated after completion without invalidating the result buffer.
- `swift_asyncLet_get[_throwing]`, which replaces `swift_asyncLet_wait[_throwing]`. Instead of
returning a copy of the value, this entry point concerns itself with populating the local buffer.
If the buffer hasn't been populated, then it awaits completion of the task and emplaces the
result in the buffer; otherwise, it simply returns. The caller can then read the result out of
its owned memory. These entry points are intended to be used before every read from the
`async let` binding, after which point the local buffer is guaranteed to contain an initialized
value.
- `swift_asyncLet_finish`, which replaces `swift_asyncLet_end`. Unlike `_end`, this variant
is async and will suspend the parent task after cancelling the child to ensure it finishes
before cleaning up. The local buffer will also be deinitialized if necessary. This is intended
to be used on exit from an `async let` scope, to handle cleaning up the local buffer if necessary
as well as cancelling, awaiting, and deallocating the child task.
- `swift_asyncLet_consume[_throwing]`, which combines `get` and `finish`. This will await completion
of the task, leaving the result value in the result buffer (or propagating the error, if it
throws), while destroying and deallocating the child task. This is intended as an optimization
for reading `async let` variables that are read exactly once by their parent task.
To avoid an epoch break with existing swiftinterfaces and ABI clients, the old builtins and entry
points are kept intact for now, but SILGen now only generates code using the new interface.
This new interface fixes several issues with the old async let codegen, including use-after-free
crashes if the `async let` was never awaited, and the inability to read from an `async let` variable
more than once.
rdar://77855176
Tracking this as a single bit is actually largely uninteresting
to the runtime. To handle priority escalation properly, we really
need to track this at a finer grain of detail: recording that the
task is running on a specific thread, enqueued on a specific actor,
or so on. But starting by tracking a single bit is important for
two reasons:
- First, it's more realistic about the performance overheads of
tasks: we're going to be doing this tracking eventually, and
the cost of that tracking will be dominated by the atomic
access, so doing that access now sets the baseline about right.
- Second, it ensures that we've actually got runtime involvement
in all the right places to do this tracking.
A propos of the latter: there was no runtime involvement with
awaiting a continuation, which is a point at which the task
potentially transitions from running to suspended. We must do
the tracking as part of this transition, rather than recognizing
in the run-loops that a task is still active and treating it as
having suspended, because the latter point potentially races with
the resumption of the task. To do this, I've had to introduce
a runtime function, swift_continuation_await, to do this awaiting
rather than inlining the atomic operation on the continuation.
As part of doing this work, I've also fixed a bug where we failed
to load-acquire in swift_task_escalate before walking the task
status records to invoke escalation actions.
I've also fixed several places where the handling of task statuses
may have accidentally allowed the task to revert to uncancelled.
introduce new options parameter to all task spawning
[Concurrency] ABI for asynclet start to accept options
[Concurrency] fix unittest usages of changed task creation ABI
[Concurrency] introduce constants for parameter indexes in ownership
[Concurrency] fix test/SILOptimizer/closure_lifetime_fixup_concurrency.swift
I added Builtin.buildMainActorExecutor before, but because I never
implemented it correctly in IRGen, it's not okay to use it on old
versions, so I had to introduce a new feature only for it.
The shim dispatch queue class in the Concurrency runtime is rather
awful, but I couldn't think of a reasonable alternative without
just entirely hard-coding the witness table in the runtime.
It's not ABI, at least.
Changes the task, taskGroup, asyncLet wait funtion call ABIs.
To reduce code size pass the context parameters and resumption function
as arguments to the wait function.
This means that the suspend point does not need to store parent context
and resumption to the suspend point's context.
```
void swift_task_future_wait_throwing(
OpaqueValue * result,
SWIFT_ASYNC_CONTEXT AsyncContext *callerContext,
AsyncTask *task,
ThrowingTaskFutureWaitContinuationFunction *resume,
AsyncContext *callContext);
```
The runtime passes the caller context to the resume entry point saving
the load of the parent context in the resumption function.
This patch adds a `Metadata *` field to `GroupImpl`. The await entry
pointer no longer pass the metadata pointer and there is a path through
the runtime where the task future is no longer available.
`pthread_self` is not portable to all platforms. Introduce a
`_swift_get_current_thread_id` to abstract over accessing the current
thread ID. On Windows, the thread ID and thread handle are two separate
entities, unlike POSIX threads which treats them the same.
This commit changes JobFlags storage to be 32bits, but leaves the runtime
API expressed in terms of size_t. This allows us to pack an Id in the
32bits we freed up.
The offset of this Id in the AsyncTask is an ABI constant. This way
introspection tools can extract the currently running task identifier
without any need for special APIs.
- Introduce an UnownedSerialExecutor type into the concurrency library.
- Create a SerialExecutor protocol which allows an executor type to
change how it executes jobs.
- Add an unownedExecutor requirement to the Actor protocol.
- Change the ABI for ExecutorRef so that it stores a SerialExecutor
witness table pointer in the implementation field. This effectively
makes ExecutorRef an `unowned(unsafe) SerialExecutor`, except that
default actors are represented without a witness table pointer (just
a bit-pattern).
- Synthesize the unownedExecutor method for default actors (i.e. actors
that don't provide an unownedExecutor property).
- Make synthesized unownedExecutor properties `final`, and give them
a semantics attribute specifying that they're for default actors.
- Split `Builtin.buildSerialExecutorRef` into a few more precise
builtins. We're not using the main-actor one yet, though.
Pitch thread:
https://forums.swift.org/t/support-custom-executors-in-swift-concurrency/44425
Also, do this for the initial slab for the task's allocator itself.
This avoids memory allocations for async-lets.
In case the async-task's memory demand does not exceed the initial slab size, it is now completely malloc-free.
The refcount bits of an async-let task are initialized to "immortal" so that ARC operations don't have an effect on the task.
The closure does not escape the startAsyncLet - endAsyncLet scope. Even though it's (potentially) running on a different thread.
The substantial change in the runtime is to not call swift_release on the closure context if it's a non-escaping closure.
Previously, if this happened, we simply left the actor in a running
state, causing any further jobs submitted to it to never be executed.
I can only speculate why this wasn't showing up in testing.
Also, change swift_job_run so that it prevents switching if the executor
passed in is not generic. This is an entrypoint for arbitrary executors
and generally should not allow unexpected switching (if someday custom
executors participate in that scheme). This infrastructure will also
be useful for implementing the `async let` semantics of running
synchronously until the task reaches a suspension point.
Finally, improve the #if'ed logging code throughout the task/actor runtime.
Fill out the metadata for Job to have a Dispatch-compatible vtable. When available, use the dispatch_enqueue_onto_queue_4Swift to enqueue Jobs directly onto queues. Otherwise, keep using dispatch_async_f as we have been.
rdar://75227953
Throwing functions pass the error result in `swiftself` to the resume
partial function.
Therefore, `() async -> ()` to `() async throws -> ()` is not ABI compatible.
TODO: go through remaining failing IRGen async tests and replace the
illegal convert_functions.
Most of the async runtime functions have been changed to not
expect the task and executor to be passed in. When knowing the
task and executor is necessary, there are runtime functions
available to recover them.
The biggest change I had to make to a runtime function signature
was to swift_task_switch, which has been altered to expect to be
passed the context and resumption function instead of requiring
the caller to park the task. This has the pleasant consequence
of allowing the implementation to very quickly turn around when
it recognizes that the current executor is satisfactory. It does
mean that on arm64e we have to sign the continuation function
pointer as an argument and then potentially resign it when
assigning into the task's resume slot.
rdar://70546948
