Instead of substituting the AST type, substitute the SIL type. This preserves the calling convention.
E.g. if a function has an indirect @out T result, the substituted function must also have an indirect result.
The substituted AST type would just have a direct empty-tuple result.
Fixes a miscompile
rdar://72386504
Previously, the WitnessMetadata values that were collected were just
ignored when making a function call, although space was reserved for
them in the async context. Here, that error is corrected by actually
storing them into the async context if they are present.
Previously, swift_suspend_dispatch was passed a pointer to a function
and createAsyncDispatchFn was passed a FunctionPointer. The latter
constructed a new FunctionPointer using the passed-in function pointer
as the value, because the value inside the FunctionPointer was a value
in a different function.
That worked fine on platforms without pointer authentication. On
arm64e, however, calling a function can use two values from a
FunctionPointer: the pointer to the function and the discriminator. The
result was that on arm64e, swift_suspend_dispatch failed verification
because the discriminator value that was used in the call made by
swift_suspend_dispatch did not originate in that function.
Here, that problem is resolved by passing the discriminator to
swift_suspend_dispatch. Now, createAsyncDispatchFn creates a
FunctionPointer using not just the passed-in function pointer but also
the passed-in discriminator. The result is that swift_suspend_dispatch
no longer fails verification.
When emitting Builtin.createAsyncTask[Future], the function passed to
the builtin is passed along to swift_task_create[_f]. The latter is
expecting a AsyncFunctionType<void()>, which is an alias for
void (AsyncTask *, ExecutorRef, AsyncContext *)
Previously, no ptrauth logic was emitted when emitting the builtins, so
the function pointer was not signed in the way expected by
swift_task_create[_f]. The result was a ptrauth failure on arm64e.
Here, that problem is fixed by resigning the function pointer in the way
the runtime expects.
In irgen::getAsyncFunctionAndSize, in the case where the passed-in
function has a thin representation, the pointer that is passed in is not
actually a pointer to a function but instead a pointer to an
AsyncFunctionPointer struct. To obtain the function pointer and size,
fields must be loaded from this struct. However, in order to load these
fields, that pointer to the AsyncFunctionPointer struct must itself
first be authenticated. Here, that authentication is performed.
Subsequently, the function pointer is again signed.
In FunctionPointer::getPointer, in the case that the FunctionPointer is
actually a pointer to an AsyncFunctionPointer struct, the relative
address of the function must be loaded from the AsyncFunctionPointer
struct. Doing so requires authentication that pointer to the
AsyncFunctionPointer struct. Here, that authentication is done.
In irgen::getAsyncFunctionAndSize, in the case where the passed-in
function has a thick representation but is dynamically thin, the pointer
that is passed in is not actually a pointer to a function but instead a
pointer to an AsyncFunctionPointer struct. To obtain the function
pointer and size, fields must be loaded from this struct. However, in
order to load these fields, that pointer to the AsyncFunctionPointer
struct must itself first be authenticated. Here, that authentication is
performed.
Previously, an empty PtrAuthInfo was used to construct a FunctionPointer
in AsyncCallEmission::getCalleeFunctionPointer. Here that is corrected
to use the auth info that was attached to the callee function pointer
(which is actually a pointer to an AsyncFunctionPointer struct).
Now that AsyncFunctionPointers are destructured in
getAsyncFunctionAndSize--rather than just calling through to
FunctionPointer::getPointer--in order to avoid authing the same
pointer-to-AsyncFunctionPointer twice in a row, that function must now
also handle the case--previously handled within
FunctionPointer::getPointer--where a FunctionPointer whose Kind is
AsyncFunctionPointer but which has isFunctionPointerWithoutContext set
to true. Here, that consideration is done.
In irgen::getAsyncFunctionAndSize, an AsyncFunctionPointer is
destructured into one or both of (1) the underlying function pointer and
(2) the async context size.
Previously, the underlying function pointer was obtained via calling
FunctionPointer::getPointer, which function did the work of casting the
function pointer to an AsyncFunctionPointer, loading the relative
function address from it, and using that relative address to obtain the
function pointer. The size, then, if it was also obtained, was obtained
by again casting the function pointer to an AsyncFunctionPointer and
loading a field from it.
To avoid this repetition, here, the relevant portion of the body of
FunctionPointer::getPointer is inlined. Now only a single cast to
AsyncFunctionPointerPtrTy is required and one or both fields can be
loaded from it.
The real benefit of this change is as follows: when these function
pointers are authenticated as is required to dereference it on arm64e,
the authentication will only need to be performed once rather than once
for the function pointer and once for the size.
For this, store those 3 values on the stack at function entry and update them with the return values of coro_suspend_async intrinsic calls.
This fixes a correctness issue, because the executor may be different after a resume.
It also is more efficient, because this means that the 3 values don't have to preserved in the context over a suspension point.
In derivatives of loops, no longer allocate boxes for indirect case payloads. Instead, use a custom pullback context in the runtime which contains a bump-pointer allocator.
When a function contains a differentiated loop, the closure context is a `Builtin.NativeObject`, which contains a `swift::AutoDiffLinearMapContext` and a tail-allocated top-level linear map struct (which represents the linear map struct that was previously directly partial-applied into the pullback). In branching trace enums, the payloads of previously indirect cases will be allocated by `swift::AutoDiffLinearMapContext::allocate` and stored as a `Builtin.RawPointer`.
Previously, when lowering the entry point of an async function, the
returned values were lowered to explosions that matched those of sync
functions, namely native explosions. That is incorrect for async
functions where the structured values are within the async context.
Here, that error is fixed, by loading the values returned from the call
out of the async context passed in.
rdar://problem/71641793
Thick async functions store their async context size in the closure
context. Only if the closure context is nil can we assume the
partial_apply_forwarder function to be the address of an async function
pointer struct value.
Previously, the thick context was passed as a fourth parameter to
partial apply forwarders. Here, the thick context is instead moved into
the async context at the local context position. To support this, the
local context is made always available.
An AsyncFunctionPointer, defined in Task.h, is a struct consisting of
two i32s: (1) the relative address of the async function and (2) the
size of the async context to be allocated when calling that function.
Here, such structs are emitted for every async SILFunction that is
emitted.
CfgTraits was reverted almost two weeks ago upstream but will presumably
come back. See: e025d09b216dc2239e1b502f4f277abb6fb4648a
The PPC MMA clang types were added nine days ago.
The stdlib is still crashing deep in LLVM:
```
swifterror value can only be loaded and stored from, or as a swifterror argument!
%swift.error** %2
%7 = bitcast %swift.error** %2 to %swift.opaque*
in function $ss7DecoderP16unkeyedContainers015UnkeyedDecodingC0_pyKFTj
```
From a lldb session, the function in question:
```
define protected swiftcc void @"$ss7DecoderP16unkeyedContainers015UnkeyedDecodingC0_pyKFTj"(%Ts24UnkeyedDecodingContainerP* noalias nocapture sret %0, %swift.opaque* noalias nocapture swiftself %1, %swift.error** noalias nocapture swifterror dereferenceable(8) %2, %swift.type* %3, i8** %4) #0 {
%6 = bitcast %Ts24UnkeyedDecodingContainerP* %0 to %swift.opaque*
%7 = bitcast %swift.error** %2 to %swift.opaque*
tail call swiftcc void @"$sSK5index6before5IndexQzAD_tFTj"(%swift.opaque* noalias nocapture sret %6, %swift.opaque* noalias nocapture %1, %swift.opaque* noalias nocapture swiftself %7, %swift.type* %3, i8** %4) #0
ret void
}
```
`Builtin.createAsyncTask` takes flags, an optional parent task, and an
async/throwing function to execute, and passes it along to the
`swift_task_create_f` entry point to create a new (potentially child)
task, returning the new task and its initial context.
Implement a new builtin, `cancelAsyncTask()`, to cancel the given
asynchronous task. This lowers down to a call into the runtime
operation `swift_task_cancel()`.
Use this builtin to implement Task.Handle.cancel().
