Both return the pack immediately if its already heap allocated, by
checking the least significant bit of the pack pointer.
Then,
- swift_allocateMetadataPack() uniques the metadata pack by the
pointer equality of elements.
- swift_allocateWitnessTablePack() does not unique the pack.
Both return a pack pointer with the least significant bit set,
indicating heap allocation.
rdar://105837040
* WIP: Store layout string in type metadata
* WIP: More cases working
* WIP: Layout strings almost working
* Add layout string pointer to struct metadata
* Fetch bytecode layout strings from metadata in runtime
* More efficient bytecode layout
* Add support for interpreted generics in layout strings
* Layout string instantiation, take and more
* Remove duplicate information from layout strings
* Include size of previous object in next objects offset to reduce number of increments at runtime
* Add support for existentials
* Build type layout strings with StructBuilder to support target sizes and metadata pointers
* Add support for resilient types
* Properly cache layout strings in compiler
* Generic resilient types working
* Non-generic resilient types working
* Instantiate resilient type in layout when possible
* Fix a few issues around alignment and signing
* Disable generics, fix static alignment
* Fix MultiPayloadEnum size when no extra tag is necessary
* Fixes after rebase
* Cleanup
* Fix most tests
* Fix objcImplementattion and non-Darwin builds
* Fix BytecodeLayouts on non-Darwin
* Fix Linux build
* Fix sizes in linux tests
* Sign layout string pointers
* Use nullptr instead of debug value
Replace the `assert(false)` with `swift_unreachable`.
This is more robust in release builds where the `assert(false)` is a no-op and would result in a wrong fall-though to the next switch case.
Fixes a warning in release builds.
@differentiable function is actually a triple (function, jvp, vjp). Previously normal thick function value witness table was used. As a result, for example, only function was copied, but none of differential components.
This was the cause of uninitialized memory accesses and subsequent segfaults.
Should fix now unavailable TF-1122
Swift vtables can contain NULL entries. This happens when the compiler is able to prove that a method is never called.
When ptrauth is enabled, those NULL entries are still signed. However, there's an inconsistency in the runtime. Most places where we copy a vtable with swift_ptrauth_copy_code_or_data, we accept an unsigned NULL value. But when fixing up a dynamic ObjC subclass with swift_objc_classCopyFixupHandler, we don't.
Since the compiler seems to always sign these NULL values, this still works, except in the case where a signed NULL value coincidentally has a ptrauth signature of all zeroes. When that happens, the code to accept an unsigned NULL value sees this NULL with an all-zero signature as an unsigned NULL, and copies it verbatim. Then we really do have an unsigned NULL in the destination.
If that class is then dynamically subclassed, we'll call swift_objc_classCopyFixupHandler which does not accept unsigned NULL values, and fail ptrauth.
Adjust swift_objc_classCopyFixupHandler to accept unsigned NULL like the rest. These entries should never be called, and any attempt to call them will crash regardless, so it makes little difference.
rdar://92800322
We shouldn't include <windows.h> implicitly from .cpp files, but should
do it directly so that we know it's there.
Also, if we're including <windows.h>, do it at the top of the file.
rdar://90776105
Moved all the threading code to one place. Added explicit support for
Darwin, Linux, Pthreads, C11 threads and Win32 threads, including new
implementations of Once for Linux, Pthreads, C11 and Win32.
rdar://90776105
We shouldn't include <windows.h> implicitly from .cpp files, but should
do it directly so that we know it's there.
Also, if we're including <windows.h>, do it at the top of the file.
rdar://90776105
Moved all the threading code to one place. Added explicit support for
Darwin, Linux, Pthreads, C11 threads and Win32 threads, including new
implementations of Once for Linux, Pthreads, C11 and Win32.
rdar://90776105
Tidy up the metadata definitions.
* Generalize a number of metadata kinds for out-of-process clients
* Introduce conveniences to make runtime lookups easier
* Introduce TargetExistentialTypeExpression to TrailingObjects stops complaining about OverloadTokens being ambiguous
Note that there is no impact on the layout of the metadata - the changes here are all ABI-compatible.
I wrote out this whole analysis of why different existential types
might have the same logical content, and then I turned around and
immediately uniqued existential shapes purely by logical content
rather than the (generalized) formal type. Oh well. At least it's
not too late to make ABI changes like this.
We now store a reference to a mangling of the generalized formal
type directly in the shape. This type alone is sufficient to unique
the shape:
- By the nature of the generalization algorithm, every type parameter
in the generalization signature should be mentioned in the
generalized formal type in a deterministic order.
- By the nature of the generalization algorithm, every other
requirement in the generalization signature should be implied
by the positions in which generalization type parameters appear
(e.g. because the formal type is C<T> & P, where C constrains
its type parameter for well-formedness).
- The requirement signature and type expression are extracted from
the existential type.
As a result, we no longer rely on computing a unique hash at
compile time.
Storing this separately from the requirement signature potentially
allows runtimes with general shape support to work with future
extensions to existential types even if they cannot demangle the
generalized formal type.
Storing the generalized formal type also allows us to easily and
reliably extract the formal type of the existential. Otherwise,
it's quite a heroic endeavor to match requirements back up with
primary associated types. Doing so would also only allows us to
extract *some* matching formal type, not necessarily the *right*
formal type. So there's some good synergy here.
Apply a blanket pass of including `new` for the placement new allocation
and namespacing the call to the global placement new allocator. This
should repair the Android ARMv7 builds.
Not all targets have a 16-byte type alignment guarantee. For the types
which are not naturally aligned, provide a type specific `operator new`
overload to ensure that we are properly aligning the type on allocation
as we run the risk of under-aligned allocations otherwise.
This should no longer be needed with C++17 and newer which do a two
phase `operator new` lookup preferring
`operator new(std::size, std::align_val_t)` if needed. The base type
would be fully pre-processed away. The empty base class optimization
should help ensure that we do not pay any extra size costs for the
alignment fixes.
As we are a C++14 codebase, we must locally implement some of the
standard type_traits utilities, namely `void_t`. We take the minimal
definition here, assuming that the compiler is up-to-date with C++14 DR
reports which fixed an issue in SFINAE. We use the SFINAE for detecting
the presence of the `operator new` overload to guide the over-alignment,
which is inherited through the new `swift::overaligned_type<>` base
type.
Annotate the known classes which request explicit alignment which is
non-pointer alignment. This list was identified by
`git grep ' alignas(.*) '`.
When SWIFT_COMPACT_ABSOLUTE_FUNCTION_POINTER is enabled, relative direct
pointers whose pointees are functions will be turned into absolute
pointer at compile-time.
The immediate use case is only concretely-constrained existential
types, which could use a much simpler representation, but I've
future-proofed the representation as much as I can; thus, the
requirement signature can have arbitrary parameters and
requirements, and the type can have an arbitrary type as the
sub-expression. The latter is also necessary for existential
metatypes.
The chief implementation complexity here is that we must be able
to agree on the identity of an existential type that might be
produced by substitution. Thus, for example, `any P<T>` when
`T == Int` must resolve to the same type metadata as
`any P<Int>`. To handle this, we identify the "shape" of the
existential type, consisting of those parts which cannot possibly
be the result of substitution, and then abstract the substitutable
"holes" as an application of a generalization signature. That
algorithm will come in a later patch; this patch just represents
it.
Uniquing existential shapes from the requirements would be quite
complex because of all the symbolic mangled names they use.
This is particularly true because it's not reasonable to require
translation units to agree about what portions they mangle vs.
reference symbolically. Instead, we expect the compiler to do
a cryptographic hash of a mangling of the shape, then use that
as the unique key identifying the shape.
This is just the core representation and runtime interface; other
parts of the runtime, such as dynamic casting and demangling
support, will come later.
The `AllocationPool` type is reflected upon and inspected by debugging
tools. This requires that the layout of the atomically wrapped type
must store the value at offset 0. However, the `std::atomic` does not
make any such guarantees, and a layout differing would render debug
utilities useless. Switch instead from a `std::atomic` to
`swift::atomic`. In order to do so, we also need to introduce the
`compare_exchange_strong` helper which maps to
`compare_exchange_strong_explicit` mirroring `compare_exchange_weak`.
* When ptrauth-copying vtable/wtables, allow NULL entries (due to VFE)
* Mark virtual-function-elimination-generics-exec.swift UNSUPPORTED: arm64e until the rebranch
* Fix test expectations
The current system is based on MetadataCompletionQueueEntry
objects which are allocated and then enqueued on dependencies.
Blocking is achieved using a condition variable associated
with the lock on the appropriate metadata cache. Condition
variables are inherently susceptible to priority inversions
because the waiting threads have no dynamic knowledge of
which thread will notify the condition. In the current system,
threads that unblock dependencies synchronously advance their
dependent metadata completions, which means the signaling
thread is unreliable even if we could represent it in condition
variables. As a result, the current system is wholly unsuited
for eliminating these priority inversions.
An AtomicWaitQueue is an object containing a lock. The queue
is eagerly allocated, and the lock is held, whenever a thread
is doing work that other threads might wish to block on. In
the metadata completion system, this means whenever we construct
a metadata cache entry and the metadata isn't already allocated
and transitively complete after said construction. Blocking
is done by safely acquiring a shared reference to the queue
object (which, in the current implementation, requires briefly
taking a lock that's global to the surrounding metadata cache)
and then acquiring the contained lock. For typical lock
implementations, this avoids priority inversions by temporarily
propagating the priority of waiting threads to the locking
threads.
Dependencies are unblocked by simply releasing the lock held
in the queue. The unblocking thread doesn't know exactly what
metadata are blocked on it and doesn't make any effort to
directly advance their completion; instead, the blocking
thread will wake up and then attempt to advance the dependent
metadata completion itself, eliminating a source of priority
overhang that affected the old system. Successive rounds of
unblocking (e.g. when a metadata makes partial progress but
isn't yet complete) can be achieved by creating a new queue
and unlocking the old one. We can still record dependencies
and use them to dynamically diagnose metadata cycles.
The new system allocates more eagerly than the old one.
Formerly, metadata completions which were never blocked never
needed to allocate a MetadataCompletionQueueEntry; we were
then unable to actually deallocate those entries once they
were allocated. The new system will allocate a queue for
most metadata completions, although, on the positive side,
we can reliably deallocate these queues. Cache entries are
also now slightly smaller because some of the excess storage
for status has been folded into the queue.
The fast path of an actual read of the metadata remains a
simple load-acquire. Slow paths may require a bit more
locking. On Darwin, the metadata cache lock can now use
os_unfair_lock instead of pthread_mutex_t (which is a massive
improvement) because it does not need to support associated
condition variables.
The excess locking could be eliminated with some sort of
generational scheme. Sadly, those are not portable, and I
didn't want to take it on up-front.
rdar://76127798
MetadataAllocator should never return NULL, but bugs or corruption could potentially make that happen. On the large path, switch from malloc to swift_slowAlloc, which aborts on failure. On the pool path, check for a NULL allocation pointer, and log a bunch of information about the allocation request and the allocator's current state.
rdar://84503396
Mangling can fail, usually because the Node structure has been built
incorrectly or because something isn't supported with the old remangler.
We shouldn't just terminate the program when that happens, particularly
if it happens because someone has passed bad data to the demangler.
rdar://79725187
If anyone else is building Windows ARM64 they should be using a new
enough Visual Studio. This workaround is more difficult to keep working
properly and the CI hosts should have a new enough Visual Studio
installation hopefully in order to enable the ARM64 builds of the
runtime. If they do not, we can re-evaluate whether to re-instate the
workaround. This allows building part of the runtime with Visual Studio
2022 and reduces the maintenance overheads for the runtime.
