The current system is based on MetadataCompletionQueueEntry
objects which are allocated and then enqueued on dependencies.
Blocking is achieved using a condition variable associated
with the lock on the appropriate metadata cache. Condition
variables are inherently susceptible to priority inversions
because the waiting threads have no dynamic knowledge of
which thread will notify the condition. In the current system,
threads that unblock dependencies synchronously advance their
dependent metadata completions, which means the signaling
thread is unreliable even if we could represent it in condition
variables. As a result, the current system is wholly unsuited
for eliminating these priority inversions.
An AtomicWaitQueue is an object containing a lock. The queue
is eagerly allocated, and the lock is held, whenever a thread
is doing work that other threads might wish to block on. In
the metadata completion system, this means whenever we construct
a metadata cache entry and the metadata isn't already allocated
and transitively complete after said construction. Blocking
is done by safely acquiring a shared reference to the queue
object (which, in the current implementation, requires briefly
taking a lock that's global to the surrounding metadata cache)
and then acquiring the contained lock. For typical lock
implementations, this avoids priority inversions by temporarily
propagating the priority of waiting threads to the locking
threads.
Dependencies are unblocked by simply releasing the lock held
in the queue. The unblocking thread doesn't know exactly what
metadata are blocked on it and doesn't make any effort to
directly advance their completion; instead, the blocking
thread will wake up and then attempt to advance the dependent
metadata completion itself, eliminating a source of priority
overhang that affected the old system. Successive rounds of
unblocking (e.g. when a metadata makes partial progress but
isn't yet complete) can be achieved by creating a new queue
and unlocking the old one. We can still record dependencies
and use them to dynamically diagnose metadata cycles.
The new system allocates more eagerly than the old one.
Formerly, metadata completions which were never blocked never
needed to allocate a MetadataCompletionQueueEntry; we were
then unable to actually deallocate those entries once they
were allocated. The new system will allocate a queue for
most metadata completions, although, on the positive side,
we can reliably deallocate these queues. Cache entries are
also now slightly smaller because some of the excess storage
for status has been folded into the queue.
The fast path of an actual read of the metadata remains a
simple load-acquire. Slow paths may require a bit more
locking. On Darwin, the metadata cache lock can now use
os_unfair_lock instead of pthread_mutex_t (which is a massive
improvement) because it does not need to support associated
condition variables.
The excess locking could be eliminated with some sort of
generational scheme. Sadly, those are not portable, and I
didn't want to take it on up-front.
rdar://76127798
os_unfair_lock is much smaller than pthread_mutex_t (4 bytes versus 64) and a bit faster.
However, it doesn't support condition variables. Most of our uses of Mutex don't use condition variables, but a few do. Introduce ConditionMutex and StaticConditionMutex, which allow condition variables and continue to use pthread_mutex_t.
On all other platforms, we continue to use the same backing mutex type for both Mutex and ConditionMutex.
rdar://problem/45412121
* [Runtime] Switch MetadataCache to ConcurrentReadableHashMap.
Use StableAddressConcurrentReadableHashMap. MetadataCacheEntry's methods for awaiting a particular state assume a stable address, where it will repeatedly examine `this` in a loop while waiting on a condition variable, so we give it a stable address to accommodate that. Some of these caches may be able to tolerate unstable addresses if this code is changed to perform the necessary table lookup each time through the loop instead. Some of them store metadata inline and we assume metadata never moves, so they'll have to stay this way.
* Have StableAddressConcurrentReadableHashMap remember the last found entry and check that before doing a more expensive lookup.
* Make a SmallMutex type that stores the mutex data out of line, and use it to get LockingConcurrentMapStorage to fit into the available space on 32-bit.
rdar://problem/70220660
This gives us faster lookups and a small advantage in memory usage. Most of these maps need stable addresses for their entries, so we add a level of indirection to ConcurrentReadableHashMap for these cases to accommodate that. This costs some extra memory, but it's still a net win.
A new StableAddressConcurrentReadableHashMap type handles this indirection and adds a convenience getOrInsert to take advantage of it.
ConcurrentReadableHashMap is tweaked to avoid any global constructors or destructors when using it as a global variable.
ForeignWitnessTables does not need stable addresses and it now uses ConcurrentReadableHashMap directly.
rdar://problem/70056398
The new function swift_compareProtocolConformanceDescriptors calls
through to the preexisting code in MetadataCacheKey which has been
extracted out from MetadataCacheKey::compareWitnessTables into a new
public static function
MetadataCacheKey::compareProtocolConformanceDescriptors.
The new function's availability is "future" for now.
This removes the last reference to the `llvm::` namespace in the
standard library. All uses of the LLVMSupport library now are
namespaced into the `__swift::__runtime` namespace. This allows us to
incrementally vend the LLVMSupport library and make the separation
explicit.
This adds a new copy of LLVMSupport into the runtime. This is the final
step before changing the inline namespace for the runtime support. This
will allow us to avoid the ODR violations from the header definitions of
LLVMSupport.
LLVMSupport forked at: 22492eead218ec91d349c8c50439880fbeacf2b7
Changes made to LLVMSupport from that revision:
process.inc forward declares `_beginthreadex` due to compilation issues due to custom flag handling
API changes required that we alter the `Deallocate` routine to account
for the alignment.
This is a temporary state, meant to simplify the process. We do not use
the entire LLVMSupport library and there is no value in keeping the
entire library. Subsequent commits will prune the library to the needs
for the runtime.
This was identified by UBSAN: signed-integer-overflow. Explicitly mark
the value as unsigned to ensure that the value does not overflow. There
is a second instance of a raw literal, however, because it is a `*=` the
value is implicitly understood to be unsigned.
Rather than scanning through the generic parameters and generic requirements
each time we form a key for the generic metadata cache, compute these
values once, when the cache itself is first initialized.
Rather than scanning the type descriptor each time we perform a comparison
or hash of a metadata cache entry, do so only once to establish the number
of key parameters and the number of witness tables. Use those values to
more efficiently compare keys.
Metadata uniquing might encounter witness tables that were distinctly
generated but come from identical descriptors. Handle this case in metadata
uniquing be looking into the protocol conformance descriptors themselves.
- `swift_getForeignTypeMetadata` is now a request/response function.
- The initialization function is now a completion function, and the
pointer to it has moved into the type descriptor.
- The cache variable is no longer part of the ABI; it's an
implementation detail of the access function.
- The two points above mean that there is no special header on foreign
type metadata and therefore that they can be marked constant when
there isn't something about them that needs to be initialized.
The only foreign-metadata initialization we actually do right now is
of the superclass field of a foreign class, and since that relationship
is a proper DAG, it's not actually possible to have recursive
initialization problems. But this is the right long-term thing to do,
and it removes one of the last two clients of once-based initialization.
I was going to put this off for awhile, but it turns out that a lot of
my testcases are enums with multi-payload cases, which we currently
compile as tuples, so they were all still hanging until this patch.
I de-templated MetadataState and MetadataRequest because we weren't
relying on the template and because using the template was causing
conversion problems due to the inability to directly template an enum
in C++.
This includes global generic and non-generic global access
functions, protocol associated type access functions,
swift_getGenericMetadata, and generic type completion functions.
The main part of this change is that the functions now need to take
a MetadataRequest and return a MetadataResponse, which is capable
of expressing that the request can fail. The state of the returned
metadata is reported as an second, independent return value; this
allows the caller to easily check the possibility of failure without
having to mask it out from the returned metadata pointer, as well
as allowing it to be easily ignored.
Also, change metadata access functions to use swiftcc to ensure that
this return value is indeed returned in two separate registers.
Also, change protocol associated conformance access functions to use
swiftcc. This isn't really related, but for some reason it snuck in.
Since it's clearly the right thing to do, and since I really didn't
want to retroactively tease that back out from all the rest of the
test changes, I've left it in.
Also, change generic metadata access functions to either pass all
the generic arguments directly or pass them all indirectly. I don't
know how we ended up with the hybrid approach. I needed to change all
the code-generation and calls here anyway in order to pass the request
parameter, and I figured I might as well change the ABI to something
sensible.
I was trying to make the entry-delegation thing do *way* too much.
Just give the entry access to the lock/queue and introduce subclasses
which simplify most of the work.
Also, fix some bad reasoning around the attempts to avoid acquiring
locks in the absence of waiters. It really is always necessary to
acquire the lock when notifying; waiters cannot atomically set the
has-waiters flag and wait, so we have to protect against the
possibility that we notify before they can wait.
Change generic witness table instantiation to use a more lightweight
entry scheme that allocates the witness table as part of the entry.
In contrast, change generic metadata instantiation to use a more
straightforward allocation scheme where the metadata is a totally
independent allocation.
This is preparation for proper cyclic-dependency handling.
This seems to more than fix a performance regression that we
detected on a metadata-allocation microbenchmark.
A few months ago, I improved the metadata cache representation
and changed the metadata allocation scheme to primarily use malloc.
Previously, we'd been using malloc in the concurrent tree data
structure but a per-cache slab allocator for the metadata itself.
At the time, I was concerned about the overhead of per-cache
allocators, since many metadata patterns see only a small number
of instantiations. That's still an important factor, so in the
new scheme we're using a global allocator; but instead of using
malloc for individual allocations, we're using a slab allocator,
which should have better peak, single-thread performance, at the
cost of not easily supporting deallocation. Deallocation is
only used for metadata when there's contention on the cache, and
specifically only when there's contention for the same key, so
leaking a little isn't the worst thing in the world.
The initial slab is a 64K globally-allocated buffer.
Successive slabs are 16K and allocated with malloc.
rdar://28189496
This seems to more than fix a performance regression that we
detected on a metadata-allocation microbenchmark.
A few months ago, I improved the metadata cache representation
and changed the metadata allocation scheme to primarily use malloc.
Previously, we'd been using malloc in the concurrent tree data
structure but a per-cache slab allocator for the metadata itself.
At the time, I was concerned about the overhead of per-cache
allocators, since many metadata patterns see only a small number
of instantiations. That's still an important factor, so in the
new scheme we're using a global allocator; but instead of using
malloc for individual allocations, we're using a slab allocator,
which should have better peak, single-thread performance, at the
cost of not easily supporting deallocation. Deallocation is
only used for metadata when there's contention on the cache, and
specifically only when there's contention for the same key, so
leaking a little isn't the worst thing in the world.
The initial slab is a 64K globally-allocated buffer.
Successive slabs are 16K and allocated with malloc.
rdar://28189496
IIRC we never had any evidence that the performance impact of a
separate allocator here was actually measurable, and it does come
at a significant fragmentation cost because every single cache
allocates at least a page of memory. Sharing that with the system
allocator makes more sense, even if these allocations are typically
permanent.
This also means that standard memory-debugging tools will actually
find problems with out-of-bounds accesses to metadata.
MetadataCache's allocator into it.
The major functional change here is that MetadataCache will now use
the slab allocator for tree nodes, but I also switched the Hashable
conformances cache to use ConcurrentMap directly instead of a
Lazy<ConcurrentMap<>>.
- added read / write lock support
- added non-fatal error support to allow use of mutex in fatal error reporting pathway
- isolated pthread implementation to it own header/cpp file pair
- expanded unit tests to cover new code as well as better test existing mutex
- removed a layer of complexity that added no real value
initialization in-place on demand. Initialize parent metadata
references correctly on struct and enum metadata.
Also includes several minor improvements related to relative
pointers that I was using before deciding to simply switch the
parent reference to an absolute reference to get better access
patterns.
Includes a fix since the earlier commit to make enum metadata
writable if they have an unfilled payload size. This didn't show
up on Darwin because "constant" is currently unenforced there in
global data containing relocations.
This patch requires an associated LLDB change which is being
submitted in parallel.
