- Ensure that no callbacks are called when either of these mutexes are held:
- `SwiftASTConsumer::CancellationRequestCallbackAndIsCancelledMtx`
- `ASTBuildOperation::DependencyStampsMtx`
- Exception: We allow calls to `getBuferStamp` because that doesn’t claim any locks)
- No change was necessary here
- `ASTBuildOperation::ConsumersAndResultMtx`
- `ASTBuildOperation::CacheMtx`
- No change was necessary here
- `ASTBuildOperation::ScheduledConsumersMtx`
- Make sure we always inform the consumers asynchronously, even if the operation was already cancelled when the consumer got enqueued by introducing `ConsumerNotificationQueue`
- Ensure that all callbacks to `SwiftASTConsumer` (i.e. `requestCancellation` and `handlePrimaryAST`, `failed`, `cancelled`) are performed asyncronously from a queue to avoid deadlocks
rdar://110357502
This fixes a deadlock that could occur when a cursor info request is cancelled while another one is inside the cancellation callback. The schematic deadlock backtrace is
```
sourcekitd::cancelRequest
SourceKit::RequestTracker::cancel -> holds RequestTracker.RequestsMtx
anonymous func in SourceKit::SwiftASTManager::processASTAsync
SourceKit::SwiftASTConsumer::requestCancellation
anonymous func in ASTBuildOperation::addConsumer -> tries to get ASTBuildOperation.ConsumersAndResultMtx
resolveCursor
SourceKit::SwiftASTManager::processASTAsync
SourceKit::SwiftASTConsumer::requestCancellation
anonymous func in ASTBuildOperation::addConsumer
[probably inlined] ASTBuildOperation::requestConsumerCancellation -> holds ASTBuildOperation.ConsumersAndResultMtx
resolveCursor::CursorInfoConsumer::cancelled
anonymous func in SourceKit::SwiftLangSupport::getCursorInfo
anonymous func in handleRequestCursorInfo
reportCursorInfo
anonymous func in sourcekitd::handleRequest -> tries to get RequestTracker.RequestsMtx
```
rdar://110357502
Solver based results are fast within a function, where the `ASTContext`
can be re-used. But it is significantly slower than the AST based
results when outside of a function. Instead of using solver based as the
primary results, only use them as a fallback for when AST based fails.
Resolves rdar://108930110.
Introduce a new key `generated_buffers`, which
stores an array of generated buffers. These
include the buffer text, as well as its original
location and any parent buffers.
While here, also fix rdar://107281079 such that
only apply the filename fallback logic to the
pretty-printed Decl case. We ought to remove this
fallback once the editor can handle it though.
rdar://107281079
rdar://107952288
* Factor out ASTContext plugin loading to newly introduced 'PluginLoader'
* Insert 'DependencyTracker' to 'PluginLoader'
* Add dependencies right before loading the plugins
rdar://104938481
Make a single 'PluginRegistry' and share it between SwiftASTManager,
IDEInspectionInstance, and CompileInstance. And inject the plugin
registry to ASTContext right after 'CompilerInstance.setup()'
That way, all sema-capable ASTContext in SourceKit share a single
PluginRegistry.
This brings up the ability to compute cursor info results using the completion-like type checking paradigm, which an reuse ASTContexts and doesn’t need to type check the entire file.
For now, the new implementation only supports cursor info on `ValueDecl`s (not on references) because they were easiest to implement. More cursor info kinds are coming soon.
At the moment, we only run the new implementation in a verification mode: It is only invoked in assert toolchains and when run, we check that the results are equivalent to the old implementation. Once more cursor info kinds are implemented and if the SourceKit stress tester doesn’t find any verification issues, we can enable the new implementation, falling back to the old implementation if the new one didn’t produce any results.
`getValue` -> `value`
`getValueOr` -> `value_or`
`hasValue` -> `has_value`
`map` -> `transform`
The old API will be deprecated in the rebranch.
To avoid merge conflicts, use the new API already in the main branch.
rdar://102362022
This enables the ability to cancel requests, which aren’t code completion requests, again.
Previous crashes in SILGen are prevented by disabling cancellation during the SIL stages. Instead, we add dedicated cancellation checkpoints before and after SIL.
rdar://98390926
#58786 (rdar://93030932) was failing because the `swift-frontend` invocations passed a `swiftExecutablePath` to `Invocation.parseArgs`. This caused the `ClangImporter` instance to point to a `clang` binary next to the `swift-frontend` executable while SourceKit used PATH to find `clang`. The clang executable next to `swift-frontend` doesn’t actually exist because `clang` lives in `llvm-linux-aarch64/bin` and `swift-frontend` lives in `swift-linux-aarch64/bin`.
So some checks for a minimum clang verison failed for the normal build (because the executable doesn’t actually exists) while they pass during the SourceKit build (which used `clang` from `PATH`). This in turn caused the `outline-atomics` to be enabled to the SourceKit clang compiler arguments but not the clang compiler arguments for a normal build and thus resulted in two separate module cache directories (which includes the enabled features in the module directory hash).
To fix this issue, also set the swift executable path for compiler invocations created from SourceKit.
Fixes#58786 (rdar://93030932)
We might run into situations where `matchesSourceState` reads `DependencyStamps` while they are being written in `buildASTUnit`, causing a crash.
Guard `DependencyStamps` by a mutex to avoid this problem. We don’t need to provide any ordering guarantees about whether `DependencyStamps` have been computed in are accessed because we already assume that the dependencies shouldn't change (much) in the time between an `ASTBuildOperation` is created and until it produced an AST.
rdar://92748564
This fixes a race-conditioned deadlock which could occur while cancelling SourceKit AST build request
We have one thread that claimed `CancellationRequestCallbackMtx` in `SwiftASTConsumer::requestCancellation` and wants to claim `ConsumersAndResultMtx` in `ASTBuildOperation::requestConsumerCancellation`
Another thread claimed `ConsumersAndResultMtx` in `ASTBuildOperation::schedule` and now wants to claim `CancellationRequestCallbackMtx` in `SwiftASTConsumer::removeCancellationRequestCallback`.
In both cases we could actually release one lock before claiming the other.
Fixes rdar://90870793
We need to run SILGen for diagnostics (to actually get all diagnostics).
All non-completion requests share an AST and thus they too run SILGen.
Any lazy typechecking run in SILGen assumes that it succeeds.
Cancellation can cause typechecking to fail here though, since we simply
check the flag and error if it's set. This unfortunately has the ability
to cause any any number of crashes since various invariants in SILGen
are then broken.
Disable cancellation of in-flight non-completion requests for now until
we have a proper fix in place.
Resolves rdar://91251017.
Two paths missed setting up overlays:
- `CompletionInstance` when checking files from dependencies
- `SwiftASTManager` when reading in files that it would later replace
all inputs with
(1) would cause the AST context not to be re-used, even though nothing
had changed. (2) caused all non-completion functionality to fail for any
symbols within files only specified by the overlay.
Resolves rdar://85508213.
When opening a file for the first time, we don’t store a snapshot for it. This could cause a crash when trying to consult its snapshot to see whether an AST can be reused for cursor info.
Instead of checking that the stdlib can be loaded in a variety of places, check it when setting up the compiler instance. This required a couple more checks to avoid loading the stdlib in cases where it’s not needed.
To be able to differentiate stdlib loading failures from other setup errors, make `CompilerInstance::setup` return an error message on failure via an inout parameter. Consume that error on the call side, replacing a previous, more generic error message, adding error handling where appropriate or ignoring the error message, depending on the context.
Enqueuing `SwiftASTConsumer`s might be expensive because `getBuildOperationForConsumer` consults the file system. Since all results from the AST build are processed asynchronously anyway, there’s no need to perform the enqueuing synchronously.
rdar://86289703
We never need to have two copies of the same `FileContent` object, so we don’t need a copy constructor and can thus pass it on the stack again, instead of storing it on the heap.
We need to modify the pointer pointing to the cancellation flag when reusing an ASTContext for code completion. This is not possible by the previous design because `TypeCheckerOptions` was `const`. Moving the cancellation flag to `ASTContext` will also allow other stages of the compiler to honor a cancellation request.
Previously, `SwiftASTManager` and `SlowRequestSimulator` maintained their own list of in-progress cancellation tokens. With code completion cancellation coming up, there would need to be yet another place to track in-progress requests, so let’s centralize it.
While at it, also support cancelling requests before they are scheduled, eliminating the need for a `sleep` in a test case.
The current implementaiton leaks tiny amounts of memory if a request is cancelled after if finishes. I think this is fine because it is a pretty nieche case and the leaked memory is pretty small (a `std::map` entry pointing to a `std::function` + `bool`). Alternatively, we could require the client to always dispose of the cancellation token manually.
The key changes here are
- To keep track of cancellation tokens for all `ScheduledConsumer`s in `SwiftASTManager`
- Generate unique request handles for all incoming requests (`create_request_handle `), use these request handles as cancellation tokens and return them from the `sourcekitd_send_request` methods
- Implement cancellation with `sourcekitd_cancel_request` as the entry point and `SwiftASTManager::cancelASTConsumer` as the termination point
Everything else is just plumbing the cancellation token through the various abstraction layers.
rdar://83391505
This commit refactors the way ASTs are being built in SourceKit and how `SwiftASTConsumer`s are served by the built ASTs. `SwiftASTManager.h` should give an overview of the new design.
This commit does not change the cancellation paradigm in SourceKit (yet). That is, subsequent requests with the same `OncePerASTToken` still cancel previous requests with the same token. But while previously, we were only able to cancel requests that haven’t started an AST build yet, we can now also cancel the AST build of the to-be-cancelled requests.
With this change in place, we can start looking into explicit cancellation of requests or other cancellation paradigms.
Have SourceKit return locations for symbols outside of the current
module as well. Callsites of location and comment information should
explicitly disable retrieving serialized information where performance
is a concern.
Resolves rdar://75582627
We have seen a couple of crashes that look like they are being caused by `sourcekitd` being unable to load the Swift stdlib. Emit a proper error message explaining the issue instead of crashing.
This change assumes that all sourcekitd requests that create an AST require the stdlib. IMHO this is a reasonable assumption.
Fixes rdar://75740572 by catching the crash in the new check.
To help consolidate our various types describing imports, this commit moves the following types and methods to Import.h:
* ImplicitImports
* ImplicitStdlibKind
* ImplicitImportInfo
* ModuleDecl::ImportedModule
* ModuleDecl::OrderImportedModules (as ImportedModule::Order)
* ModuleDecl::removeDuplicateImports() (as ImportedModule::removeDuplicates())
* SourceFile::ImportFlags
* SourceFile::ImportOptions
* SourceFile::ImportedModuleDesc
This commit is large and intentionally kept mechanical—nothing interesting to see here.
This makes it easier to specify OptionSet arguments.
Also modify appropriate uses of ModuleDecl::ImportFilter to take
advantage of the new constructor.
