The exact cause of this is a mystery to me - each test case should be
finished (and thus any ASTs fully built) before the next runs/we exit.
But one of the recent failures contained a stacktrace:
```
...
20 [ra] 0x0000aaaad9d182f0 llvm::opt::OptTable::internalParseOneArg(llvm::opt::ArgList const&, unsigned int&, std::function<bool (llvm::opt::Option const&)>) const + 127 in SourceKitSwiftLangTests
21 [ra] 0x0000aaaad9d188d0 llvm::opt::OptTable::internalParseArgs(llvm::ArrayRef<char const*>, unsigned int&, unsigned int&, std::function<bool (llvm::opt::Option const&)>) const + 303 in SourceKitSwiftLangTests
22 [ra] 0x0000aaaad9d1877c llvm::opt::OptTable::ParseArgs(llvm::ArrayRef<char const*>, unsigned int&, unsigned int&, llvm::opt::Visibility) const + 47 in SourceKitSwiftLangTests
23 [ra] 0x0000aaaad6fad0d0 getLibcFileMapping[abi:cxx11](swift::ASTContext&, llvm::StringRef, std::optional<llvm::ArrayRef<llvm::StringRef> >, llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem> const&, bool) + 311 in SourceKitSwiftLangTests
24 [ra] 0x0000aaaad6fab304 swift::getClangInvocationFileMapping(swift::ASTContext&, llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem>, bool) + 895 in SourceKitSwiftLangTests
25 [ra] 0x0000aaaad6f4dcac swift::ClangImporter::create(swift::ASTContext&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, swift::DependencyTracker*, swift::DWARFImporterDelegate*, bool) + 587 in SourceKitSwiftLangTests
26 [ra] 0x0000aaaad5ba07ac swift::CompilerInstance::setUpModuleLoaders() + 531 in SourceKitSwiftLangTests
27 [ra] 0x0000aaaad5ba0530 swift::CompilerInstance::setUpASTContextIfNeeded() + 447 in SourceKitSwiftLangTests
28 [ra] 0x0000aaaad5ba269c swift::CompilerInstance::setup(swift::CompilerInvocation const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, llvm::ArrayRef<char const*>) + 231 in SourceKitSwiftLangTests
29 [ra] 0x0000aaaad593d634 void SourceKit::WorkQueue::DispatchData::callAndDelete<(anonymous namespace)::ASTBuildOperation::schedule(SourceKit::WorkQueue)::$_11>(void*) + 2155 in SourceKitSwiftLangTests
30 [ra] 0x0000aaaad597c2fc executeBlock(void*) + 27 in SourceKitSwiftLangTests
31 [ra] 0x0000aaaad597c330 void* llvm::thread::ThreadProxy<std::tuple<void (*)(void*), void*> >(void*) + 27 in SourceKitSwiftLangTests
32 [ra] 0x0000ffffab37595c <unknown> in libc.so.6
```
So the assertion above clearly isn't true - we are definitely still
building an AST on exit here.
Regardless, this test is not worth CI failing as often as it is. Let's
disable it for now while we look into the cause.
This allows us to load plugins into the sourcekitd service to handle requests (currently only used for code completion). This allows us to implement parts of sourcekitd in Swift and outside of the compiler repository, making it easier to iterated on them because the compiler doesn’t need to be rebuilt.
Introduces the new DeclarationsArrayBuilder and adds it to the
EditorConsumer. Declaration info always includes a kind, offset, and
length, and includes a USR where applicable.
As the USR is already available for editor.open.interface type requests,
this doesn't compute any new information, it just exposes more of what's
there already.
Introduce a new key `generated_buffers`, which
stores an array of generated buffers. These
include the buffer text, as well as its original
location and any parent buffers.
While here, also fix rdar://107281079 such that
only apply the filename fallback logic to the
pretty-printed Decl case. We ought to remove this
fallback once the editor can handle it though.
rdar://107281079
rdar://107952288
Update requests to handle being passed a separate `key.primary_file`
which specifies the file to use for building the AST. `key.sourcefile`
is then the file to find in `SourceManager`, which could be a generated
buffer.
Resolves rdar://106863186.
Running the SourceKit stress tester with verification of solver-based cursor info returned quite a few differences but in all of them, the old AST-based implementation was actually incorrect. So, instead of verifying the results, deliver the results from solver-baesd cursor info and only fall back to AST-based cursor info if the solver-based implementation returned no results.
rdar://103369449
This allows us to mark expected deviations between the AST-based and the solver-based implementation in the stress tester as XFails without breaking actual clients
We always verify if a cursor info request is issued through `sourcekitd-test`.
Use the std-equivalent names as the LLVM ones are now deprecated
(eventually `llvm::Optional` will disappear):
- `getValue` -> `value`
- `getValueOr` -> `value_or`
- `hasValue` -> `has_value`
Follow up from ab1b343dad and
7d8bf37e5e with some missing cases.
This enables the ability to cancel requests, which aren’t code completion requests, again.
Previous crashes in SILGen are prevented by disabling cancellation during the SIL stages. Instead, we add dedicated cancellation checkpoints before and after SIL.
rdar://98390926
#58786 (rdar://93030932) was failing because the `swift-frontend` invocations passed a `swiftExecutablePath` to `Invocation.parseArgs`. This caused the `ClangImporter` instance to point to a `clang` binary next to the `swift-frontend` executable while SourceKit used PATH to find `clang`. The clang executable next to `swift-frontend` doesn’t actually exist because `clang` lives in `llvm-linux-aarch64/bin` and `swift-frontend` lives in `swift-linux-aarch64/bin`.
So some checks for a minimum clang verison failed for the normal build (because the executable doesn’t actually exists) while they pass during the SourceKit build (which used `clang` from `PATH`). This in turn caused the `outline-atomics` to be enabled to the SourceKit clang compiler arguments but not the clang compiler arguments for a normal build and thus resulted in two separate module cache directories (which includes the enabled features in the module directory hash).
To fix this issue, also set the swift executable path for compiler invocations created from SourceKit.
Fixes#58786 (rdar://93030932)
We need to run SILGen for diagnostics (to actually get all diagnostics).
All non-completion requests share an AST and thus they too run SILGen.
Any lazy typechecking run in SILGen assumes that it succeeds.
Cancellation can cause typechecking to fail here though, since we simply
check the flag and error if it's set. This unfortunately has the ability
to cause any any number of crashes since various invariants in SILGen
are then broken.
Disable cancellation of in-flight non-completion requests for now until
we have a proper fix in place.
Resolves rdar://91251017.
Previously, `SwiftASTManager` and `SlowRequestSimulator` maintained their own list of in-progress cancellation tokens. With code completion cancellation coming up, there would need to be yet another place to track in-progress requests, so let’s centralize it.
While at it, also support cancelling requests before they are scheduled, eliminating the need for a `sleep` in a test case.
The current implementaiton leaks tiny amounts of memory if a request is cancelled after if finishes. I think this is fine because it is a pretty nieche case and the leaked memory is pretty small (a `std::map` entry pointing to a `std::function` + `bool`). Alternatively, we could require the client to always dispose of the cancellation token manually.
The key changes here are
- To keep track of cancellation tokens for all `ScheduledConsumer`s in `SwiftASTManager`
- Generate unique request handles for all incoming requests (`create_request_handle `), use these request handles as cancellation tokens and return them from the `sourcekitd_send_request` methods
- Implement cancellation with `sourcekitd_cancel_request` as the entry point and `SwiftASTManager::cancelASTConsumer` as the termination point
Everything else is just plumbing the cancellation token through the various abstraction layers.
rdar://83391505
This commit refactors the way ASTs are being built in SourceKit and how `SwiftASTConsumer`s are served by the built ASTs. `SwiftASTManager.h` should give an overview of the new design.
This commit does not change the cancellation paradigm in SourceKit (yet). That is, subsequent requests with the same `OncePerASTToken` still cancel previous requests with the same token. But while previously, we were only able to cancel requests that haven’t started an AST build yet, we can now also cancel the AST build of the to-be-cancelled requests.
With this change in place, we can start looking into explicit cancellation of requests or other cancellation paradigms.
We were only keeping track of `RawSyntax` node IDs to incrementally transfer a syntax tree via JSON. However, AFAICT the incremental JSON transfer option has been superceeded by `SyntaxParseActions`, which are more efficient.
So, let’s clean up and remove the `RawSyntax` node ID and JSON incremental transfer option.
In places that still need a notion of `RawSyntax` identity (like determining the reused syntax regions), use the `RawSyntax`’s pointer instead of the manually created ID.
In `incr_transfer_round_trip.py` always use the code path that uses the `SyntaxParseActions` and remove the transitional code that was still using the incremental JSON transfer but was never called.
Cursor info for a constructor would previously give the cursor info for
the containing type only. It now also adds cursor info for the
constructor itself in a "secondary_symbols" field.
Refactor `passCursorInfoForDecl` to use a single allocator rather than
keeping track of positions in a buffer and assigning everything at the
end of the function.
Refactor the various available refactoring gathering functions to take a
SmallVectorImpl and to not copy strings where they don't need to.
Resolves rdar://75385556
Adds a new 'key.retrieve_symbol_graph' option to the request. When set to 1 it
includes the JSON for a SymbolGraph containing a single node for the symbol at
the requested position.
This also extends the SymbolGraph library with a new entry point to get a graph
for a single symbol, and to additionally support type substitution to match the
existing CursorInfo behavior (e.g. so that when invoked on `first` in
`Array<Int>().first`, the type is given as `Int?` rather than `Element?`).
Resolves rdar://problem/70551509
In order to make range-shifting for semantic highlighting testable,
disable returning semantic information during an "open" request. This
has no real value anyway, since it only happens very rarely, and it
makes testing range shifting impossible to do deterministically.
rdar://problem/66386179
If a semantic update finishes fast enough, the token snapshot may be
identical to the edit snapshot, but because of getBufferForSnapshot
consolidating edits into a new buffer, we were not detecting that case
properly, and it could cause an assertion failure (or potentially
incorrect range shifting in a release build). This would have reproduced
very rarely in practice, but I can reproduce it by putting `sleep(2)`
calls right before we read the semantic info in open and edit requests.
Incidentally, fix sourcekit-test and unit tests for the (rare) case
where an open or edit already has updated semantic info.
The goal of the test is to test the behaviour when the edit is
range-shifted, but in the (rare) case where the document update happens
before the edit finishes, we need the ranges to be the same. In
particular, using separate statements ensures that the tokens not
touched by the edit are not affected by the edit.
Re-enable the test disabled on ASan, since this seems to be the
underlying issue.
rdar://65934938
This started failing in one CI job that has asan, but it's unclear
whether there is a sanitizer issue or if it's a timing issue, etc.
Disabling while we investigate the root cause.
rdar://65934938
When performing an insertion (replacement length = 0) inside an existing
annotation, we were forming a closed range instead of a half-open range,
causing us to shift the effected token instead of throwing it out. There
were also no tests for this functionality, so add a bunch of annotations
tests.
One area thing that is not tested is what if there have been multiple
edits since the tokens were created. This is difficult to engineer,
because right now making an edit immediately removes the semantic tokens
and returns them. It could happen if the AST build takes longer than the
edits, but there is no way to guarantee that in the current API.
rdar://65748892
The VFS tests were using Unix absolute paths, which does not play well
when Windows see them as relative to the current drive letter.
By using the temporal directory, both Windows and Unix can use the same
paths and avoid the problem.
Additionally, a couple of inputs have to be transformed into the native
path format, because sourcekitd-test compares the inputs as strings, and
they need to match exactly. So the source file and the name of the VFS
entries are transformed into native using the helper from LLVM support.
We weren't accounting for the possibility that the AST would build
faster than we could return editorOpen. While that is very unlikely, we
hit a CI failure at least once.
rdar://52211324
Previously, requests would fail silently by returning an empty struct
in the response.
With this change, responses will properly report fail with the internal
error.
