#58786 (rdar://93030932) was failing because the `swift-frontend` invocations passed a `swiftExecutablePath` to `Invocation.parseArgs`. This caused the `ClangImporter` instance to point to a `clang` binary next to the `swift-frontend` executable while SourceKit used PATH to find `clang`. The clang executable next to `swift-frontend` doesn’t actually exist because `clang` lives in `llvm-linux-aarch64/bin` and `swift-frontend` lives in `swift-linux-aarch64/bin`.
So some checks for a minimum clang verison failed for the normal build (because the executable doesn’t actually exists) while they pass during the SourceKit build (which used `clang` from `PATH`). This in turn caused the `outline-atomics` to be enabled to the SourceKit clang compiler arguments but not the clang compiler arguments for a normal build and thus resulted in two separate module cache directories (which includes the enabled features in the module directory hash).
To fix this issue, also set the swift executable path for compiler invocations created from SourceKit.
Fixes#58786 (rdar://93030932)
We were only keeping track of `RawSyntax` node IDs to incrementally transfer a syntax tree via JSON. However, AFAICT the incremental JSON transfer option has been superceeded by `SyntaxParseActions`, which are more efficient.
So, let’s clean up and remove the `RawSyntax` node ID and JSON incremental transfer option.
In places that still need a notion of `RawSyntax` identity (like determining the reused syntax regions), use the `RawSyntax`’s pointer instead of the manually created ID.
In `incr_transfer_round_trip.py` always use the code path that uses the `SyntaxParseActions` and remove the transitional code that was still using the incremental JSON transfer but was never called.
In order to make range-shifting for semantic highlighting testable,
disable returning semantic information during an "open" request. This
has no real value anyway, since it only happens very rarely, and it
makes testing range shifting impossible to do deterministically.
rdar://problem/66386179
If a semantic update finishes fast enough, the token snapshot may be
identical to the edit snapshot, but because of getBufferForSnapshot
consolidating edits into a new buffer, we were not detecting that case
properly, and it could cause an assertion failure (or potentially
incorrect range shifting in a release build). This would have reproduced
very rarely in practice, but I can reproduce it by putting `sleep(2)`
calls right before we read the semantic info in open and edit requests.
Incidentally, fix sourcekit-test and unit tests for the (rare) case
where an open or edit already has updated semantic info.
The goal of the test is to test the behaviour when the edit is
range-shifted, but in the (rare) case where the document update happens
before the edit finishes, we need the ranges to be the same. In
particular, using separate statements ensures that the tokens not
touched by the edit are not affected by the edit.
Re-enable the test disabled on ASan, since this seems to be the
underlying issue.
rdar://65934938
This started failing in one CI job that has asan, but it's unclear
whether there is a sanitizer issue or if it's a timing issue, etc.
Disabling while we investigate the root cause.
rdar://65934938
When performing an insertion (replacement length = 0) inside an existing
annotation, we were forming a closed range instead of a half-open range,
causing us to shift the effected token instead of throwing it out. There
were also no tests for this functionality, so add a bunch of annotations
tests.
One area thing that is not tested is what if there have been multiple
edits since the tokens were created. This is difficult to engineer,
because right now making an edit immediately removes the semantic tokens
and returns them. It could happen if the AST build takes longer than the
edits, but there is no way to guarantee that in the current API.
rdar://65748892
The VFS tests were using Unix absolute paths, which does not play well
when Windows see them as relative to the current drive letter.
By using the temporal directory, both Windows and Unix can use the same
paths and avoid the problem.
Additionally, a couple of inputs have to be transformed into the native
path format, because sourcekitd-test compares the inputs as strings, and
they need to match exactly. So the source file and the name of the VFS
entries are transformed into native using the helper from LLVM support.
We weren't accounting for the possibility that the AST would build
faster than we could return editorOpen. While that is very unlikely, we
hit a CI failure at least once.
rdar://52211324
Rather than using the `usleep` function which is both deprecated and
non-portable (does not exist on Windows), prefer using the C++11 chrono library
for the sleep. This also allows us to use the UDL to simplify the code rather
than relying on the comments. NFCI
These tests are occassionaly timing out in CI. I have been unable to
reproduce this even with TSan and varying the pauses between operations.
On the other hand, I can get this test to run slowly if the machine is
heavily loaded, so double the timeout to see if it resolves the CI
issues.
rdar://44842835
When the server shuts down we may still have outstanding async work that
can attempt to trigger a notification, so use a shared_ptr + weak_ptr
instead of unique_ptr + unowned references.
After fixing the lock inversion from the previous commit, this test
appears to be clean under TSan on Linux and macOS, so re-enable it and
hope for the best.
I haven't found any cases where this changes the visible behaviour of
the test, but having the file remain open across invocations was
certainly unexpected and affects how the first part of the test works at
runtime.
We were always returning true from those functions in SKEditorConsumer
and false in the test consumers. On the client side we would then ignore
the return value. So it's clearer to have the functions not return
anything.
The recommended way forward is to use the SyntaxClassifier on the Swift
side.
By removing the C++ SyntaxClassifier, we can also eliminate the
-force-libsyntax-based-processing option that was used to bootstrap
incremental parsing and would generate the syntax map from a syntax
tree.
With more options coming for incremental syntax parsing, the list of
arguments will grow way to large and unhandy, so just extract them into
one common struct.
When adding to the AST consumer queue, keep track of the snapshots
expected and only run such AST consumers when a new enough AST is built.
Progress is ensured because we always run the AST consumer that
triggered the build.
This prevents cases where enqueuing a consumer during an AST build has
different behaviour than enqueuing it after the AST has finished.
rdar://40340631
The enhanced SourceKitd requests are EditorOpen and EdtiorReplaceText. In these two requests, the clients can specify a flag "key. enablesyntaxtree = 1" to get a serialize libSyntax tree with the response.
To help this integration, we added a function in SyntaxParsingContext to explicitly finalize the creation of a SourceFileSyntax to incorporate the fact that SourceKit needs the tree before its destroying the parser instance.
To test this integration, we diff the syntax tree serialized from the frontend action and the tree serialized from a SourceKitd response. They should be identical.
The previous code actually had a "semantic" error as well as a parse
error, so depending on the timing we could get 2 diagnosticsl
rdar://problem/34373697
