The idea here is to unify the different ways in which we can currently set options on SourceKit-LSP in a scalable way: Environment variables, command line arguments to `sourcekit-lsp` and initialization options.
The idea is that a user can define a `~/.sourcekit-lsp/.sourcekit-lsp` file (we store logs in `~/.sourcekit-lsp/logs` on non-Darwin platforms), which will be used as the default configuration for all SourceKit-LSP instances. They can also place a `.sourcekit-lsp` file in the root of a workspace to configure SourceKit-LSP for that project specifically, eg. setting arguments that need to be passed to `swift build` for that project and which thus also need to be set on SourceKit-LSP.
For compatibility reasons, I’m mapping the existing command line options into the new options structure for now. I hope to delete the command line arguments in the future and solely rely on `.sourcekit-lsp` configuration files.
Environment variable will be migrated to `.sourcekit-lsp` in a follow-up commit.
# Conflicts:
# Sources/SourceKitLSP/SourceKitLSPServer+Options.swift
# Sources/SourceKitLSP/Swift/SwiftLanguageService.swift
# Sources/sourcekit-lsp/SourceKitLSP.swift
# Tests/SourceKitLSPTests/BackgroundIndexingTests.swift
# Tests/SourceKitLSPTests/ExecuteCommandTests.swift
This allows us to fix a toolchain when using a `SwiftPMBuildSystem`, which is critical to ensure that a target gets prepared using the same toolchain that is used to index it and that is used for sourcekitd.
This also means that you can use the index log to view which tasks are currently being executed.
Since we only have a single log stream we can write to, I decided to prefix every line in the index log with two colored emojis that an easy visual association of every log line to the task that generated them.
If the user has enabled background indexing in sourcekit-lsp but opens a project that doesn’t support background indexing (compilation database, build server), we should show a message after opening the workspace, informing the user that background indexing is only supported in SwiftPM projects at the moment.
Fixes#1255
rdar://127474711
When looking for a workspace that can handle a file, we were creating full-fledged workspaces along the way, which we would then discard if they couldn’t handle the file being opened. This had multiple problems:
1. When background indexing is enabled, it caused semantic indexing of the workspace, which wrote files to a `.index-build` directory and was a waste of work
2. When background indexing is enabled, it caused package resolution, which also created a `.index-build` folder to be created
3. It caused a syntactic test index of the workspace, which was a waste of work.
To fix this, do multiple things:
1. When creating a workspace, add a check right after build system creation. This allows us to early exit if the build system can’t handle the file and prevents us from generating the `Workspace`, fixing (1) and (3)
2. Don’t call `reloadPackage` when creating a `SwiftPMWorkspace`. Instead, explicitly call `generateBuildGraph` once we committed to creating the workspace.
This allows a user of SourceKit-LSP to inspect the result of background indexing. This allows a user of SourceKit-LSP to inspect the result of background indexing. I think this gives useful insights into what SourceKit-LSP is indexing and why/how it fails, if it fails, also for users of SourceKit-LSP.
rdar://127474136
Fixes#1265
Instead of asking for build settings of a file, the build system manager asks for the targets of a file and then asks for the build settings of that file in a specific target. This has two advantages:
- We know about targets and can prepare the targets for background indexing
- Once we support build systems in which a single file can be part of multiple targets, we can have a centralized place that picks preferred targets for a file, eg. based on user configuration
This workspace-wide syntactic test index is used for two purposes:
- It is used for XCTests instead of the semantic index for files that have on-disk or in-memory modifications to files
- It is uses for swift-testing tests, which are only discovered syntactically.
rdar://119191037
The client ID was needed when a `MessageHandler` could handle messages from multiple connections. We don’t support this anymore (because it wasn’t needed) and so the client ID doesn’t need to get passed through as well.
Instead of just having barriers and non-barriers, this allows `AsyncQueue` to track dependencies between tasks at a more fine-grained level.
For example, we can now specify that requests that affect one document only depend on edits to that same document and are not blocked by edits to any other document. As a consequence, a busy `sourcekitd` will not block requests from `clangd` to be executed and vice versa.
Resolves#875
rdar://116705652
OSLog is the suggesting logging solution on Apple platforms and we should be using it there, taking advantage of the different log levels and privacy masking.
Switch sourcekit-lsp to use OSLog on Apple platforms and implement a logger that is API-compatible with OSLog for all uses in sourcekit-lsp and which can be used on non-Darwin platforms.
The goal of this commit is to introduce the new logging API. There are still improvements about what we log and we can display more privacy-insensitive information after masking. Those changes will be in follow-up commits.
Add `.swift-format` to the repo and format the repo with `swift-format`.
This commit does not add any automation to enforce formatting of sourcekit-lsp in CI. The goal of this commit is to get the majority of source changes out of the way so that the diff of actually enforcing formatting will have fewer changes or conflicts.
This generally seems like the cleaner design because `SourceKitServer` is actually able to semantically inspect the message and decide whether it can be handled concurrently with other requests.
This defines away an entire class of data races if delegate callbacks are delivered out-of-order. If we aren’t providing the new build settings in the delegate callback, then it doesn’t matter if two `fileBuildSettingsChanged` calls change order since they don’t carry any state.
Instead of storing build settings inside the language servers based on update notifications from the build system, always call into the `BuildSystemManager` to get the build settings.
Overall, I think this is a much clearer separation of concerns and will allow us to remove `SourceKitServer.documentToPendingQueue` in a follow-up commit as `SwiftLanguageServer` can always directly call into `BuildSystemManager` to get build settings and we don’t need to wait for the initial notification to receive the first build settings.
This requies `BuildServerBuildSystem` to keep track of the build settings it has received from the BSP server.
`ClangLanguageServer` still caches build settings locally. `ClangLanguageServer` will change to the same pull-based model in a follow-up commit.
The asyncification changes caused some non-deterministic test failures. I believe that some of these are due to race conditions that are the result of the partial transition to actors.
Instead of merging the asyncification piece by piece, I will collect the changes asyncification changes in a branch and then qualify that branch througougly (running CI multiple times) before merging it into `main`.
Instead of storing build settings inside the language servers based on update notifications from the build system, always call into the `BuildSystemManager` to get the build settings.
Overall, I think this is a much clearer separation of concerns and will allow us to remove `SourceKitServer.documentToPendingQueue` in a follow-up commit as `SwiftLanguageServer` can always directly call into `BuildSystemManager` to get build settings and we don’t need to wait for the initial notification to receive the first build settings.
This requies `BuildServerBuildSystem` to keep track of the build settings it has received from the BSP server.
`ClangLanguageServer` still caches build settings locally. `ClangLanguageServer` will change to the same pull-based model in a follow-up commit.
Instead of storing build settings inside the language servers based on update notifications from the build system, always call into the `BuildSystemManager` to get the build settings.
Overall, I think this is a much clearer separation of concerns and will allow us to remove `SourceKitServer.documentToPendingQueue` in a follow-up commit as `SwiftLanguageServer` can always directly call into `BuildSystemManager` to get build settings and we don’t need to wait for the initial notification to receive the first build settings.
This requies `BuildServerBuildSystem` to keep track of the build settings it has received from the BSP server.
`ClangLanguageServer` still caches build settings locally. `ClangLanguageServer` will change to the same pull-based model in a follow-up commit.
They always existed in tandem and it makes migration to actors a lot easier if the handling of build server messages occurs within the same actor isolation domain as the build system iteself.
Since we moved `BuilderServerHandler` and `ClangLanguageServerShim` to directly conform to `MessageHandler`, `SourceKitServer` was the only type inheriting from `LanguageServerEndpoint` and we can thus remove the type.
We are moving to a better model for TSC's path APIs in apple/swift-tools-support-core#353. The previous API is still available (but deprecated) as much as possible, but since SourceKit-LSP was using `resolveSymlinks` (which is now throwing) quite a bit, there are some changes necessary.
Explicitly import interfaces from TSCBasic which now allows us to
identify all the swift-tools-support-core interfaces which are in
use in SourceKit-LSP.
The test system depended on the shebang to locate the python
interpreter. However, this is not a portable system. Instead, prefer
to explicitly search for the interpreter prior to the execution. This
enables supporting execution of the script support on all platforms. A
secondary change of the printed string is required for Windows. Python
will replace `\n` with `\r\n` resulting in `\r\n` being emitted as
`\r\r\n` on Windows breaking the expectations on the receiver. Adjust
this by explicitly writing out a binary string to the raw underlying
buffer to avoid the translation.
