This allows us to flip the default in the future more easily. It also allows users to disable background indexing when it’s enabled by default.
rdar://130280855
The idea here is to unify the different ways in which we can currently set options on SourceKit-LSP in a scalable way: Environment variables, command line arguments to `sourcekit-lsp` and initialization options.
The idea is that a user can define a `~/.sourcekit-lsp/.sourcekit-lsp` file (we store logs in `~/.sourcekit-lsp/logs` on non-Darwin platforms), which will be used as the default configuration for all SourceKit-LSP instances. They can also place a `.sourcekit-lsp` file in the root of a workspace to configure SourceKit-LSP for that project specifically, eg. setting arguments that need to be passed to `swift build` for that project and which thus also need to be set on SourceKit-LSP.
For compatibility reasons, I’m mapping the existing command line options into the new options structure for now. I hope to delete the command line arguments in the future and solely rely on `.sourcekit-lsp` configuration files.
Environment variable will be migrated to `.sourcekit-lsp` in a follow-up commit.
We need to watch for changes to `Package.resolved` so that we can update the dependency checkouts in `.index-build` when the user runs `swift package update`.
Since the `Atomic*` types can not be marked as `Sendable` (because they aren’t C structs), we can change the variables to constants and can remove `nonisolated(unsafe)`.
The `BuildParameters` that we were previously inspecting contained some parameters that SwiftPM synthesizes and that weren’t specified on the command line. For example, it contains `-g` for all languages. This causes us to add `-Xswiftc -g -Xcc -g -Xcxx -g` to the `swift build` invocations for preparation, which is not necessary.
This allows us to fix a toolchain when using a `SwiftPMBuildSystem`, which is critical to ensure that a target gets prepared using the same toolchain that is used to index it and that is used for sourcekitd.
We can have two targets with the same name in a SwiftPM workspace, one for a build target and one for the destination. We need to be able to tell them apart based on the run destination.
We used C atomics but these were allocated as Swift variables. Even thought they were atomic, concurrent accesses to them could violate Swift’s exclusivity laws, raising thread sanitizer errors.
Allocate the C atomics using malloc to fix this problem.
rdar://129170128
Unfortunately, `setpriority` only allows reduction of a process’s priority and doesn’t support priority elevation (unless you are a super user). I still think that it’s valuable to set the process’s priority based on the task priority when it is launched because many indexing processes never get their priority escalated and should thus run in the background.
On Windows, we can elevate the process’s priority.
rdar://127474245
I’d like to qualify `--experimental-prepare-for-indexing` independently of background indexing using `swift build`. Because of this, I think there is value in using SourceKit-LSP using background indexing but without `--experimental-prepare-for-indexing`. It could also be useful to determine if bugs we may find are due to `--experimental-prepare-for-indexing` or also occur when running plain `swift build` commands.
This also means that you can use the index log to view which tasks are currently being executed.
Since we only have a single log stream we can write to, I decided to prefix every line in the index log with two colored emojis that an easy visual association of every log line to the task that generated them.
This improves serial test execution time of eg. `DocumentTestDiscoveryTests` from 36s to 22s because we don’t need to re-build the XCTest module from its interface when using an open source toolchain.
This also uncovered that we weren‘t passing the build setup flags to the prepare command.
rdar://126493151
Package manifests don’t have an associated target to prepare and are represented by a `ConfiguredTarget` with an empty target ID. We were mistakingly running `swift build` with an empty target name to prepare them, which failed. There is nothing to prepare.
We are no longer skipping these tests, which means that we need to make them account for different error messages emitted by Swift 5.10.
I added warnings behind `#if compiler(>=6.1)` to give us a reminder that we can remove these checks when we no longer support running SourceKit-LSP with SwiftPM from Swift 5.10. Swift 6.1 doesn’t have to be this cut-off point but it’s the most likely candidate for now if we want to support the current and last Swift version from tip SourceKit-LSP.
When we detect that we’re running using a 5.10 toolchain, adjust the compiler arguments that we received from SwiftPM to drop any `/Modules` suffixes in the build directory, to account for the fact that 5.10 stores the modules one directory higher up (https://github.com/apple/swift-package-manager/pull/7103).
If the user has enabled background indexing in sourcekit-lsp but opens a project that doesn’t support background indexing (compilation database, build server), we should show a message after opening the workspace, informing the user that background indexing is only supported in SwiftPM projects at the moment.
Fixes#1255
rdar://127474711
When looking for a workspace that can handle a file, we were creating full-fledged workspaces along the way, which we would then discard if they couldn’t handle the file being opened. This had multiple problems:
1. When background indexing is enabled, it caused semantic indexing of the workspace, which wrote files to a `.index-build` directory and was a waste of work
2. When background indexing is enabled, it caused package resolution, which also created a `.index-build` folder to be created
3. It caused a syntactic test index of the workspace, which was a waste of work.
To fix this, do multiple things:
1. When creating a workspace, add a check right after build system creation. This allows us to early exit if the build system can’t handle the file and prevents us from generating the `Workspace`, fixing (1) and (3)
2. Don’t call `reloadPackage` when creating a `SwiftPMWorkspace`. Instead, explicitly call `generateBuildGraph` once we committed to creating the workspace.
This allows a user of SourceKit-LSP to inspect the result of background indexing. This allows a user of SourceKit-LSP to inspect the result of background indexing. I think this gives useful insights into what SourceKit-LSP is indexing and why/how it fails, if it fails, also for users of SourceKit-LSP.
rdar://127474136
Fixes#1265
When the user opens documents from three targets A, B, and C in quick succession, then we don’t want to schedule preparation of wait until A *and* B are finished preparing before preparing C.
Instead, we want to
- Finish for preparation of A to finish if it has already started by the time the file in C is opened. This is done so we always make progress during preparation and don’t get into a scenario where preparation is always cancelled if a user switches between two targets more quickly than it takes to prepare those targets.
- Not prepare B because it is no longer relevant and we haven’t started any progress here. Essentially, we pretend that the hop to B never happened.
Whenever we get request for a document, open it or edit it, trigger a preparation of its target, but don’t block any interaction based on it. This should ensure that the target is usually prepared when the user is interacting with it.
We need to track the preparation status of targets somewhat accurately in `SemanticIndexManager`, so we don’t unnecessarily re-prepare a target. When updating the index store, it is acceptable to schedule another `UpdateIndexStoreTaskDescription` because it will early exit based on an `mtime` check with the unit file. Null builds of a target take significantly longer and thus we want to avoid them.
Fixes#1252
rdar://127474003
When `SwiftPMBuildSystem` operates on a ` .index-build` directory, it owns the checkouts and is thus also allowed to resolve the package versions. This is necessary
