Users should not need to rely on this request. The index should always be updated automatically in the background. Having to invoke this request manes there is a bug in SourceKit-LSP's automatic re-indexing. It does, however, offer a workaround to re-index files when such a bug occurs where otherwise there would be no workaround.
rdar://127476221
Resolves#1263
We were sending `SIGINT` to `swift-frontend` processes if they didn’t terminate after 2 minutes. However, `swift-frontend` doesn’t listen to `SIGINT`.
If a task running `waitUntilExitStoppingProcessOnTaskCancellation` is cancelled and the process doesn’t terminate on a `SIGINT` after 2 seconds, kill it.
rdar://130103147
Since the `Atomic*` types can not be marked as `Sendable` (because they aren’t C structs), we can change the variables to constants and can remove `nonisolated(unsafe)`.
We used C atomics but these were allocated as Swift variables. Even thought they were atomic, concurrent accesses to them could violate Swift’s exclusivity laws, raising thread sanitizer errors.
Allocate the C atomics using malloc to fix this problem.
rdar://129170128
Unfortunately, `setpriority` only allows reduction of a process’s priority and doesn’t support priority elevation (unless you are a super user). I still think that it’s valuable to set the process’s priority based on the task priority when it is launched because many indexing processes never get their priority escalated and should thus run in the background.
On Windows, we can elevate the process’s priority.
rdar://127474245
Fallback build settings don’t even have an indexstore path set, so we would never pick up any index data generated from them. Also, indexing with fallback args has some other problems:
- If it did generate a unit file, we would consider the file’s index up-to-date even if the compiler arguments change, which basically means that we wouldn’t get any up-to-date-index even when we do get build settings for the file.
- It’s unlikely that the index from a single file with fallback arguments will be very useful as it can’t tie into the rest of the project.
This also means that you can use the index log to view which tasks are currently being executed.
Since we only have a single log stream we can write to, I decided to prefix every line in the index log with two colored emojis that an easy visual association of every log line to the task that generated them.
I noticed that we were missing a bunch more arguments (in particular for clang index invocations), while diagnosing rdar://129071600, which is fixed by passing `-experimental-allow-module-with-compiler-errors` to the Swift index invocations.
rdar://129071600
Time out updating of the index store after 2 minutes. We don't expect any single file compilation to take longer than 2 minutes in practice, so this indicates that the compiler has entered some kind of loop. We will try indexing the file again when it is edited or when the project is re-opened.
rdar://128732571
Swift packages can have source files that we can’t index (like assembly files). When re-opening, we schedule indexing for those files, which requires the targets to be re-prepared. We skip them earlier.
rdar://128711633
The signposts aren’t easily visible in the log (you need to add `--signpost` to `log show`) and don’t get logged on non-Darwin platforms at all. Add logging for it.
This allows a user of SourceKit-LSP to inspect the result of background indexing. This allows a user of SourceKit-LSP to inspect the result of background indexing. I think this gives useful insights into what SourceKit-LSP is indexing and why/how it fails, if it fails, also for users of SourceKit-LSP.
rdar://127474136
Fixes#1265
When the user opens documents from three targets A, B, and C in quick succession, then we don’t want to schedule preparation of wait until A *and* B are finished preparing before preparing C.
Instead, we want to
- Finish for preparation of A to finish if it has already started by the time the file in C is opened. This is done so we always make progress during preparation and don’t get into a scenario where preparation is always cancelled if a user switches between two targets more quickly than it takes to prepare those targets.
- Not prepare B because it is no longer relevant and we haven’t started any progress here. Essentially, we pretend that the hop to B never happened.
We were mixing the up-to-date status and in-progress status of an index task in `SemanticIndexManager`. This meant that a single `QueuedTask` in the task scheduler could be needed for eg. both preparation for editor functionality in a file of that target and to re-index a file in that target. This dual ownership made it unclear, which caller would be entitled to cancel the task. Furthermore, we needed to duplicate some logic from the preparation task dependencies in `SemanticIndexManager.prepare`.
To simplify things:
- Split the up-to-date status and the in-progress status into two different data structures
- Make the caller of `prepare` and `scheduleIndex` responsible for cancellation of the task it has scheduled. `TaskScheduler` might receive more scheduled tasks this way but the additional tasks should all be no-ops because the status is known to be up-to-date when they execute.
Essentially fix two issues in updating the index store:
1. If there was one task to index `HeaderA.h` through `main.c` and one to index `HeaderB.h` through `main.c`, we would not declare a dependency between them in the task scheduler, which meant that we could have two concurrent and racing index tasks for `main.c`. Declare a dependency between any two files that have the same main file
2. `UpdateIndexStoreTaskDescription` was computing the target to index a file in independently of `SemanticIndexManager`. While they currently always line up, we should pass the target in which to index a file to the `UpdateIndexStoreTaskDescription`. Only this way can we guarantee that we actually prepared the target that the file will be indexed in.
The existing ad-hoc logic was not quite correct because it didn’t eg. remove `-MT/depfile` because it assumed that `-MT` was followed by a space. It also didn’t take into account that `serialize-diagnostics` can be spelled with a single dash or two dashes.
Create a `CompilerCommandLineOption` type that forces decisions to be made about the dash spelling and argument styles, which should help avoid problems like this in the future.
When a header is modified, we don’t we want to re-index all main files that include it. Instead, we just want to index one main to effectively re-index the header itself.
I originally implemented re-indexing of all files that include the header but on second thought, headers are like Swift modules, where we also don’t re-index all dependencies either. And if you change a low-level header that’s included by the entire project, you probably don’t want the indexer to go off and re-index the entire project.
This would have caught a race condition in background indexing that was caused by accessing `CheckedIndex` from multiple threads despite it not being thread-safe.
Details from https://github.com/apple/sourcekit-lsp/issues/1271
> Amazon Linux 2 and CentOS 7 have a glibc that doesn’t support `posix_spawn_file_actions_addchdir_np` and thus `TSCBasic.Process` can’t launch a process on these platforms with the working directory set. We currently fall back to launching the index tasks without a working directory on these platforms, which I think is fine because SwiftPM gives us compiler arguments with absolute paths. But we should figure something out.
>
> Using `Foundation.Process` is not an option because it runs `chdir` on the current process for Posix platforms, which is racy if there are multiple subprocesses being spawned simultaneously. On Windows `TSCBasic.Processs` uses `Foundation.Process` and `Foundation.Process` properly set the working directory of the subprocesses on Windows, so Windows is not a problem.
rdar://127797048
Implements an initial background index when the project is opened.
The following will be implemented in follow-up PRs:
- Resolving package dependencies
- Preparing dependent modules
- Watching for file updates
