This allows a user of SourceKit-LSP to inspect the result of background indexing. This allows a user of SourceKit-LSP to inspect the result of background indexing. I think this gives useful insights into what SourceKit-LSP is indexing and why/how it fails, if it fails, also for users of SourceKit-LSP.
rdar://127474136
Fixes#1265
When the user opens documents from three targets A, B, and C in quick succession, then we don’t want to schedule preparation of wait until A *and* B are finished preparing before preparing C.
Instead, we want to
- Finish for preparation of A to finish if it has already started by the time the file in C is opened. This is done so we always make progress during preparation and don’t get into a scenario where preparation is always cancelled if a user switches between two targets more quickly than it takes to prepare those targets.
- Not prepare B because it is no longer relevant and we haven’t started any progress here. Essentially, we pretend that the hop to B never happened.
We were mixing the up-to-date status and in-progress status of an index task in `SemanticIndexManager`. This meant that a single `QueuedTask` in the task scheduler could be needed for eg. both preparation for editor functionality in a file of that target and to re-index a file in that target. This dual ownership made it unclear, which caller would be entitled to cancel the task. Furthermore, we needed to duplicate some logic from the preparation task dependencies in `SemanticIndexManager.prepare`.
To simplify things:
- Split the up-to-date status and the in-progress status into two different data structures
- Make the caller of `prepare` and `scheduleIndex` responsible for cancellation of the task it has scheduled. `TaskScheduler` might receive more scheduled tasks this way but the additional tasks should all be no-ops because the status is known to be up-to-date when they execute.
Essentially fix two issues in updating the index store:
1. If there was one task to index `HeaderA.h` through `main.c` and one to index `HeaderB.h` through `main.c`, we would not declare a dependency between them in the task scheduler, which meant that we could have two concurrent and racing index tasks for `main.c`. Declare a dependency between any two files that have the same main file
2. `UpdateIndexStoreTaskDescription` was computing the target to index a file in independently of `SemanticIndexManager`. While they currently always line up, we should pass the target in which to index a file to the `UpdateIndexStoreTaskDescription`. Only this way can we guarantee that we actually prepared the target that the file will be indexed in.
The existing ad-hoc logic was not quite correct because it didn’t eg. remove `-MT/depfile` because it assumed that `-MT` was followed by a space. It also didn’t take into account that `serialize-diagnostics` can be spelled with a single dash or two dashes.
Create a `CompilerCommandLineOption` type that forces decisions to be made about the dash spelling and argument styles, which should help avoid problems like this in the future.
When a header is modified, we don’t we want to re-index all main files that include it. Instead, we just want to index one main to effectively re-index the header itself.
I originally implemented re-indexing of all files that include the header but on second thought, headers are like Swift modules, where we also don’t re-index all dependencies either. And if you change a low-level header that’s included by the entire project, you probably don’t want the indexer to go off and re-index the entire project.
This would have caught a race condition in background indexing that was caused by accessing `CheckedIndex` from multiple threads despite it not being thread-safe.
Details from https://github.com/apple/sourcekit-lsp/issues/1271
> Amazon Linux 2 and CentOS 7 have a glibc that doesn’t support `posix_spawn_file_actions_addchdir_np` and thus `TSCBasic.Process` can’t launch a process on these platforms with the working directory set. We currently fall back to launching the index tasks without a working directory on these platforms, which I think is fine because SwiftPM gives us compiler arguments with absolute paths. But we should figure something out.
>
> Using `Foundation.Process` is not an option because it runs `chdir` on the current process for Posix platforms, which is racy if there are multiple subprocesses being spawned simultaneously. On Windows `TSCBasic.Processs` uses `Foundation.Process` and `Foundation.Process` properly set the working directory of the subprocesses on Windows, so Windows is not a problem.
rdar://127797048
Implements an initial background index when the project is opened.
The following will be implemented in follow-up PRs:
- Resolving package dependencies
- Preparing dependent modules
- Watching for file updates
