This allows us to flip the default in the future more easily. It also allows users to disable background indexing when it’s enabled by default.
rdar://130280855
The idea here is to unify the different ways in which we can currently set options on SourceKit-LSP in a scalable way: Environment variables, command line arguments to `sourcekit-lsp` and initialization options.
The idea is that a user can define a `~/.sourcekit-lsp/.sourcekit-lsp` file (we store logs in `~/.sourcekit-lsp/logs` on non-Darwin platforms), which will be used as the default configuration for all SourceKit-LSP instances. They can also place a `.sourcekit-lsp` file in the root of a workspace to configure SourceKit-LSP for that project specifically, eg. setting arguments that need to be passed to `swift build` for that project and which thus also need to be set on SourceKit-LSP.
For compatibility reasons, I’m mapping the existing command line options into the new options structure for now. I hope to delete the command line arguments in the future and solely rely on `.sourcekit-lsp` configuration files.
Environment variable will be migrated to `.sourcekit-lsp` in a follow-up commit.
We were sending `SIGINT` to `swift-frontend` processes if they didn’t terminate after 2 minutes. However, `swift-frontend` doesn’t listen to `SIGINT`.
If a task running `waitUntilExitStoppingProcessOnTaskCancellation` is cancelled and the process doesn’t terminate on a `SIGINT` after 2 seconds, kill it.
rdar://130103147
The purpose of the different modules wasn’t clearly defined, which lead to inconsistent responsibilities between the different modules. Define each module’s purpose and move a few files between modules to satisfy these definitions.
There are a few more larger changes that will need to be made for a fully consistent module structure. These are FIXMEs in the new Modules.md document and I’ll address them in follow-up PRs.
------
Simplify `SemanticRefactoring` with new `Refactoring` protocol to handle sourcekitd requests
Create and implement `ExpandMacroCommand` while temporarily storing generated expansions.
Create test case `testFreestandingMacroExpansion`
Manually inject `ExpandMacroCommand` into `retrieveRefactorCodeActions` upon an "Inline Macro" from sourcekitd
Address Review Comments
Mark `@_spi(Testing) public` for `MacroExpansionEdit`
Address Review Comments
Create separate directory for each buffer with its name, containing a generated file named as the source file along with position range
Fixed generated macro expansion file extension not recognised, by switching to file names which don't contain fragments
Address Review Comments
Wrap the entire feature under `ExperimentalFeatures`
Address Review Comments
Make Swift Lint Pass
Fix Windows Build not passing
Since the `Atomic*` types can not be marked as `Sendable` (because they aren’t C structs), we can change the variables to constants and can remove `nonisolated(unsafe)`.
We used C atomics but these were allocated as Swift variables. Even thought they were atomic, concurrent accesses to them could violate Swift’s exclusivity laws, raising thread sanitizer errors.
Allocate the C atomics using malloc to fix this problem.
rdar://129170128
Unfortunately, `setpriority` only allows reduction of a process’s priority and doesn’t support priority elevation (unless you are a super user). I still think that it’s valuable to set the process’s priority based on the task priority when it is launched because many indexing processes never get their priority escalated and should thus run in the background.
On Windows, we can elevate the process’s priority.
rdar://127474245
This also means that you can use the index log to view which tasks are currently being executed.
Since we only have a single log stream we can write to, I decided to prefix every line in the index log with two colored emojis that an easy visual association of every log line to the task that generated them.
Otherwise, I think `ThreadSafeBox` might still have data races. This also requires us to make `TestSourceKitLSPClient.RequestHandler` sendable.
rdar://128572489
Time out updating of the index store after 2 minutes. We don't expect any single file compilation to take longer than 2 minutes in practice, so this indicates that the compiler has entered some kind of loop. We will try indexing the file again when it is edited or when the project is re-opened.
rdar://128732571
We need this function anyway to escalate process priorities when we set `nice`ness values for them. It also simplifies the task scheduler and I’m hoping that it fixes a non-deterministic failure that causes task priority elevation to not work properly.
When the user opens documents from three targets A, B, and C in quick succession, then we don’t want to schedule preparation of wait until A *and* B are finished preparing before preparing C.
Instead, we want to
- Finish for preparation of A to finish if it has already started by the time the file in C is opened. This is done so we always make progress during preparation and don’t get into a scenario where preparation is always cancelled if a user switches between two targets more quickly than it takes to prepare those targets.
- Not prepare B because it is no longer relevant and we haven’t started any progress here. Essentially, we pretend that the hop to B never happened.
We were mixing the up-to-date status and in-progress status of an index task in `SemanticIndexManager`. This meant that a single `QueuedTask` in the task scheduler could be needed for eg. both preparation for editor functionality in a file of that target and to re-index a file in that target. This dual ownership made it unclear, which caller would be entitled to cancel the task. Furthermore, we needed to duplicate some logic from the preparation task dependencies in `SemanticIndexManager.prepare`.
To simplify things:
- Split the up-to-date status and the in-progress status into two different data structures
- Make the caller of `prepare` and `scheduleIndex` responsible for cancellation of the task it has scheduled. `TaskScheduler` might receive more scheduled tasks this way but the additional tasks should all be no-ops because the status is known to be up-to-date when they execute.
This would have caught a race condition in background indexing that was caused by accessing `CheckedIndex` from multiple threads despite it not being thread-safe.
Details from https://github.com/apple/sourcekit-lsp/issues/1271
> Amazon Linux 2 and CentOS 7 have a glibc that doesn’t support `posix_spawn_file_actions_addchdir_np` and thus `TSCBasic.Process` can’t launch a process on these platforms with the working directory set. We currently fall back to launching the index tasks without a working directory on these platforms, which I think is fine because SwiftPM gives us compiler arguments with absolute paths. But we should figure something out.
>
> Using `Foundation.Process` is not an option because it runs `chdir` on the current process for Posix platforms, which is racy if there are multiple subprocesses being spawned simultaneously. On Windows `TSCBasic.Processs` uses `Foundation.Process` and `Foundation.Process` properly set the working directory of the subprocesses on Windows, so Windows is not a problem.
rdar://127797048
This workspace-wide syntactic test index is used for two purposes:
- It is used for XCTests instead of the semantic index for files that have on-disk or in-memory modifications to files
- It is uses for swift-testing tests, which are only discovered syntactically.
rdar://119191037
Instead of returning `nil` to indicate that the position conversion failed, log a fault and perform a best-effort recovery.
I think this allows us to perform better recovery and also makes code calling these position conversions a lot simpler because it doesn’t need to make decisions about what to do if position conversions fail.
Instead of logging errors in position translation ad-hoc at the caller’s side (and ofter forgetting to do so), log these errors in `LineTable`. To be able to debug where the position conversion error is coming from, also log the file name and line number of the caller.
rdar://125545620
