The purpose of the different modules wasn’t clearly defined, which lead to inconsistent responsibilities between the different modules. Define each module’s purpose and move a few files between modules to satisfy these definitions.
There are a few more larger changes that will need to be made for a fully consistent module structure. These are FIXMEs in the new Modules.md document and I’ll address them in follow-up PRs.
Previously, we `fatalError`ing when `JSONDecoder` failed to decode a message from the client. Instead of crashing, try recovering from such invalid messages as best as possible. If we know that the state might have gotten out of sync with the client, show a notification message to the user, asking them to file an issue.
rdar://112991102
With the `test-sourcekit-lsp.py` integration test rewritten to not use `--sync`, there are no more users of this option. Remove it since we don’t really test it anyway.
This started off by guarding `state` against concurrent access and then escalated slightly.
The main changes are:
- `state` must only be accessed on `queue` now, preventing race conditions of message handling while the connection is being closed.
- A few functions were renamed, merged, and documented to simplify the code
- A few dead members were removed
The client ID was needed when a `MessageHandler` could handle messages from multiple connections. We don’t support this anymore (because it wasn’t needed) and so the client ID doesn’t need to get passed through as well.
OSLog is the suggesting logging solution on Apple platforms and we should be using it there, taking advantage of the different log levels and privacy masking.
Switch sourcekit-lsp to use OSLog on Apple platforms and implement a logger that is API-compatible with OSLog for all uses in sourcekit-lsp and which can be used on non-Darwin platforms.
The goal of this commit is to introduce the new logging API. There are still improvements about what we log and we can display more privacy-insensitive information after masking. Those changes will be in follow-up commits.
Add `.swift-format` to the repo and format the repo with `swift-format`.
This commit does not add any automation to enforce formatting of sourcekit-lsp in CI. The goal of this commit is to get the majority of source changes out of the way so that the diff of actually enforcing formatting will have fewer changes or conflicts.
This generally seems like the cleaner design because `SourceKitServer` is actually able to semantically inspect the message and decide whether it can be handled concurrently with other requests.
Unfortuantely, we have a few potential out-of-order exeuction possibilities while we migrate everything else to also be asyncronous. But those should be low-probability issues that we can fix in follow-up commits, so I think it’s fine for now. All of these places are marked with `FIXME: (async)`
This is the prerequisite for making `SourceKitServer` an actor, which will mean that the `handle` methods will be `async`.
The current paradigm of returning from `handle` once we can guarantee that there’s no out-of-order execution and then returning the actual result via the callback that’s attached to `Request` is a little weird still. I am hoping to change this paradigm to return the actual result and have a callback function that `handle` can call to indicate that it’s ready to accept another message while guaranteeing in-order execution, essentially flipping the role of the return value and the closure callback. But that’s something to be done after the entire stack has been asyncificied.
When we switch `SourceKitServer`, `SwiftLanguageServer` etc. to be actors, we can’t rely on them to provide ordering guarantees anymore because Swift concurrency doesn’t provide any ordering guarantees.
What we should thus do, is to handle all messages on a serial queue on the `Connection` level. This queue will be blocked from handling any new messages until a message has been sufficiently handled to avoid out-of-order handling of messages. For sourcekitd, this means that
a request has been sent to sourcekitd and for clangd, this means that we have forwarded the request to clangd.
Note that this serial queue is not the main thread, so we will continue accepting data over stdin, just the handling of those messages is blocked.
The asyncification changes caused some non-deterministic test failures. I believe that some of these are due to race conditions that are the result of the partial transition to actors.
Instead of merging the asyncification piece by piece, I will collect the changes asyncification changes in a branch and then qualify that branch througougly (running CI multiple times) before merging it into `main`.
We were waiting for the semaphores outside of the `messageHandlingQueue`, which means that we didn’t actually block the message handling queue until the previous request received a result, effectively rendering `--sync` useless.
Unfortuantely, we have a few potential out-of-order exeuction possibilities while we migrate everything else to also be asyncronous. But those should be low-probability issues that we can fix in follow-up commits, so I think it’s fine for now. All of these places are marked with `FIXME: (async)`
This is the prerequisite for making `SourceKitServer` an actor, which will mean that the `handle` methods will be `async`.
The current paradigm of returning from `handle` once we can guarantee that there’s no out-of-order execution and then returning the actual result via the callback that’s attached to `Request` is a little weird still. I am hoping to change this paradigm to return the actual result and have a callback function that `handle` can call to indicate that it’s ready to accept another message while guaranteeing in-order execution, essentially flipping the role of the return value and the closure callback. But that’s something to be done after the entire stack has been asyncificied.
When we switch `SourceKitServer`, `SwiftLanguageServer` etc. to be actors, we can’t rely on them to provide ordering guarantees anymore because Swift concurrency doesn’t provide any ordering guarantees.
What we should thus do, is to handle all messages on a serial queue on the `Connection` level. This queue will be blocked from handling any new messages until a message has been sufficiently handled to avoid out-of-order handling of messages. For sourcekitd, this means that
a request has been sent to sourcekitd and for clangd, this means that we have forwarded the request to clangd.
Note that this serial queue is not the main thread, so we will continue accepting data over stdin, just the handling of those messages is blocked.
Since Musl is sufficiently different from Glibc (see https://wiki.musl-libc.org/functional-differences-from-glibc.html), it requires a different import, which now should be applied to files that have `import Glibc` in them.
Musl is a low footprint libc that's used in Linux distributions such as Alpine Linux, which allows producing fairly small container images. Additionally, unlike Glibc, musl allows full static linking, meaning apps can be easily distributed to an arbitrary Linux distribution that may have a version of Glibc incompatible with the one that Swift is usually built with or no Glibc installed at all.
Workaround for SR-16097 to avoid hitting SR-16095: VSCode will send shutdown requests without a 'params' key. On macOS getting the superDecoder for a non-existent container returns an empty decoder, on Linux it throws `DecodingError.keyNotFound`, causing a crash.
Perform a targeted fix: If we don't have 'params' and the request is a ShutdownRequest, manually create a `ShutdownRequest` without any parameters.
rdar://91288093
This adjusts the sourcekit-lsp build to use static linking for the
internal libraries. It is not currently possible to build
SourceKitLSP.dll as that requires re-exporting the interfaces from the
consumed modules. However, this allows us to reduce the overall size of
the distribution of SourceKit-LSP by ~1 MiB and reduces the
distributed file set. The values here assume partial static linking of
swift-package-manager, which helps reduce the total size.
Before:
228,352 BuildServerProtocol.dll
1,773,056 LanguageServerProtocol.dll
114,688 LanguageServerProtocolJSONRPC.dll
49,152 LSPLogging.dll
262,656 SKCore.dll
54,784 SKSupport.dll
80,896 SKSwiftPMWorkspace.dll
150,528 SourceKitD.dll
645,632 SourceKitLSP.dll
70,144 sourcekit-lsp.exe
3,429,888 bytes
After:
2,416,640 sourcekit-lsp.exe
2,416,640 bytes
This will add an additional link request for dispatch and Foundation
libraries. These are really required on non-Darwin targets, and should
be satisfied either by the library search path or by explicitly
indicating where the dependencies can be found.
We receive a `SIGPIPE` if we write to a pipe that points to a crashed process. This in particular happens if the target of a `JSONRPCConnection` has crashed and we try to send it a message.
On Darwin, `DispatchIO` ignores `SIGPIPE` for the pipes handled by it, but that features is not available on Linux.
Instead, globally ignore `SIGPIPE` on Linux to prevent us from crashing if the `JSONRPCConnection`'s target crashes.
Fixes rdar://75580936
This actually addresses the real issue that was ignored earlier about
pipes on Windows. The FileHandle cannot provide a non-owning file
descriptor (the returned file descriptor would need to be explicitly
`_close`'d by the receiver). Foundation now vends a `_handle` accessor
to the OS primitive handle. Use this to create the dispatch loop for
messaging. We now create the JSONRPCConnection from handles on Windows
which actually should help enable running some of the tests on Windows
as well.
* Replaces two logs since those don't result in response messages anymore
* Adds a func for sending messages synchronously, to get the two messages out before fatalError
This adjusts the use of Dispatch to build on Windows. Windows does not
provide `stdout_fileno` and `stderr_fileno`. However, it is possible to
use `fileno` to get the associated fileno from the descriptor.
Dispatch on Windows does not deal with fd's but rather with handles.
Convert the file descriptor to a handle and pass that off to dispatch.
The handle is a non-owning reference, and should not be closed.
Fortunately, dispatch does not close the handle when the DispatchIO is
closed.
Specifically, we care that all outstanding **writes** are finished
before we call the close handler, because otherwise we may (a) send
corrupted output during shutdown, or (b) drop notifications and replies
sent during the shutdown process. The former is a potential issue for
clients that are not robust about parse failures, and the latter is an
issue for reproducibility and robustness during testing/debugging - in
particular, we have some integration tests that send data without
waiting for individual replies and they need to finish outstanding
replies before exiting.
rdar://60159448
Since the connection and message handler have a reciprocal need to know
about each other, move the closeHandler so that it has the opportunity
to call a method on the message handler if desired.
