Previously, we `fatalError`ing when `JSONDecoder` failed to decode a message from the client. Instead of crashing, try recovering from such invalid messages as best as possible. If we know that the state might have gotten out of sync with the client, show a notification message to the user, asking them to file an issue.
rdar://112991102
With the `test-sourcekit-lsp.py` integration test rewritten to not use `--sync`, there are no more users of this option. Remove it since we don’t really test it anyway.
This started off by guarding `state` against concurrent access and then escalated slightly.
The main changes are:
- `state` must only be accessed on `queue` now, preventing race conditions of message handling while the connection is being closed.
- A few functions were renamed, merged, and documented to simplify the code
- A few dead members were removed
The client ID was needed when a `MessageHandler` could handle messages from multiple connections. We don’t support this anymore (because it wasn’t needed) and so the client ID doesn’t need to get passed through as well.
OSLog is the suggesting logging solution on Apple platforms and we should be using it there, taking advantage of the different log levels and privacy masking.
Switch sourcekit-lsp to use OSLog on Apple platforms and implement a logger that is API-compatible with OSLog for all uses in sourcekit-lsp and which can be used on non-Darwin platforms.
The goal of this commit is to introduce the new logging API. There are still improvements about what we log and we can display more privacy-insensitive information after masking. Those changes will be in follow-up commits.
Add `.swift-format` to the repo and format the repo with `swift-format`.
This commit does not add any automation to enforce formatting of sourcekit-lsp in CI. The goal of this commit is to get the majority of source changes out of the way so that the diff of actually enforcing formatting will have fewer changes or conflicts.
This generally seems like the cleaner design because `SourceKitServer` is actually able to semantically inspect the message and decide whether it can be handled concurrently with other requests.
Unfortuantely, we have a few potential out-of-order exeuction possibilities while we migrate everything else to also be asyncronous. But those should be low-probability issues that we can fix in follow-up commits, so I think it’s fine for now. All of these places are marked with `FIXME: (async)`
This is the prerequisite for making `SourceKitServer` an actor, which will mean that the `handle` methods will be `async`.
The current paradigm of returning from `handle` once we can guarantee that there’s no out-of-order execution and then returning the actual result via the callback that’s attached to `Request` is a little weird still. I am hoping to change this paradigm to return the actual result and have a callback function that `handle` can call to indicate that it’s ready to accept another message while guaranteeing in-order execution, essentially flipping the role of the return value and the closure callback. But that’s something to be done after the entire stack has been asyncificied.
When we switch `SourceKitServer`, `SwiftLanguageServer` etc. to be actors, we can’t rely on them to provide ordering guarantees anymore because Swift concurrency doesn’t provide any ordering guarantees.
What we should thus do, is to handle all messages on a serial queue on the `Connection` level. This queue will be blocked from handling any new messages until a message has been sufficiently handled to avoid out-of-order handling of messages. For sourcekitd, this means that
a request has been sent to sourcekitd and for clangd, this means that we have forwarded the request to clangd.
Note that this serial queue is not the main thread, so we will continue accepting data over stdin, just the handling of those messages is blocked.
The asyncification changes caused some non-deterministic test failures. I believe that some of these are due to race conditions that are the result of the partial transition to actors.
Instead of merging the asyncification piece by piece, I will collect the changes asyncification changes in a branch and then qualify that branch througougly (running CI multiple times) before merging it into `main`.
We were waiting for the semaphores outside of the `messageHandlingQueue`, which means that we didn’t actually block the message handling queue until the previous request received a result, effectively rendering `--sync` useless.
Unfortuantely, we have a few potential out-of-order exeuction possibilities while we migrate everything else to also be asyncronous. But those should be low-probability issues that we can fix in follow-up commits, so I think it’s fine for now. All of these places are marked with `FIXME: (async)`
This is the prerequisite for making `SourceKitServer` an actor, which will mean that the `handle` methods will be `async`.
The current paradigm of returning from `handle` once we can guarantee that there’s no out-of-order execution and then returning the actual result via the callback that’s attached to `Request` is a little weird still. I am hoping to change this paradigm to return the actual result and have a callback function that `handle` can call to indicate that it’s ready to accept another message while guaranteeing in-order execution, essentially flipping the role of the return value and the closure callback. But that’s something to be done after the entire stack has been asyncificied.
When we switch `SourceKitServer`, `SwiftLanguageServer` etc. to be actors, we can’t rely on them to provide ordering guarantees anymore because Swift concurrency doesn’t provide any ordering guarantees.
What we should thus do, is to handle all messages on a serial queue on the `Connection` level. This queue will be blocked from handling any new messages until a message has been sufficiently handled to avoid out-of-order handling of messages. For sourcekitd, this means that
a request has been sent to sourcekitd and for clangd, this means that we have forwarded the request to clangd.
Note that this serial queue is not the main thread, so we will continue accepting data over stdin, just the handling of those messages is blocked.
We receive a `SIGPIPE` if we write to a pipe that points to a crashed process. This in particular happens if the target of a `JSONRPCConnection` has crashed and we try to send it a message.
On Darwin, `DispatchIO` ignores `SIGPIPE` for the pipes handled by it, but that features is not available on Linux.
Instead, globally ignore `SIGPIPE` on Linux to prevent us from crashing if the `JSONRPCConnection`'s target crashes.
Fixes rdar://75580936
This actually addresses the real issue that was ignored earlier about
pipes on Windows. The FileHandle cannot provide a non-owning file
descriptor (the returned file descriptor would need to be explicitly
`_close`'d by the receiver). Foundation now vends a `_handle` accessor
to the OS primitive handle. Use this to create the dispatch loop for
messaging. We now create the JSONRPCConnection from handles on Windows
which actually should help enable running some of the tests on Windows
as well.
* Replaces two logs since those don't result in response messages anymore
* Adds a func for sending messages synchronously, to get the two messages out before fatalError
This adjusts the use of Dispatch to build on Windows. Windows does not
provide `stdout_fileno` and `stderr_fileno`. However, it is possible to
use `fileno` to get the associated fileno from the descriptor.
Dispatch on Windows does not deal with fd's but rather with handles.
Convert the file descriptor to a handle and pass that off to dispatch.
The handle is a non-owning reference, and should not be closed.
Fortunately, dispatch does not close the handle when the DispatchIO is
closed.
Specifically, we care that all outstanding **writes** are finished
before we call the close handler, because otherwise we may (a) send
corrupted output during shutdown, or (b) drop notifications and replies
sent during the shutdown process. The former is a potential issue for
clients that are not robust about parse failures, and the latter is an
issue for reproducibility and robustness during testing/debugging - in
particular, we have some integration tests that send data without
waiting for individual replies and they need to finish outstanding
replies before exiting.
rdar://60159448
Since the connection and message handler have a reciprocal need to know
about each other, move the closeHandler so that it has the opportunity
to call a method on the message handler if desired.
We will be able to split the LSP modules off later. These LSP modules
will provide the ability to write custom LSP servers and clients in
Swift. The sourcekit-lsp repository will build on top of this new
package to provide an LSP implementation that creates a language server
for Swift and C-based-languages.
This code was doing a sync dispatch to a queue it was already running
on. Instead, go through the same path as other close calls. This would
hang, or if you're lucky crash when dispatch detects the issue.
We implicitly close the connection when the input file descriptor is
closed, or if it has an error. But we also close things down explicitly
when we see the exit notification. We need to protect against a race
triggering us to call the close handler multiple times. This was
resulting in `sourcekit-lsp` racing to call `exit(0)`, which triggered a
double-free and abort on Linux.
When testing the sourcekit-lsp binary, it is handy to be able to force
requests to be handled synchronously. This only affects the protocol
layer, not the implementation. This option is hidden from the help text
and should only be used for testing/debugging.
