Re-core the "lexical" token stream on top of SwiftSyntax and call into the parser to (re-)generate these "lexical tokens" when the document changes. Leave the semantic tokens alone since they take some non-zero amount of name lookup to fully resolve references to fully replicate.
This is an implementation of LSP's semantic tokens for Swift. Both
lexical and semantic tokens are provided by using the syntaxmap and the
semantic annotations provided as part of SourceKit's open responses and
document update notifications.
While lexical tokens are parsed and stored in the DocumentManager
synchronously, semantic tokens are provided asynchronously. If an edit
occurs, tokens are automatically shifted by the DocumentManager. This is
especially relevant for lexical tokens, which are updated in deltas.
In addition, unit tests are added that assert that both lexical and
semantic tokens are provided and shifted correctly upon edits.
