SourceKit-LSP currently parses the XML comment to generate Markdown again but round-tripping a (probably markdown) doc comment to XML to Markdown is lossy in many cases and unnecessary work. Include the comment as it is spelled in source in the cursor info response so that sourcekit-lsp can display it.
Part of rdar://120685874
This adjusts the tests for the difference between line endings on
different platforms. Windows uses CRLF while most Unicies use LF. This
was exposed during the update to the new LLVM snapshot.
Store leading a trailing "trivia" around a token, such as whitespace,
comments, doc comments, and escaping backticks. These are syntactically
important for preserving formatting when printing ASTs but don't
semantically affect the program.
Tokens take all trailing trivia up to, but not including, the next
newline. This is important to maintain checks that statements without
semicolon separators start on a new line, among other things.
Trivia are now data attached to the ends of tokens, not tokens
themselves.
Create a new Syntax sublibrary for upcoming immutable, persistent,
thread-safe ASTs, which will contain only the syntactic information
about source structure, as well as for generating new source code, and
structural editing. Proactively move swift::Token into there.
Since this patch is getting a bit large, a token fuzzer which checks
for round-trip equivlence with the workflow:
fuzzer => token stream => file1
=> Lexer => token stream => file 2 => diff(file1, file2)
Will arrive in a subsequent commit.
This patch does not change the grammar.
This makes it more clear what the response string contains, plus you can copy the
printed request directly for passing to sourcekitd_request_create_from_yaml.
The code goes into its own sub-tree under 'tools' but tests go under 'test',
so that running 'check-swift' will also run all the SourceKit tests.
SourceKit is disabled on non-darwin platforms.
