If an edit didn't intersect with an existing highlighted tokens but caused a
later highlighted token to change kind, syntax highlighting would be lost
between the edit and that token.
Resolves rdar://problem/33463141.
We still need to adjust the affected range to the line boundaries and return all
tokens on the line when there are no new tokens, as the client will clear all
tokens on that line in its copy of the syntax map leaving the other tokens
unhighlighted. We also need to extend the affected range to include the ranges
of the mismatched tokens from the previous syntaxmap, so their highlighting will
be cleared.
Also add more comments to better document the new syntax map structure and
behaviour.
This patch changes the syntax map data structure it uses to be offset based
rather than line/col based in order to avoid calling getLineAndColumn for the
start and end offset of every token. This removes the 30% of time spent in
getLineAndColumn for this request in large files (rdar://problem/28965123).
The logic for returning the affected range and the token ranges to highlight
following an edit also made several assumptions that no longer hold. This
patch changes it to compare the syntax maps from before and after the edit,
find the first mismtaching tokens from the start and end of the syntax maps
and return the tokens in that range (adjusted to line boundaries). This fixes
syntax highlighting issues with interpolated multi-line strings
(rdar://problem/32148117) and block comments.
With the above changes the per-keystroke time spent for syntax highlighting
(with sematic info disabled) dropped from ~80ms to just under 50ms for a
12KLOC file.
Multiline strings (and multiline tokens in general) were not well supported by the existing highlighting logic. Edits
on one line can make tokens appear/disappear on previous and later lines, which broke assumptions in the existing
logic, and left odd ranges of source unhighlighted or out of date. This patch accounts for these changes, and also
changes unterminated multiline (and regular strings) to still be highlighted as strings, so the rest of the
file doesn't look like plain text.
Resolves rdar://problem/32148117.
Unlike other prefix operators, unary minus is folded into the
NumberLiteralExpr in the parser. This commit recreates this effect in
the lexer-based syntax map so that token ranges will include the leading
minus.
rdar://problem/20205885
The code goes into its own sub-tree under 'tools' but tests go under 'test',
so that running 'check-swift' will also run all the SourceKit tests.
SourceKit is disabled on non-darwin platforms.
