Store leading a trailing "trivia" around a token, such as whitespace,
comments, doc comments, and escaping backticks. These are syntactically
important for preserving formatting when printing ASTs but don't
semantically affect the program.
Tokens take all trailing trivia up to, but not including, the next
newline. This is important to maintain checks that statements without
semicolon separators start on a new line, among other things.
Trivia are now data attached to the ends of tokens, not tokens
themselves.
Create a new Syntax sublibrary for upcoming immutable, persistent,
thread-safe ASTs, which will contain only the syntactic information
about source structure, as well as for generating new source code, and
structural editing. Proactively move swift::Token into there.
Since this patch is getting a bit large, a token fuzzer which checks
for round-trip equivlence with the workflow:
fuzzer => token stream => file1
=> Lexer => token stream => file 2 => diff(file1, file2)
Will arrive in a subsequent commit.
This patch does not change the grammar.
Most keywords can be used as argument labels. When they are they shouldn't be
colored as a keyword. This patch fixes three issues with this:
- keywords as local argument labels (where an external label was also present)
were still colored as keywords
- 'let', 'var', and 'inout' cannot be used as argument labels, but were colored
as identifiers rather than keywords when used like one
- the check to see if a keyword token is being used as an argument label wasn't
quite restrictive enough, e.g. treating the 'let's in 'case (let x, let y):' as
identifiers.
This gave a roughly 40-45% improvement in sourcekitd's incremental
syntax-only parse time in files with a lot of doc comments (test case
was ~6000 lines, with ~780 lines being doc comments). This is on the
critical path for every edit.
While there were a few smaller improvements we could have made to the
original code, ultimately std::regex is slow, and it was better to just
use a custom parser for these simple patterns.
rdar://problem/28809397
[SyntaxModel] When searching URLs in doc comments, reduce the number of protocol name comparisons by looking ahead more characters, NFC. rdar://28298506
Searching URL in doc comments can be expensive. We used to look for
every colon as an indicator of potential URLs. However, this is not
efficient enough. Suggested by Ben, we further divide protocols into
categories so that most protocols can use "://" as an indicator of its
existence.
Not sure whether this is enough to close the radar, but I believe it is
a valuable performance improvement anyway.
Keywords like 'let' can serve as argument labels. When they do so, we should
highlight them as identifiers instead of keywords. However, the check for this
situation seems overly lenient so that when 'let', 'var' appear in conditions
of IfStmt or GuardStmt, they are wrongly highlighted as identifiers too. This
commit strengthens the checking to preserve keywords' identity in these statements.
rdar://28297337
Argument labels are allowed to use keywords, in which case we want to
treat them as identifiers in the syntax map (except for '_'). This
commit moves calculation of that into the original lexing instead of
in the model walker, which makes it much more robust, since the model
walker was only guessing about what was next on the the TokenNodes list.
This fixes a bug where arrays of object literals would only have the
first object correct (the following ones were identifiers), as well as
some incorrect cases where we treated keywords as identifiers.
rdar://problem/27726422
What I've implemented here deviates from the current proposal text
in the following ways:
- I had to introduce a FunctionArrowPrecedence to capture the parsing
of -> in expression contexts.
- I found it convenient to continue to model the assignment property
explicitly.
- The comparison and casting operators have historically been
non-associative; I have chosen to preserve that, since I don't
think this proposal intended to change it.
- This uses the precedence group names and higherThan/lowerThan
as agreed in discussion.
When creating the token map for highlighting, there is some glitchy
behavior that needed to be cleaned up:
- New image literal syntax wasn't getting highlighted as an object
literal keyword.
- Other #-prefixed keywords had no data
- #line would appear as a build config ("brown") at the start of a line
- #available showed up as a build config. However, this should be a keyword
because it is a runtime check, not a build configuration.
Add some test cases for these.
Fixes:
rdar://problem/26451674
* Implement the majority of parsing support for SE-0039.
* Parse old object literals names using new syntax and provide FixIt.
For example, parse "#Image(imageLiteral:...)" and provide a FixIt to
change it to "#imageLiteral(resourceName:...)". Now we see something like:
test.swift:4:9: error: '#Image' has been renamed to '#imageLiteral
var y = #Image(imageLiteral: "image.jpg")
^~~~~~ ~~~~~~~~~~~~
#imageLiteral resourceName
Handling the old syntax, and providing a FixIt for that, will be handled in a separate
commit.
Needs tests. Will be provided in later commit once full parsing support is done.
* Add back pieces of syntax map for object literals.
* Add parsing support for old object literal syntax.
... and provide fixits to new syntax.
Full tests to come in later commit.
* Improve parsing of invalid object literals with old syntax.
* Do not include bracket in code completion results.
* Remove defunct code in SyntaxModel.
* Add tests for migration fixits.
* Add literals to code completion overload tests.
@akyrtzi told me this should be fine.
* Clean up response tests not to include full paths.
* Further adjust offsets.
* Mark initializer for _ColorLiteralConvertible in UIKit as @nonobjc.
* Put attribute in the correct place.
Unlike other prefix operators, unary minus is folded into the
NumberLiteralExpr in the parser. This commit recreates this effect in
the lexer-based syntax map so that token ranges will include the leading
minus.
rdar://problem/20205885
Basic implementatation of SE-0021, naming functions with argument
labels. Handle parsing of compound function names in various
unqualified-identifier productions, updating the AST representation of
various expressions from Identifiers to DeclNames. The result doesn't
capture all of the source locations we want; more on that later.
As part of this, remove the parsing code for the "selector-style"
method names, since we now have a replacement. The feature was never
publicized and doesn't make sense in Swift, so zap it outright.
Fix spurious docs warning that @in and @in_guaranteed should be ``fn``.
Add ``#ifdef SWIFT_OBJC_INTEROP`` to silence a -Wunused-function
on linux since that function is only used from within that #ifdef
elsewhere.
Fix three -Wunused-function warnings on linux.
Fix two -Wunreachable-code warnings on linux dealing with
SWIFT_HAVE_WORKING_STD_REGEX.
