This patch allows Parser to generate a refined token stream to satisfy tooling's need. For syntax coloring, token stream from lexer is insufficient because (1) we have contextual keywords like get and set; (2) we may allow keywords to be used as argument labels and names; and (3) we need to split tokens like "==<". In this patch, these refinements are directly fulfilled through parsing without additional heuristics. The refined token vector is optionally saved in SourceFile instance.
This adds support for SE-0168, multi-line string literals.
Extend the lexer to recognize the new literals. Test cases added.
There are still areas for future diagnostic improvement, such as fixits and notes as to why a multi-line string literal will be malformed. Multi-line literals are explicitly forbidden inside of string interpolation, though this may be relaxed in the future.
Add an option to the lexer to go back and get a list of "full"
tokens, which include their leading and trailing trivia, which
we can index into from SourceLocs in the current AST.
This starts the Syntax sublibrary, which will support structured
editing APIs. Some skeleton support and basic implementations are
in place for types and generics in the grammar. Yes, it's slightly
redundant with what we have right now. lib/AST conflates syntax
and semantics in the same place(s); this is a first step in changing
that to separate the two concepts for clarity and also to get closer
to incremental parsing and type-checking. The goal is to eventually
extract all of the syntactic information from lib/AST and change that
to be more of a semantic/symbolic model.
Stub out a Semantics manager. This ought to eventually be used as a hub
for encapsulating lazily computed semantic information for syntax nodes.
For the time being, it can serve as a temporary place for mapping from
Syntax nodes to semantically full lib/AST nodes.
This is still in a molten state - don't get too close, wear appropriate
proximity suits, etc.
Store leading a trailing "trivia" around a token, such as whitespace,
comments, doc comments, and escaping backticks. These are syntactically
important for preserving formatting when printing ASTs but don't
semantically affect the program.
Tokens take all trailing trivia up to, but not including, the next
newline. This is important to maintain checks that statements without
semicolon separators start on a new line, among other things.
Trivia are now data attached to the ends of tokens, not tokens
themselves.
Create a new Syntax sublibrary for upcoming immutable, persistent,
thread-safe ASTs, which will contain only the syntactic information
about source structure, as well as for generating new source code, and
structural editing. Proactively move swift::Token into there.
Since this patch is getting a bit large, a token fuzzer which checks
for round-trip equivlence with the workflow:
fuzzer => token stream => file1
=> Lexer => token stream => file 2 => diff(file1, file2)
Will arrive in a subsequent commit.
This patch does not change the grammar.
Allow all keywords except for parameter introducers (var/let/inout) to
be argument labels when declaring or calling a
function/initializer/subscript, e.g., this
func touchesMatching(phase: NSTouchPhase, `in` view: NSView?) -> Set<NSTouch>
can now be expressed as
func touchesMatching(phase: NSTouchPhase, in view: NSView?) -> Set<NSTouch>
and the call goes from
event.touchesMatching(phase, `in`: view)
to
event.touchesMatching(phase, in: view)
Fixes [SR-344](https://bugs.swift.org/browse/SR-344) /
rdar://problem/22415674.
Sink the actual logic for omitting needless words way down into
Basic, so we can re-use it elsewhere. Tie the Clang importer into that
logic, mapping Clang types down to strings appropriately. NFC
Swift SVN r31233
This way they can be used from other projects, like LLDB. The downside
is we now have to make sure the header is included consistently in all
the places we care about, but I think in practice that won't be a problem,
especially not with tests.
rdar://problem/22240127
Swift SVN r31173
The API review list found it confusing that if #os() and #if os() looked so similar, so
change the availability checking query to be spelled #available:
if #available(iOS >= 9.0, *) {
...
}
Swift SVN r26995
- Strength reduce isAtStartOfBindingName() to just check for
identifier or _ and inline into its two callers.
- Rename Token::isIdentifierOrNone to isIdentifierOrUnderscore.
- Teach InVarOrLetPattern about matching patterns, so that the
parser knows when it is parsing an expression as a matching
pattern but is not yet inside a let/var pattern.
- Use newfound knowledge of matching patterns to refine handling
of unexpected let/var when parsing an expression, but not in a
pattern context, slightly improving QoI in invalid cases.
Swift SVN r26172
This patch adds a new 'pound_os' token, a new case for it in parseExprPostfix, and parsing of platform version constraints, e.g., OSX >= 10.10.
It also adds enough type checking and SILGen to get the parsing tests to run without triggering "Unimplemented" assertions.
Swift SVN r21865
eliminating the @'s from them when used on func's. This is progress towards
<rdar://problem/17527000> change operator declarations from "operator prefix" to "prefix operator" & make operator a keyword
This also consolidates rejection of custom operator definitions into one
place and makes it consistent, and adds postfix "?" to the list of rejected
operators.
This also changes the demangler to demangle weak/inout/postfix and related things
without the @.
Swift SVN r19929
attribute is a "modifier" of a decl, not an "attribute" and thus shouldn't
be spelt with an @ sign. Teach the parser to parse "@foo" but reject it with
a nice diagnostic and a fixit if "foo" is a decl modifier.
Move 'dynamic' over to this (since it simplifies some code), and switch the
@optional and @required attributes to be declmodifiers (eliminating their @'s).
Swift SVN r19787
Parse 'dynamic' as a contextual keyword, and check that it's applied only to ObjC-compatible members of classes. We don't handle inheritance of the attribute yet.
Swift SVN r19781
- Change the parser to accept "objc" without an @ sign as a contextual
keyword, including the dance to handle the general parenthesized case.
- Update all comments to refer to "objc" instead of "@objc".
- Update all diagnostics accordingly.
- Update all tests that fail due to the diagnostics change.
- Switch the stdlib to use the new syntax.
This does not switch all tests to use the new syntax, nor does it warn about
the old syntax yet. That will be forthcoming. Also, this needs a bit of
refactoring, which will be coming up.
Swift SVN r19555
backtracking, because we didn't restore the lexer state to before the comment, preventing
it from being attached to the token that followed it after backtracking was restored.
This is super obscure right now, but causes two tests to fail with my forthcoming patch,
lets nip this in the bud.
Swift SVN r19553
This patch extends the syntax with a new #line directive that is inspired
by the homonymous CPP directive. It can be specified in all locations a #if
is legal (Stmt, Decl).
Semantics
---------
#line 42 "file.swift"
This makes diagnostics and debug information behave as if the subsequent
lines came from file.swift+42.
#line // without arguments
This switches back to the main source file and the switches back to the
normal line numbering. Any previous #line directives will result in gaps
in the main file.
Rationale
---------
LLDB and the REPL need this for making expressions that are entered into
the expression evaluator or REPL debugable. For more info see
<rdar://problem/17441710> Need #line directive or something similar so we can enhance the debugging of expressions and REPL
Also, I believe the stdlib would benefit from this and it would allow us
to get rid of the line-directive wrapper script.
Swift SVN r19384
Turns out 64 Kib is closer than it seems -- I hit this limit with an
autogenerated source file where a part of it was commented out, and the
compiler crashed in an obscure way.
Swift SVN r19339
used on init decls, with the same semantics as "-> Self". Switch the ast
printer, and fixits to use it.
As driveby's, simplify verification of contextual keywords in declparsing,
and rename parseConstructor/Destructor to parseInit/Deinit.
Swift SVN r17356
This is part of <rdar://problem/16782966> make weak and unowned be context sensitive keywords
The part still missing is where we ban the attribute with a fixit to use the
non-attribute syntax.
Swift SVN r17235
