Add an option to the lexer to go back and get a list of "full"
tokens, which include their leading and trailing trivia, which
we can index into from SourceLocs in the current AST.
This starts the Syntax sublibrary, which will support structured
editing APIs. Some skeleton support and basic implementations are
in place for types and generics in the grammar. Yes, it's slightly
redundant with what we have right now. lib/AST conflates syntax
and semantics in the same place(s); this is a first step in changing
that to separate the two concepts for clarity and also to get closer
to incremental parsing and type-checking. The goal is to eventually
extract all of the syntactic information from lib/AST and change that
to be more of a semantic/symbolic model.
Stub out a Semantics manager. This ought to eventually be used as a hub
for encapsulating lazily computed semantic information for syntax nodes.
For the time being, it can serve as a temporary place for mapping from
Syntax nodes to semantically full lib/AST nodes.
This is still in a molten state - don't get too close, wear appropriate
proximity suits, etc.
Store leading a trailing "trivia" around a token, such as whitespace,
comments, doc comments, and escaping backticks. These are syntactically
important for preserving formatting when printing ASTs but don't
semantically affect the program.
Tokens take all trailing trivia up to, but not including, the next
newline. This is important to maintain checks that statements without
semicolon separators start on a new line, among other things.
Trivia are now data attached to the ends of tokens, not tokens
themselves.
Create a new Syntax sublibrary for upcoming immutable, persistent,
thread-safe ASTs, which will contain only the syntactic information
about source structure, as well as for generating new source code, and
structural editing. Proactively move swift::Token into there.
Since this patch is getting a bit large, a token fuzzer which checks
for round-trip equivlence with the workflow:
fuzzer => token stream => file1
=> Lexer => token stream => file 2 => diff(file1, file2)
Will arrive in a subsequent commit.
This patch does not change the grammar.
Swift parser splits tokens in few cases, but it swift::tokenize(...) does not know
about that. In order to reconstruct token stream as it was seen by the parser,
we need to collect the tokens it decided to split and use this information
in swift::tokenize(...).
Introduce a new "swift" build configuration that guards declarations
and statements with a language version - if the current language version
of the compiler is at least that version, the block will parse as normal.
For inactive blocks, the code will not be parsed an no diagnostics will
be emitted there.
Example:
#if swift(>=2.2)
print("Active")
#else
this code will not parse or emit diagnostics
#endif
https://github.com/apple/swift-evolution/blob/master/proposals/0020-if-swift-version.md
rdar://problem/19823607
This is a WIP to make CompilerVersion more general.
- Rename CompilerVersion to just "Version"
- Make version comparison general and put _compiler_version special logic
with its second version component in a specialized parsing function
- Add a generic version parsing function
Swift SVN r32726
This configuration clause will suppress lex diagnostics and skip parsing
altogether if the code under the clause isn't active - the compiler must
have a repository version greater than or equal to the version given to
_compiler_version.
This option is only meant to be used sparingly and not to track the
Swift *language* version.
Example, if using a compiler versioned 700.0.28:
#if _compiler_version("700.0.23")
print("This code will compile for versions 700.0.23 and later.")
#else
This + code + will + not + be + parsed
#endif
Included are new diagnostics for checking that the version is formatted
correctly and isn't empty.
New tests:
- Compiler version comparison unit tests
- Build configuration diagnostics
- Skipping parsing of code under inactive clauses
rdar://problem/22730282
Swift SVN r32195
I didn't want to rip this logic out wholesale. There is a possibility
the character lexing can be reborn/revisited later, and
disabling it in the parser was easy.
Swift SVN r18102
outside of debugger-support mode. Rip out the existing special-case
code when parsing expr-identifier.
This means that the Lexer needs a LangOptions. Doug and I
talked about just adding that as a field of SourceMgr, but
decided that it was worth it to preserve the possibility of
parsing different dialects in different source files.
By design, the lexer doesn't tokenize fundamentally differently
in different language modes; it might decide something is invalid,
or it might (eventually) use a different token kind for the
same consumed text, but we don't want it deciding to consume more or
less of the stream per token.
Note that SIL mode does make that kind of difference, and that
arguably means that various APIs for tokenizing need to take a
"is SIL mode" flag, but we're getting away with it because we
just don't really care about fidelity of SIL source files.
rdar://14899000
Swift SVN r13896
Also remove the SourceLoc parameter from addNewSourceBuffer(). In llvm::SourceMgr
it is used to indicate textual inclusion, which we don't have in swift.
Swift SVN r10014
Replace uses of it with the newly introduced constructor that accepts a buffer ID.
The StringRef constructor was rather unsafe since it had the implicit requirement that the StringRef
was null-terminated.
Swift SVN r6942
