Since all parsing contexts need a reference to the current token of the
parser, we should pass the token reference to the root context. Therefore, the derived
sub-contexts can just copy it while being spawned.
For very large source files, the parser's syntax map---which contains a
very large number of TrivialLists---was taking an inordinate amount of
memory due to the inefficiency of std::deque. Specifically, a
std::deque containing just one trivial element would allocate 4k of
memory. With the ~120MB SIL output of one of the parse_stdlib tests,
these std::deques would add up to > 6GB of memory, most of which is
wasted.
Replacing the std::deque with a std::vector knocks the memory required
for one of the parse_stdlib tests from > 8GB down closer to 2 GB. The
parser's syntax map is still large (e.g., a 512MB allocation for the
overall vector plus a few hundred MB of raw-syntax data), but not
prohibitively so.
Part of rdar://problem/34771322.
For very large source files, the parser's syntax map---which contains a
very large number of TrivialLists---was taking an inordinate amount of
memory due to the inefficiency of std::deque. Specifically, a
std::deque containing just one trivial element would allocate 4k of
memory. With the ~120MB SIL output of one of the parse_stdlib tests,
these std::deques would add up to > 6GB of memory, most of which is
wasted.
Replacing the std::deque with a std::vector knocks the memory required
for one of the parse_stdlib tests from > 8GB down closer to 2 GB. The
parser's syntax map is still large (e.g., a 512MB allocation for the
overall vector plus a few hundred MB of raw-syntax data), but not
prohibitively so.
Part of rdar://problem/34771322.
This patch allows Parser to generate a refined token stream to satisfy tooling's need. For syntax coloring, token stream from lexer is insufficient because (1) we have contextual keywords like get and set; (2) we may allow keywords to be used as argument labels and names; and (3) we need to split tokens like "==<". In this patch, these refinements are directly fulfilled through parsing without additional heuristics. The refined token vector is optionally saved in SourceFile instance.
I forgot to reset the macro parameters after converting them to
varargs, which didn't get caught running PR testing.
This patch ensures they're all the same.
* Generate libSyntax API
This patch removes the hand-rolled libSyntax API and replaces it with an
API that's entirely automatically generated. This means the API is
guaranteed to be internally stylistically and functionally consistent.
Previously, users of TokenSyntax would always deal with RC<TokenSyntax>
which is a subclass of RawSyntax. Instead, provide TokenSyntax as a
fully-realized Syntax node, that will always exist as a leaf in the
Syntax tree.
This hides the implementation detail of RawSyntax and SyntaxData
completely from clients of libSyntax, and paves the way for future
generation of Syntax nodes.
This patch will allow for serialization of RawSyntax trees to JSON,
which allows external tools to get access to a RawSyntax tree.
This also adds a hook into swift-syntax-test to generate JSON for a
given Swift source file, which will be used in tests in subsequent
commits.
* [Parse] Refactored internal structure of Tokens.def and documented usage.
Added a level of structure to the macro definitions to allow Swift
keywords to be cleanly accessed separately from SIL and Swift keywords
together. Documented structure and usage.
* [Parse] Made use of new guarantees and abstractions in Tokens.def
Used guarantees about undefining macros after import and new
SWIFT_KEYWORD abstraction to simplify usage of the Token.def
imports.
* Gardening
This introduces a few unfortunate things because the syntax is awkward.
In particular, the period and following token in \.[a], \.? and \.! are
token sequences that don't appear anywhere else in Swift, and so need
special handling. This is somewhat compounded by \foo.bar.baz possibly
being \(foo).bar.baz or \(foo.bar).baz (parens around the type), and,
furthermore, needing to distinguish \Foo?.bar from \Foo.?bar.
rdar://problem/31724243
The Swift 4 Migrator is invoked through either the driver and frontend
with the -update-code flag.
The basic pipeline in the frontend is:
- Perform some list of syntactic fixes (there are currently none).
- Perform N rounds of sema fix-its on the primary input file, currently
set to 7 based on prior migrator seasons. Right now, this is just set
to take any fix-it suggested by the compiler.
- Emit a replacement map file, a JSON file describing replacements to a
file that Xcode knows how to understand.
Currently, the Migrator maintains a history of migration states along
the way for debugging purposes.
- Add -emit-remap frontend option
This will indicate the EmitRemap frontend action.
- Don't fork to a separte swift-update binary.
This is going to be a mode of the compiler, invoked by the same flags.
- Add -disable-migrator-fixits option
Useful for debugging, this skips the phase in the Migrator that
automatically applies fix-its suggested by the compiler.
- Add -emit-migrated-file-path option
This is used for testing/debugging scenarios. This takes the final
migration state's output text and writes it to the file specified
by this option.
- Add -dump-migration-states-dir
This dumps all of the migration states encountered during a migration
run for a file to the given directory. For example, the compiler
fix-it migration pass dumps the input file, the output file, and the
remap file between the two.
State output has the following naming convention:
${Index}-${MigrationPassName}-${What}.${extension}, such as:
1-FixitMigrationState-Input.swift
rdar://problem/30926261
* Refactor Tuple Type Syntax
This patch:
- Refactors TypeArgumentListSyntax and
TypeArgumentListSyntaxData to use the SyntaxCollection and
SyntaxCollectionData APIs.
- Refactors TupleTypeElementSyntax to own its trailing comma, and
updates the tests accordingly.
- Provides an infrastructure for promoting types to use
the SyntaxCollection APIs
* Addressed comments.
* Renamed makeBlankTypeArgumentList()
* Update makeTupleType
* Changed makeTupleType to take an element list.
* Updated comment.
* Improved API for creating TupleTypeElementListSyntax'es
* Added round-trip test
* Removed last TypeArgumentList holdovers.
* Fixed round-trip test invocation
This operator[] relies on having the cache contain the right number of
elements, and each element be initialized to nullptr, which was only
happening if NDEBUG was not defined.
rdar://problem/30832595
