libSyntax nodes don't maintain absolute source location on each
individual node. Instead, the absolute locations are calculated on
demand with a given root by accumulating the length of all the other
nodes before the target node. This bridging is important for issuing
diagnostics from libSyntax entities.
With the observation that our current implementation of the source
location calculation has multiple bugs, this patch re-implemented this
bridging by using the newly-added syntax visitor. Also, we moved the function
from RawSyntax to Syntax for better visibility.
To test this source location calculation, we added a new action in
swift-syntax-test. This action parses a given file as a
SourceFileSyntax, calculates the absolute location of the
EOF token in the SourceFileSyntax, and dump the buffer from the start
of the input file to the absolute location of the EOF. Finally, we compare
the dump with the original input to ensure they are identical.
This allows the root context to have a separate place to keep track of
the global data that each sub-context can access to, for instance,
SourceFile, DiagnosticEngine, etc.
A string interpolation expression is composed of { OpenQuote, Segments,
CloseQuote }. To represent OpenQuote, CloseQuote and StringSegment, we have to
introduce new token kinds correspondingly.
This patch adds a python function to syntax node gyb support called
"check_child_condition". Given a child's definition, this function
generate a C++ closure to check whether a given syntax node can satisfy
the condition of the child node. This function recursively generates code
for node choices too, therefore we don't need to hard code the
condition checking for node choices.
Some structures of syntax nodes can have children choices, e.g. a
dictionary expression can either contain a single ':' token or a list of
key-value pairs.
This patch gives the existing code generation infrastructure a way to
specify such node choices. Node choices are specified under a child
declaration with two constraints: a choice cannot be declared as
optional, and a choice cannot have further recursive choices.
Since we don't have too many node structures with choices, part of the
SyntaxFactory code for these nodes is manually typed.
This patch also teaches AccessorBlock to use node choices.
To construct struct syntax, this patch first specialized type
inheritance clause. For protocol's class requirement, we currently
treat it as an unknown type.
This patch also teaches SyntaxParsingContext to collect syntax nodes
from back in place. This is useful to squash multiple decl modifiers
for declarations like function. This is not used for struct declaration
because only accessibility modifier is allowed.
Because generic where clause doesn't coerce well to our existing syntax
context kinds, we add a new syntax context kind with this patch called
"Syntax". This context kind indicates that when error occurs, the
collection of syntax nodes falling into the context should be coerced
to UnknownSyntax.
RawTokenSyntax is a derived class from RawSyntax that is reference
counted with its own destructor function registered. Unfortunately, the destructor
function of RawSyntax is non-virtual before this patch. This means when reference counter
releases a pointer of RawSyntax, it won't clean-up the additional stuff in RawTokenSyntax.
Avoid heap-allocated memory for syntax parsing context.
Add more assertions to ensure syntax nodes are created only at the top of context stack.
Allow syntax parsing context to delay the specifying of context kind and target syntax kind.
This commit also adds ArrayExpr and DictionaryExpr to the libSyntax nodes
family. Also, it refactors the original parser code for these two
expressions to better fit to the design of SyntaxParsingContext.
This commit has also fixed two crashers.
This commit teaches parser to parse two libSyntax nodes: FunctionCallArgument and
FunctionCallArgumentList. Along with the change, some libSyntax parsing infrastructure changes
as well: (1) parser doesn't directly insert token into the buffer for libSyntax node creation;
instead, when creating a simple libSyntax node like integer literal expression, parser should indicate the location of the last token in the node; (2) implicit libSyntax nodes like empty
statement list must contain a source location indicating where the implicit nodes should appear
(immediately before the token at the given location).
This commit teaches parser to generate code block syntax node. As a support for this,
SyntaxParsingContext can be created by a single syntax kind, indicating the whole context
should be parsed into a node of that given syntax. Another change is to bridge created syntax
node with the given context kind. For instance, if a statement context results into an expression
node, the expression node will be bridged to a statement by wrapping it with a ExpressionStmt
node.
* Re-apply "libSyntax: Ensure round-trip printing when we build syntax tree from parser incrementally. (#12709)"
* Re-apply "libSyntax: Root parsing context should hold a reference to the current token in the parser, NFC."
* Re-apply "libSyntax: avoid copying token text when lexing token syntax nodes, NFC. (#12723)"
* Actually fix the container-overflow issue.
Since all parsing contexts need a reference to the current token of the
parser, we should pass the token reference to the root context. Therefore, the derived
sub-contexts can just copy it while being spawned.
For very large source files, the parser's syntax map---which contains a
very large number of TrivialLists---was taking an inordinate amount of
memory due to the inefficiency of std::deque. Specifically, a
std::deque containing just one trivial element would allocate 4k of
memory. With the ~120MB SIL output of one of the parse_stdlib tests,
these std::deques would add up to > 6GB of memory, most of which is
wasted.
Replacing the std::deque with a std::vector knocks the memory required
for one of the parse_stdlib tests from > 8GB down closer to 2 GB. The
parser's syntax map is still large (e.g., a 512MB allocation for the
overall vector plus a few hundred MB of raw-syntax data), but not
prohibitively so.
Part of rdar://problem/34771322.
For very large source files, the parser's syntax map---which contains a
very large number of TrivialLists---was taking an inordinate amount of
memory due to the inefficiency of std::deque. Specifically, a
std::deque containing just one trivial element would allocate 4k of
memory. With the ~120MB SIL output of one of the parse_stdlib tests,
these std::deques would add up to > 6GB of memory, most of which is
wasted.
Replacing the std::deque with a std::vector knocks the memory required
for one of the parse_stdlib tests from > 8GB down closer to 2 GB. The
parser's syntax map is still large (e.g., a 512MB allocation for the
overall vector plus a few hundred MB of raw-syntax data), but not
prohibitively so.
Part of rdar://problem/34771322.
