Currently, when creating a `RawSyntax` layout node, the `RawSyntax` constructor needs to iterate over all child nodes to
a) sum up their sub node count
b) add their arena as a child arena of the new node's arena
But we are already iterating over all child nodes in every place that calls these constructors. So instead of looping twice, we can perform the above operations in the loop that already exists and pass the parameters to the `RawSyntax` constructor, which spees up `RawSyntax` node creation.
To ensure the integrity of the `RawSyntax` tree, the passed in values are still validated in release builds.
Instead, only reference count the SyntaxArena that the RawSyntax nodes
live in. The user of RawSyntax nodes must guarantee that the SyntaxArena
stays alive as long as the RawSyntax nodes are being accessed.
During parse time, the SyntaxTreeCreator holds on to the SyntaxArena
in which it creates RawSyntax nodes. When inspecting a syntax tree,
the root SyntaxData node keeps the SyntaxArena alive. The change should
be mostly invisible to the users of the public libSyntax API.
This change significantly decreases the overall reference-counting
overhead. Since we were not able to free individual RawSyntax nodes
anyway, performing the reference-counting on the level of the
SyntaxArena feels natural.
Do the same thing that we are already doing for trivia: Since RawSyntax
nodes always live inside a SyntaxArena, we don't need to tail-allocate
an OwnedString to store the token's text. Instead we can just copy it
to the SyntaxArena. If we copy the entire source buffer to the syntax
arena at the start of parsing, this means that no more copies are
required later on. Plus we also avoid ref-counting the OwnedString which
should also increase performance.
This way, we will later be able to store additional information about
the node inside the same arena with a guarantee that they will always be
alive as long as the node is alive.
These additional information will include
a) the token's text (which can be a StringRef into a copy of the source
code that lives inside the SyntaxArena)
b) the token's unparsed trivia, which can be decomposed into pieces when
needed.
To represent a type with code completion.
type? '.'? <code-completion-token>
This is "parser only" node which is not exposed to SwiftSyntax.
Using this, defer to set the parsed type to code-completion callbacks.
Instead of creating syntax nodes directly, modify the parser to invoke an abstract interface 'SyntaxParseActions' while it is parsing the source code.
This decouples the act of parsing from the act of forming a syntax tree representation.
'SyntaxTreeCreator' is an implementation of SyntaxParseActions that handles the logic of creating a syntax tree.
To enforce the layering separation of parsing and syntax tree creation, a static library swiftSyntaxParse is introduced to compose the two.
This decoupling is important for introducing a syntax parser library for SwiftSyntax to directly access parsing.
This allows an elegant design in which we can still allocate RawSyntax
nodes using a bump allocator but are able to automatically free that
buffer once the last RawSyntax node within that buffer is freed.
This also resolves a memory leak of RawSyntax nodes that was caused by
ParserUnit not freeing its underlying ASTContext.
We cannot use unowned strings for token texts of incrementally parsed
syntax trees since the source buffer to which reused nodes refer will
have been freed for reused nodes. Always copying the token text whenever
OwnedString is passed is too expensive. A reference counted copy of the
string allows us to keep the token's string alive across incremental
parses while eliminating unnecessary copies.
Introduced SyntaxArena for managing memory and cache.
SyntaxArena holds BumpPtrAllocator as a allocation storage.
RawSyntax is now able to be constructed with normal heap allocation, or
by SyntaxArena. RawSyntax has ManualMemory flag which indicates it's managed by
SyntaxArena. If the flag is true, its Retain()/Release() is no-op thus it's
never destructed by IntrusiveRefCntPtr.
This speedups the memory allocation for RawSyntax.
Also, in Syntax parsing, "token" RawSyntax is reused if:
a) It's not string literal with >16 length; and
b) It doesn't contain random text trivia (e.g. comment).
This reduces the overall allocation cost.
Segments list in interpolated string literal accepts two syntax kinds as
elements: StringSegment and ExpressionSegment. This patch generates
factory methods to reject other kinds.
This patch adds a python function to syntax node gyb support called
"check_child_condition". Given a child's definition, this function
generate a C++ closure to check whether a given syntax node can satisfy
the condition of the child node. This function recursively generates code
for node choices too, therefore we don't need to hard code the
condition checking for node choices.
Some structures of syntax nodes can have children choices, e.g. a
dictionary expression can either contain a single ':' token or a list of
key-value pairs.
This patch gives the existing code generation infrastructure a way to
specify such node choices. Node choices are specified under a child
declaration with two constraints: a choice cannot be declared as
optional, and a choice cannot have further recursive choices.
Since we don't have too many node structures with choices, part of the
SyntaxFactory code for these nodes is manually typed.
This patch also teaches AccessorBlock to use node choices.
to check text-choices of a token child.
This allows us to reject the creation of a syntax node if one of its token
syntax children doesn't follow the required textual choices, e.g.
modifiers.
To construct struct syntax, this patch first specialized type
inheritance clause. For protocol's class requirement, we currently
treat it as an unknown type.
This patch also teaches SyntaxParsingContext to collect syntax nodes
from back in place. This is useful to squash multiple decl modifiers
for declarations like function. This is not used for struct declaration
because only accessibility modifier is allowed.
This patch also performs minor refactoring to align syntax parsing
context with the right scope. We start to support the generic clauses
because they are necessary pieces to construct struct or
function syntax node.
* Generate libSyntax API
This patch removes the hand-rolled libSyntax API and replaces it with an
API that's entirely automatically generated. This means the API is
guaranteed to be internally stylistically and functionally consistent.
