It turns out that the bitpacked Commons struct is actually fairly expensive because the CPU needs to apply bitmasks to fetch the IsToken and Presence flag. We've got padding space available, so we might as well properly align these boolean flags.
Also on a source level, replace a couple of bit-restricted unsigned fields by their representing type (e.g. SyntaxKind).
Finally, we can pull out the common bits to RawSyntax and have the Bits union only contain the token- or layout-specific fields.
This also allows us to initialise these fields in the constructor's initialiser list (instead of in the initialiser body).
Lastly, change copyToArenaIfNecessary to work on a char *& and length, which allows us to initialise leading/trailing trivia/token text in the initialiser list and adjust if necessary later.
By now ParsedRawSyntaxNode does not have any knowledge about deferred
node data anymore, which frees up SyntaxParseActions (and, in
particular its sublass SyntaxTreeCreator) to perform optimisations to
more efficiently create and record deferred nodes.
This is again a transitional state before SyntaxParsingContext hands
the responsibility over to SyntaxTreeCreator and from there to
SyntaxParseActions.
Previously it was backtracking for the duration of the whole property body which was preventing re-use of previously parsed nodes for incremental re-parsing.
By convention, most structs and classes in the Swift compiler include a `dump()` method which prints debugging information. This method is meant to be called only from the debugger, but this means they’re often unused and may be eliminated from optimized binaries. On the other hand, some parts of the compiler call `dump()` methods directly despite them being intended as a pure debugging aid. clang supports attributes which can be used to avoid these problems, but they’re used very inconsistently across the compiler.
This commit adds `SWIFT_DEBUG_DUMP` and `SWIFT_DEBUG_DUMPER(<name>(<params>))` macros to declare `dump()` methods with the appropriate set of attributes and adopts this macro throughout the frontend. It does not pervasively adopt this macro in SILGen, SILOptimizer, or IRGen; these components use `dump()` methods in a different way where they’re frequently called from debugging code. Nor does it adopt it in runtime components like swiftRuntime and swiftReflection, because I’m a bit worried about size.
Despite the large number of files and lines affected, this change is NFC.
The type deduction here will copy the returned ParsedTypeSyntax which
is no longer copy constructible. Use a reference to hold the result
instead which should fix the build on Windows. The returned value of
the `popToken` returns a `ParsedTokenSyntax` which is no longer copy
constructible. Explicitly move the value.
So that we can easily detect 'ParsedSyntaxNode' leaking. When it's
moved, the original node become "null" node. In the destructor of
'ParsedSyntaxNode', assert the node is not "recorded" node.
Instead of creating the AST directly in the parser (and libSyntax or
SwiftSyntax via SyntaxParsingContext), make Parser to explicitly create
a tree of ParsedSyntaxNodes. Their OpaqueSyntaxNodes can be either
libSyntax or SwiftSyntax. If AST is needed, it can be generated from the
libSyntax tree.
SyntaxParseActions::recordToken() et al. may return the same pointer
value for different nodes (e.g. `nullptr`). So we cannot use DenseMap to
associate the node from the explicit syntax parsing actions to libSyntax
node. Instead, use a structure that wraps them.
Instead of creating the AST directly in the parser (and libSyntax or
SwiftSyntax via SyntaxParsingContext), make Parser to explicitly create
a tree of ParsedSyntaxNodes. Their OpaqueSyntaxNodes can be either
libSyntax or SwiftSyntax. If AST is needed, it can be generated from the
libSyntax tree.
This eliminates the overhead of ParsedRawSyntaxNode needing to do memory management.
If ParsedRawSyntaxNode needs to point to some data the memory is allocated from a bump allocator.
There are also some improvements on how the ParsedSyntaxBuilders work.
Doing a "direct ParsedSyntaxRecorder::record[some syntax]" call from the parser is not a good idea due to possibility
of being in a backtracking context when the call is made. Replace them with "ParsedSyntaxRecorder::make[some syntax]"
which will implicitly check for backtracking and create a recorded or deferred node accordingly.
Instead of creating syntax nodes directly, modify the parser to invoke an abstract interface 'SyntaxParseActions' while it is parsing the source code.
This decouples the act of parsing from the act of forming a syntax tree representation.
'SyntaxTreeCreator' is an implementation of SyntaxParseActions that handles the logic of creating a syntax tree.
To enforce the layering separation of parsing and syntax tree creation, a static library swiftSyntaxParse is introduced to compose the two.
This decoupling is important for introducing a syntax parser library for SwiftSyntax to directly access parsing.
Instead of creating multiple CodeBlockItemList nodes, that need to get merged and discarded later on, do this:
* Ensure for libSyntax parsing that we parse the whole file
* Create top-level CodeBlockItem nodes that we just directly wrap with a single CodeBlockItemList node at the end
The importance of this change will become more obvious later on when we'll decouple syntax parsing from the formation of libSyntax tree nodes.
This allows an elegant design in which we can still allocate RawSyntax
nodes using a bump allocator but are able to automatically free that
buffer once the last RawSyntax node within that buffer is freed.
This also resolves a memory leak of RawSyntax nodes that was caused by
ParserUnit not freeing its underlying ASTContext.
To enhance the error-recovery of syntax parsing, this patch allows the
parser to synthesize missing nodes to satisfy the requirement of a
syntax node under parsing. As proof-of-concept, we synthesize r-braces
for function body to avoid regressing a function decl to an unknown
decl.
