We were only keeping track of `RawSyntax` node IDs to incrementally transfer a syntax tree via JSON. However, AFAICT the incremental JSON transfer option has been superceeded by `SyntaxParseActions`, which are more efficient.
So, let’s clean up and remove the `RawSyntax` node ID and JSON incremental transfer option.
In places that still need a notion of `RawSyntax` identity (like determining the reused syntax regions), use the `RawSyntax`’s pointer instead of the manually created ID.
In `incr_transfer_round_trip.py` always use the code path that uses the `SyntaxParseActions` and remove the transitional code that was still using the incremental JSON transfer but was never called.
Currently, when creating a `RawSyntax` layout node, the `RawSyntax` constructor needs to iterate over all child nodes to
a) sum up their sub node count
b) add their arena as a child arena of the new node's arena
But we are already iterating over all child nodes in every place that calls these constructors. So instead of looping twice, we can perform the above operations in the loop that already exists and pass the parameters to the `RawSyntax` constructor, which spees up `RawSyntax` node creation.
To ensure the integrity of the `RawSyntax` tree, the passed in values are still validated in release builds.
It turns out that the bitpacked Commons struct is actually fairly expensive because the CPU needs to apply bitmasks to fetch the IsToken and Presence flag. We've got padding space available, so we might as well properly align these boolean flags.
Also on a source level, replace a couple of bit-restricted unsigned fields by their representing type (e.g. SyntaxKind).
Finally, we can pull out the common bits to RawSyntax and have the Bits union only contain the token- or layout-specific fields.
This also allows us to initialise these fields in the constructor's initialiser list (instead of in the initialiser body).
Lastly, change copyToArenaIfNecessary to work on a char *& and length, which allows us to initialise leading/trailing trivia/token text in the initialiser list and adjust if necessary later.
In contrast to SyntaxData, SyntaxDataRef is not memory-safe, but
designed to be fast. In particular, the following guarantees from
SyntaxData are being dropped:
- SyntaxDataRef does not retain the SyntaxArena containing its
RawSyntax. The user of SyntaxDataRef has to provide that guarantee.
However, that's usually pretty easily done by just retaining the
SyntaxArena of the tree's root node.
- The parent of a SyntaxDataRef must outlive the child node. This is
the more tricky constraint, but if a tree is just walked top to
bottom with nodes stored on the stack, this is given by the way the
stack is being unrolled.
These methods are super small and setting up the stack frame etc. takes
up the majority (or at least a significant amount) of their execution
time. So let's inline them.
Instead, only reference count the SyntaxArena that the RawSyntax nodes
live in. The user of RawSyntax nodes must guarantee that the SyntaxArena
stays alive as long as the RawSyntax nodes are being accessed.
During parse time, the SyntaxTreeCreator holds on to the SyntaxArena
in which it creates RawSyntax nodes. When inspecting a syntax tree,
the root SyntaxData node keeps the SyntaxArena alive. The change should
be mostly invisible to the users of the public libSyntax API.
This change significantly decreases the overall reference-counting
overhead. Since we were not able to free individual RawSyntax nodes
anyway, performing the reference-counting on the level of the
SyntaxArena feels natural.
This decreases the size of RawSyntax nodes from 88 to 64 bytes by
- Avoiding some padding by moving RefCount further up
- Limiting the length of tokens and their trivia to 32 bits. We would
hit this limit with files >4GB but we also hit this limit at other
places like the TextLength property in the Common bits.
Do the same thing that we are already doing for trivia: Since RawSyntax
nodes always live inside a SyntaxArena, we don't need to tail-allocate
an OwnedString to store the token's text. Instead we can just copy it
to the SyntaxArena. If we copy the entire source buffer to the syntax
arena at the start of parsing, this means that no more copies are
required later on. Plus we also avoid ref-counting the OwnedString which
should also increase performance.
This way, we will later be able to store additional information about
the node inside the same arena with a guarantee that they will always be
alive as long as the node is alive.
These additional information will include
a) the token's text (which can be a StringRef into a copy of the source
code that lives inside the SyntaxArena)
b) the token's unparsed trivia, which can be decomposed into pieces when
needed.
By convention, most structs and classes in the Swift compiler include a `dump()` method which prints debugging information. This method is meant to be called only from the debugger, but this means they’re often unused and may be eliminated from optimized binaries. On the other hand, some parts of the compiler call `dump()` methods directly despite them being intended as a pure debugging aid. clang supports attributes which can be used to avoid these problems, but they’re used very inconsistently across the compiler.
This commit adds `SWIFT_DEBUG_DUMP` and `SWIFT_DEBUG_DUMPER(<name>(<params>))` macros to declare `dump()` methods with the appropriate set of attributes and adopts this macro throughout the frontend. It does not pervasively adopt this macro in SILGen, SILOptimizer, or IRGen; these components use `dump()` methods in a different way where they’re frequently called from debugging code. Nor does it adopt it in runtime components like swiftRuntime and swiftReflection, because I’m a bit worried about size.
Despite the large number of files and lines affected, this change is NFC.
This allows an elegant design in which we can still allocate RawSyntax
nodes using a bump allocator but are able to automatically free that
buffer once the last RawSyntax node within that buffer is freed.
This also resolves a memory leak of RawSyntax nodes that was caused by
ParserUnit not freeing its underlying ASTContext.
We cannot use unowned strings for token texts of incrementally parsed
syntax trees since the source buffer to which reused nodes refer will
have been freed for reused nodes. Always copying the token text whenever
OwnedString is passed is too expensive. A reference counted copy of the
string allows us to keep the token's string alive across incremental
parses while eliminating unnecessary copies.
Naming the bit-field structs is a significant readability improvement
because it's very clear that you shouldn't touch e.g. Bits.Token
without having checked/asserted that you're in a token case.
The assertions are all in statement context (which was obvious
because the NDEBUG versions all included semicolons), so there's no
reason not to use the traditional `do { } while (false)` trick instead
of a statement-expression.
This also clears up some warnings in atypical build configurations.
Introduced SyntaxArena for managing memory and cache.
SyntaxArena holds BumpPtrAllocator as a allocation storage.
RawSyntax is now able to be constructed with normal heap allocation, or
by SyntaxArena. RawSyntax has ManualMemory flag which indicates it's managed by
SyntaxArena. If the flag is true, its Retain()/Release() is no-op thus it's
never destructed by IntrusiveRefCntPtr.
This speedups the memory allocation for RawSyntax.
Also, in Syntax parsing, "token" RawSyntax is reused if:
a) It's not string literal with >16 length; and
b) It doesn't contain random text trivia (e.g. comment).
This reduces the overall allocation cost.
libSyntax nodes don't maintain absolute source location on each
individual node. Instead, the absolute locations are calculated on
demand with a given root by accumulating the length of all the other
nodes before the target node. This bridging is important for issuing
diagnostics from libSyntax entities.
With the observation that our current implementation of the source
location calculation has multiple bugs, this patch re-implemented this
bridging by using the newly-added syntax visitor. Also, we moved the function
from RawSyntax to Syntax for better visibility.
To test this source location calculation, we added a new action in
swift-syntax-test. This action parses a given file as a
SourceFileSyntax, calculates the absolute location of the
EOF token in the SourceFileSyntax, and dump the buffer from the start
of the input file to the absolute location of the EOF. Finally, we compare
the dump with the original input to ensure they are identical.
RawTokenSyntax is a derived class from RawSyntax that is reference
counted with its own destructor function registered. Unfortunately, the destructor
function of RawSyntax is non-virtual before this patch. This means when reference counter
releases a pointer of RawSyntax, it won't clean-up the additional stuff in RawTokenSyntax.
I forgot to reset the macro parameters after converting them to
varargs, which didn't get caught running PR testing.
This patch ensures they're all the same.
* Generate libSyntax API
This patch removes the hand-rolled libSyntax API and replaces it with an
API that's entirely automatically generated. This means the API is
guaranteed to be internally stylistically and functionally consistent.
Previously, users of TokenSyntax would always deal with RC<TokenSyntax>
which is a subclass of RawSyntax. Instead, provide TokenSyntax as a
fully-realized Syntax node, that will always exist as a leaf in the
Syntax tree.
This hides the implementation detail of RawSyntax and SyntaxData
completely from clients of libSyntax, and paves the way for future
generation of Syntax nodes.
* Refactor Tuple Type Syntax
This patch:
- Refactors TypeArgumentListSyntax and
TypeArgumentListSyntaxData to use the SyntaxCollection and
SyntaxCollectionData APIs.
- Refactors TupleTypeElementSyntax to own its trailing comma, and
updates the tests accordingly.
- Provides an infrastructure for promoting types to use
the SyntaxCollection APIs
* Addressed comments.
* Renamed makeBlankTypeArgumentList()
* Update makeTupleType
* Changed makeTupleType to take an element list.
* Updated comment.
* Improved API for creating TupleTypeElementListSyntax'es
* Added round-trip test
* Removed last TypeArgumentList holdovers.
* Fixed round-trip test invocation
This will make it easier to incrementally implement syntax nodes,
while allowing us to embed nodes that we do know about inside ones
that we don't.
https://bugs.swift.org/browse/SR-4062
A return statement needs something to return, so implement
integer-literal-expression too. This necessarily also forced
UnknownExprSyntax, UnknownStmtSyntax, and UnknownDeclSyntax,
which are stand-in token buckets for when we don't know
how to transform/migrate an AST.
This commit also contains the core function for caching
SyntaxData children. This is highly tricky code, with some
detailed comments in SyntaxData.{h,cpp}. The gist is that
we have to atomically swap in a SyntaxData pointer into the
child field, so we can maintain pointer identity of SyntaxData
nodes, while still being able to cache them internally.
To prove that this works, there is a multithreaded test that
checks that two threads can ask for a child that hasn't been
cached yet without crashing or violating pointer identity.
https://bugs.swift.org/browse/SR-4010
Add an option to the lexer to go back and get a list of "full"
tokens, which include their leading and trailing trivia, which
we can index into from SourceLocs in the current AST.
This starts the Syntax sublibrary, which will support structured
editing APIs. Some skeleton support and basic implementations are
in place for types and generics in the grammar. Yes, it's slightly
redundant with what we have right now. lib/AST conflates syntax
and semantics in the same place(s); this is a first step in changing
that to separate the two concepts for clarity and also to get closer
to incremental parsing and type-checking. The goal is to eventually
extract all of the syntactic information from lib/AST and change that
to be more of a semantic/symbolic model.
Stub out a Semantics manager. This ought to eventually be used as a hub
for encapsulating lazily computed semantic information for syntax nodes.
For the time being, it can serve as a temporary place for mapping from
Syntax nodes to semantically full lib/AST nodes.
This is still in a molten state - don't get too close, wear appropriate
proximity suits, etc.
