Naming the bit-field structs is a significant readability improvement
because it's very clear that you shouldn't touch e.g. Bits.Token
without having checked/asserted that you're in a token case.
The assertions are all in statement context (which was obvious
because the NDEBUG versions all included semicolons), so there's no
reason not to use the traditional `do { } while (false)` trick instead
of a statement-expression.
This also clears up some warnings in atypical build configurations.
Introduced SyntaxArena for managing memory and cache.
SyntaxArena holds BumpPtrAllocator as a allocation storage.
RawSyntax is now able to be constructed with normal heap allocation, or
by SyntaxArena. RawSyntax has ManualMemory flag which indicates it's managed by
SyntaxArena. If the flag is true, its Retain()/Release() is no-op thus it's
never destructed by IntrusiveRefCntPtr.
This speedups the memory allocation for RawSyntax.
Also, in Syntax parsing, "token" RawSyntax is reused if:
a) It's not string literal with >16 length; and
b) It doesn't contain random text trivia (e.g. comment).
This reduces the overall allocation cost.
libSyntax nodes don't maintain absolute source location on each
individual node. Instead, the absolute locations are calculated on
demand with a given root by accumulating the length of all the other
nodes before the target node. This bridging is important for issuing
diagnostics from libSyntax entities.
With the observation that our current implementation of the source
location calculation has multiple bugs, this patch re-implemented this
bridging by using the newly-added syntax visitor. Also, we moved the function
from RawSyntax to Syntax for better visibility.
To test this source location calculation, we added a new action in
swift-syntax-test. This action parses a given file as a
SourceFileSyntax, calculates the absolute location of the
EOF token in the SourceFileSyntax, and dump the buffer from the start
of the input file to the absolute location of the EOF. Finally, we compare
the dump with the original input to ensure they are identical.
RawTokenSyntax is a derived class from RawSyntax that is reference
counted with its own destructor function registered. Unfortunately, the destructor
function of RawSyntax is non-virtual before this patch. This means when reference counter
releases a pointer of RawSyntax, it won't clean-up the additional stuff in RawTokenSyntax.
I forgot to reset the macro parameters after converting them to
varargs, which didn't get caught running PR testing.
This patch ensures they're all the same.
* Generate libSyntax API
This patch removes the hand-rolled libSyntax API and replaces it with an
API that's entirely automatically generated. This means the API is
guaranteed to be internally stylistically and functionally consistent.
Previously, users of TokenSyntax would always deal with RC<TokenSyntax>
which is a subclass of RawSyntax. Instead, provide TokenSyntax as a
fully-realized Syntax node, that will always exist as a leaf in the
Syntax tree.
This hides the implementation detail of RawSyntax and SyntaxData
completely from clients of libSyntax, and paves the way for future
generation of Syntax nodes.
* Refactor Tuple Type Syntax
This patch:
- Refactors TypeArgumentListSyntax and
TypeArgumentListSyntaxData to use the SyntaxCollection and
SyntaxCollectionData APIs.
- Refactors TupleTypeElementSyntax to own its trailing comma, and
updates the tests accordingly.
- Provides an infrastructure for promoting types to use
the SyntaxCollection APIs
* Addressed comments.
* Renamed makeBlankTypeArgumentList()
* Update makeTupleType
* Changed makeTupleType to take an element list.
* Updated comment.
* Improved API for creating TupleTypeElementListSyntax'es
* Added round-trip test
* Removed last TypeArgumentList holdovers.
* Fixed round-trip test invocation
This will make it easier to incrementally implement syntax nodes,
while allowing us to embed nodes that we do know about inside ones
that we don't.
https://bugs.swift.org/browse/SR-4062
A return statement needs something to return, so implement
integer-literal-expression too. This necessarily also forced
UnknownExprSyntax, UnknownStmtSyntax, and UnknownDeclSyntax,
which are stand-in token buckets for when we don't know
how to transform/migrate an AST.
This commit also contains the core function for caching
SyntaxData children. This is highly tricky code, with some
detailed comments in SyntaxData.{h,cpp}. The gist is that
we have to atomically swap in a SyntaxData pointer into the
child field, so we can maintain pointer identity of SyntaxData
nodes, while still being able to cache them internally.
To prove that this works, there is a multithreaded test that
checks that two threads can ask for a child that hasn't been
cached yet without crashing or violating pointer identity.
https://bugs.swift.org/browse/SR-4010
Add an option to the lexer to go back and get a list of "full"
tokens, which include their leading and trailing trivia, which
we can index into from SourceLocs in the current AST.
This starts the Syntax sublibrary, which will support structured
editing APIs. Some skeleton support and basic implementations are
in place for types and generics in the grammar. Yes, it's slightly
redundant with what we have right now. lib/AST conflates syntax
and semantics in the same place(s); this is a first step in changing
that to separate the two concepts for clarity and also to get closer
to incremental parsing and type-checking. The goal is to eventually
extract all of the syntactic information from lib/AST and change that
to be more of a semantic/symbolic model.
Stub out a Semantics manager. This ought to eventually be used as a hub
for encapsulating lazily computed semantic information for syntax nodes.
For the time being, it can serve as a temporary place for mapping from
Syntax nodes to semantically full lib/AST nodes.
This is still in a molten state - don't get too close, wear appropriate
proximity suits, etc.
These APIs return SourceLocs, and eventually the Parser should consume
tokens, which now include source trivia such as whitespace and comments,
and package them into a purely syntactic tree. Just a tiny step. NFC.
