We were only keeping track of `RawSyntax` node IDs to incrementally transfer a syntax tree via JSON. However, AFAICT the incremental JSON transfer option has been superceeded by `SyntaxParseActions`, which are more efficient.
So, let’s clean up and remove the `RawSyntax` node ID and JSON incremental transfer option.
In places that still need a notion of `RawSyntax` identity (like determining the reused syntax regions), use the `RawSyntax`’s pointer instead of the manually created ID.
In `incr_transfer_round_trip.py` always use the code path that uses the `SyntaxParseActions` and remove the transitional code that was still using the incremental JSON transfer but was never called.
Instead, only reference count the SyntaxArena that the RawSyntax nodes
live in. The user of RawSyntax nodes must guarantee that the SyntaxArena
stays alive as long as the RawSyntax nodes are being accessed.
During parse time, the SyntaxTreeCreator holds on to the SyntaxArena
in which it creates RawSyntax nodes. When inspecting a syntax tree,
the root SyntaxData node keeps the SyntaxArena alive. The change should
be mostly invisible to the users of the public libSyntax API.
This change significantly decreases the overall reference-counting
overhead. Since we were not able to free individual RawSyntax nodes
anyway, performing the reference-counting on the level of the
SyntaxArena feels natural.
It was originally designed for faster trasmission of syntax trees from
C++ to SwiftSyntax, but superceded by the CLibParseActions. There's no
deserializer for it anymore, so let's just remove it.
This silences the instances of the warning from Visual Studio about not all
codepaths returning a value. This makes the output more readable and less
likely to lose useful warnings. NFC.
For ScalarTraits, a buffer was always created on the heap to which the
scalar string value was written just to be copied to the output buffer
again. In case the value already exists in a memory buffer it is way
cheaper to avoid the heap allocation and copy it straight to the output
buffer.
If enabled using the environment variable
SOURCEKIT_INCREMENTAL_PARSE_VALIDATION, the incrementally parsed syntax
tree will be compared to the from-scratch parsing syntax tree. If they
differ a warning is emitted and log files showing the difference written
to a temporary directory.
The existing libSyntax infrastructure uses external python
dictionaries to share logic between C++ and Swift implementations.
This patch teaches trivia kinds to adapt to this infrastructure
as well.
A string interpolation expression is composed of { OpenQuote, Segments,
CloseQuote }. To represent OpenQuote, CloseQuote and StringSegment, we have to
introduce new token kinds correspondingly.
* Generate libSyntax API
This patch removes the hand-rolled libSyntax API and replaces it with an
API that's entirely automatically generated. This means the API is
guaranteed to be internally stylistically and functionally consistent.
Previously, users of TokenSyntax would always deal with RC<TokenSyntax>
which is a subclass of RawSyntax. Instead, provide TokenSyntax as a
fully-realized Syntax node, that will always exist as a leaf in the
Syntax tree.
This hides the implementation detail of RawSyntax and SyntaxData
completely from clients of libSyntax, and paves the way for future
generation of Syntax nodes.
