We cannot use unowned strings for token texts of incrementally parsed
syntax trees since the source buffer to which reused nodes refer will
have been freed for reused nodes. Always copying the token text whenever
OwnedString is passed is too expensive. A reference counted copy of the
string allows us to keep the token's string alive across incremental
parses while eliminating unnecessary copies.
The recommended way forward is to use the SyntaxClassifier on the Swift
side.
By removing the C++ SyntaxClassifier, we can also eliminate the
-force-libsyntax-based-processing option that was used to bootstrap
incremental parsing and would generate the syntax map from a syntax
tree.
For ScalarTraits, a buffer was always created on the heap to which the
scalar string value was written just to be copied to the output buffer
again. In case the value already exists in a memory buffer it is way
cheaper to avoid the heap allocation and copy it straight to the output
buffer.
For now, the accessors have been underscored as `_read` and `_modify`.
I'll prepare an evolution proposal for this feature which should allow
us to remove the underscores or, y'know, rename them to `purple` and
`lettuce`.
`_read` accessors do not make any effort yet to avoid copying the
value being yielded. I'll work on it in follow-up patches.
Opaque accesses to properties and subscripts defined with `_modify`
accessors will use an inefficient `materializeForSet` pattern that
materializes the value to a temporary instead of accessing it in-place.
That will be fixed by migrating to `modify` over `materializeForSet`,
which is next up after the `read` optimizations.
SIL ownership verification doesn't pass yet for the test cases here
because of a general fault in SILGen where borrows can outlive their
borrowed value due to being cleaned up on the general cleanup stack
when the borrowed value is cleaned up on the formal-access stack.
Michael, Andy, and I discussed various ways to fix this, but it seems
clear to me that it's not in any way specific to coroutine accesses.
rdar://35399664
If enabled using the environment variable
SOURCEKIT_INCREMENTAL_PARSE_VALIDATION, the incrementally parsed syntax
tree will be compared to the from-scratch parsing syntax tree. If they
differ a warning is emitted and log files showing the difference written
to a temporary directory.
Naming the bit-field structs is a significant readability improvement
because it's very clear that you shouldn't touch e.g. Bits.Token
without having checked/asserted that you're in a token case.
The assertions are all in statement context (which was obvious
because the NDEBUG versions all included semicolons), so there's no
reason not to use the traditional `do { } while (false)` trick instead
of a statement-expression.
This also clears up some warnings in atypical build configurations.
The existing libSyntax infrastructure uses external python
dictionaries to share logic between C++ and Swift implementations.
This patch teaches trivia kinds to adapt to this infrastructure
as well.
Make the quoted parameters more portable. This is needed in order to
support building swift on Windows. Simplify the declaration of the
flags and do a small formatting clean up. This ensures that the ninja
generation gets the command line invocation correct.
* Implement #warning and #error
* Fix #warning/#error in switch statements
* Fix AST printing for #warning/#error
* Add to test case
* Add extra handling to ParseDeclPoundDiagnostic
* fix dumping
* Consume the right paren even in the failure case
* Diagnose extra tokens on the same line after a diagnostic directive
Introduced SyntaxArena for managing memory and cache.
SyntaxArena holds BumpPtrAllocator as a allocation storage.
RawSyntax is now able to be constructed with normal heap allocation, or
by SyntaxArena. RawSyntax has ManualMemory flag which indicates it's managed by
SyntaxArena. If the flag is true, its Retain()/Release() is no-op thus it's
never destructed by IntrusiveRefCntPtr.
This speedups the memory allocation for RawSyntax.
Also, in Syntax parsing, "token" RawSyntax is reused if:
a) It's not string literal with >16 length; and
b) It doesn't contain random text trivia (e.g. comment).
This reduces the overall allocation cost.
