https://bugs.swift.org/browse/SR-6926
This happens when the Parser re-lexing comment tokens that sets
ArtificialEOF at the end of comment range.
It used to cause an assertion failure:
(!value || Kind == tok::identifier) && "only identifiers can be escaped identifiers"
For example: @$S1m3fooyyF
It's needed to change the mangling prefix to $S.
The parser change only affects SIL (and not swift).
I didn't add test case because it will be fully tested when changing the mangling prefix.
* Re-apply "libSyntax: Ensure round-trip printing when we build syntax tree from parser incrementally. (#12709)"
* Re-apply "libSyntax: Root parsing context should hold a reference to the current token in the parser, NFC."
* Re-apply "libSyntax: avoid copying token text when lexing token syntax nodes, NFC. (#12723)"
* Actually fix the container-overflow issue.
This is likely the root cause for memory surge when we always turn on
syntax token lexing. Since the underlying buffer outlives the syntax
tree, it's reasonable to refer the text instead of copying and owning it.
For very large source files, the parser's syntax map---which contains a
very large number of TrivialLists---was taking an inordinate amount of
memory due to the inefficiency of std::deque. Specifically, a
std::deque containing just one trivial element would allocate 4k of
memory. With the ~120MB SIL output of one of the parse_stdlib tests,
these std::deques would add up to > 6GB of memory, most of which is
wasted.
Replacing the std::deque with a std::vector knocks the memory required
for one of the parse_stdlib tests from > 8GB down closer to 2 GB. The
parser's syntax map is still large (e.g., a 512MB allocation for the
overall vector plus a few hundred MB of raw-syntax data), but not
prohibitively so.
Part of rdar://problem/34771322.
For very large source files, the parser's syntax map---which contains a
very large number of TrivialLists---was taking an inordinate amount of
memory due to the inefficiency of std::deque. Specifically, a
std::deque containing just one trivial element would allocate 4k of
memory. With the ~120MB SIL output of one of the parse_stdlib tests,
these std::deques would add up to > 6GB of memory, most of which is
wasted.
Replacing the std::deque with a std::vector knocks the memory required
for one of the parse_stdlib tests from > 8GB down closer to 2 GB. The
parser's syntax map is still large (e.g., a 512MB allocation for the
overall vector plus a few hundred MB of raw-syntax data), but not
prohibitively so.
Part of rdar://problem/34771322.
Update error messages to mention the invalid character.
Improve the diagnostic of floating point exponents.
Add tests for error messages when parsing floating point exponents.
Update existing tests for new error messages.
Rephrased error message to indicate which character is unexpected.
Provide error message variations when parsing binary, octal, decimal (default), and hexadecimal integer literals.
Look for unexpected digits in binary and octal integer literals.
Look for unexpected letters in hex integer literals.
Resolves: SR-5236 rdar://problem/32858684
Previously, users of TokenSyntax would always deal with RC<TokenSyntax>
which is a subclass of RawSyntax. Instead, provide TokenSyntax as a
fully-realized Syntax node, that will always exist as a leaf in the
Syntax tree.
This hides the implementation detail of RawSyntax and SyntaxData
completely from clients of libSyntax, and paves the way for future
generation of Syntax nodes.
Maintain inner most string literal mode to determine whether we allow
newline character or not.
* Disallow newline after multiline string in string interpolation. (SR-5171)
* Allow unbalanced `"` in multiline string in string interpolation.
* [Parse] Refactored internal structure of Tokens.def and documented usage.
Added a level of structure to the macro definitions to allow Swift
keywords to be cleanly accessed separately from SIL and Swift keywords
together. Documented structure and usage.
* [Parse] Made use of new guarantees and abstractions in Tokens.def
Used guarantees about undefining macros after import and new
SWIFT_KEYWORD abstraction to simplify usage of the Token.def
imports.
* Gardening
