Update the lexing code for the replacement of the
`'/.../'` and `'|...|'` delimiters with `#/.../#`
and `#|...|#` respectively, in addition to
allowing the `re'...'` delimiter.
This reverts commit a67a0436f7, reversing
changes made to 9965df76d0.
This commit or the earlier commit this commit is based on (#40531) broke the
incremental bot.
Update the lexing implementation to defer to the
regex library, which will pass back the pointer
from to resume lexing, and update the emission to
call the new `Regex(_regexString:version:)`
overload, that will accept the regex string with
delimiters.
Because this uses the library's lexing
implementation, the delimiters are now `'/.../'`
and `'|...|'` instead of plain `'...'`.
With `-enable-experimental-string-processing`,
start lexing `'` delimiters as regex literals (this
is just a placeholder delimiter for now). The
contents of which gets passed to the libswift
library, which can return an error string to be
emitted, or null for success.
The libswift side isn't yet hooked up to the Swift
regex parser, so for now just emit a dummy
diagnostic for regexes starting with quantifiers.
If successful, build an AST node which will be
emitted as an implicit call to an
`init(_regexString:)` initializer of an in-scope
`Regex` decl (which will eventually be a known
stdlib decl).
These modules are part of the experimental declarative string processing feature. If accepted to the Standard Library, _StringProcessing will be available via implicit import just like _Concurrency, though _MatchingEngine will still be hidden as an implementation detail.
`_MatchingEngine` will contain the general-purpose pattern matching engine ISA, bytecode, and executor. `_StringProcessing` will contain regular expression and pattern matching APIs whose implementation depends on the matching engine..
Also consolidates frontend flag `-enable-experimental-regex` as `-enable-experimental-string-processing`.
Resolves rdar://85478647.
Previously, when we reached the maximum nesting level, we changed the current token’s kind to an EOF token. A lot of places in the parser are not set up to expect this token change. The intended workaround was to check whether pushing a structure marker failed (which would change the token kind) and bail out parsing if this happened. This was fragile and caused assertion failures in assert builds.
Instead of changing the current token’s kind, and failing to push the structure marker, let the lexer know that it should cut off lexing, essentially making the input buffer stop at the current position. The parser will continue to consume its current token (`Parser.Tok`) and the next token that’s already lexed in the lexer (`Lexer.NextToken`) before reaching the emulated EOF token. Thus two more tokens are parsed than before, but that shouldn’t make much of a difference.
This hidden frontend option lets us be more lax
when type-checking in the presence of editor
placeholders by treating them as holes during
constraint solving.
This cast causes overflows on platforms that use unsigned char as default and at least on linux-aarch64 causes false negatives in the switch statements.
This is an intermediate state in which the lexer delegates the
responsibility for trivia lexing to the parser. Later, the parser will
delegate this responsibility to SyntaxParsingContext which will hand it
over to SyntaxParseAction, which will only lex the pieces if it is
really necessary to do so.
The lexer is only responsible for skipping over trivia and noting their
length. A separate TriviaLexer can be invoked to split the raw trivia
string into its pieces.
Since most of the time the trivia pieces aren't needed, this will allow
us to later only parse trivia into pieces when they are explicitly
needed.
When the location given to getLocForStartOfLine was an empty line, it
would actually return the location of the next line rather than the
given location as it should.
If the location given to getLocForEndOfLine was inside a token on a line
that was either empty or contained whitespace, it would skip to the end
of that token and then return the location for the next line. This was
an issue for multiline strings, where the string is a single token but
it's over multiple lines.
* [Parser] Update 'Confusables.def' file to include confusable and base character names
* [Parser] Add a new utility method to return the names of the confusable and base characters for a given confusable codepoint
* [Parser] Update diagnostic for confusable character during lexing to mention confusable and base character names
* [Sema] If there is just a single confusable character, emit a tailored diagnostic that also mentions the character names
* [Diagnostics] Add new diagnostic messages to the localization file
* [Test] Update confusables test
* [Utils] Update unicode confusables txt file and update script to regenerate confusables def file
* [Parse] Regenerate 'Confusables.def' using updated script
* [Utils] Adjust generate_confusables script based on review feedback
Fix a mistake with name mapping. Updated header comment. Fix a couple of linting issues.
* [Parse] Regenerate 'Confusables.def' file once again after script changes
* [Parse] Add the newline after end of 'getConfusableAndBaseCodepointNames' method
* [Test] Update diagnostic message in 'Syntax/Parser/diags.swift'
* WIP implementation
* Cleanup implementation
* Install backedge rather than storing array reference
* Add diagnostics
* Add missing parameter to ResultFinderForTypeContext constructor
* Fix tests for correct fix-it language
* Change to solution without backedge, change lookup behavior
* Improve diagnostics for weak captures and captures under different names
* Remove ghosts of implementations past
* Address review comments
* Reorder member variable initialization
* Fix typos
* Exclude value types from explicit self requirements
* Add tests
* Add implementation for AST lookup
* Add tests
* Begin addressing review comments
* Re-enable AST scope lookup
* Add fixme
* Pull fix-its into a separate function
* Remove capturedSelfContext tracking from type property initializers
* Add const specifiers to arguments
* Address review comments
* Fix string literals
* Refactor implicit self diagnostics
* Add comment
* Remove trailing whitespace
* Add tests for capture list across multiple lines
* Add additional test
* Fix typo
* Remove use of ?: to fix linux build
* Remove second use of ?:
* Rework logic for finding nested self contexts
- Stop producing 'backtick' trivia for escaping identifier token. '`'s
are now parts of the token text
- Adjust and simplify C++ libSyntax APIs
- Add 'is_deprecated' property to Trivia.py to attribute SwiftSyntax
APIs
rdar://problem/54810608
Instead of creating the AST directly in the parser (and libSyntax or
SwiftSyntax via SyntaxParsingContext), make Parser to explicitly create
a tree of ParsedSyntaxNodes. Their OpaqueSyntaxNodes can be either
libSyntax or SwiftSyntax. If AST is needed, it can be generated from the
libSyntax tree.
Allow the use of declarations whose names start with $ in all
modes. However, normal code cannot define new entities with names that
start with $: only the implementation can do that, e.g., for property
delegates.
We’re printing a new name for lazy storage in parseable interfaces,
`$__lazy_storage_$_{propname}`. This is intentionally $-prefixed so it cannot
conflict with variables written in source, but it doesn’t use a `.` anymore
because parseable interfaces need to be...parseable.
Previously, the Lexer kept a single flag whether we’re lexing Swift or SIL. Instead, keep track if we’re parsing Swift, SIL, or a Swiftinterface file. .swiftinterface files allow $-prefixed identifiers anywhere.
We've been running doxygen with the autobrief option for a couple of
years now. This makes the \brief markers into our comments
redundant. Since they are a visual distraction and we don't want to
encourage more \brief markers in new code either, this patch removes
them all.
Patch produced by
for i in $(git grep -l '\\brief'); do perl -pi -e 's/\\brief //g' $i & done
Text on editors (e.g. Xcode) may contain identifiers following to CC
position which should be considered as "filtering text" for the code
completion.
For example, a user types '@a' the CC position is between '@' and 'a'.
The user probably expects attributes starting with 'a'.
Eat identifier characters after CC token in Lexer. By this change,
for instance for '@<token>IB class', the parser now can detect this
is a start of class declaration and emit class attributes completions.
rdar://problem/46103294
This is needed to support `print $0` in lldb where `$0` is the
shortcut for a closure argument. The associated lldb change
will be merged together with this one and will contain a test.
<rdar://problem/201719448>
when skipping to the end of the interpolated expression.
i.e. Skip the comment as a comment.
Previously, ')' or '"' in comment in interpolated expression used to
cause assertion failure or mis-compilation in no-assert build.
rdar://problem/20289969
in string interpolation in multiline string literal.
"""
\("<-this is unterminated.
"""
In this case, the outer multiline literal should form 'tok::unknown'
along with a error message.
