Introduce a macro that can stamp out wrapper
classes for underlying C++ pointers, and use
it to define BridgedDiagnosticEngine in
ASTBridging. Then, migrate users of
BridgedDiagEngine onto it.
When performing code completion at the end of a file, the IDE inspection target would point to the null byte terminating the end of the string. That would cause us to consider this null byte as a code completion marker. When continuing to scan for the actual EOF, we would walk past the end of the buffer.
Simply don’t consider the last null byte as a candidate for the code completion marker to fix the problem.
This reverts commit e9dedf3c27.
The revert is required as foreign reference types are available for SwiftStdlib 5.8 and above, but the Swift compiler
sources back deploy to older stdlibs as well.
The lexer will be responsible for knowing whether we have a code completion token, everything else will also work for other IDE inspection features.
The changes start to really make sense once I rename CodeCompletion -> IDEInspection in a lot of places.
Ban space and tab as the last character of a
`/.../` regex literal, unless escaped with a
backslash. This matches the banning of space and
tab as the first character, and helps avoid breaking
source in even more cases.
While skipping, if we encounter a token that looks
like it could be the start of a `/.../` regex
literal, fall back to parsing the function or type
body normally, as such a token could become a
regex literal. As such, it could treat `{` and
`}` as literal, or otherwise have contents that
would be lexically invalid Swift.
To avoid falling back in too many cases, we apply
the existing regex literal heuristics. Cases that
pass the heuristic fall back to regular parsing.
Cases that fail the heuristic are further checked
to make sure they wouldn't contain an unbalanced
`{` or `}`, but otherwise are allowed to be
skipped. This allows us to continue skipping for
most occurrences of infix and prefix `/`.
This is meant as a lower risk workaround to fix the
the issue, we ought to go back to handling regex
literals in the lexer.
Resolves rdar://95354010
Previously we would only check for a starting
character of `)` when performing a tentative
lex of a regex literal. Expand this to cover the
entire range of the regex literal, ensuring to
take escapes and custom character classes into
account.
This is unfortunately needed to ensure we correctly
re-lex regex literal tokens correctly, which is
needed for diagnostic logic to correctly compute
source ranges.
rdar://92469692
Teach the lexer not to consider `/` an operator
character when attempting to re-lex a regex
literal. This allows us to split off a prefix
operator.
Previously this was done after-the-fact in the
parser, but that didn't cover the unapplied infix
operator case, and didn't form a `tok::amp_prefix`
for `foo(&/.../)`, which led to a suboptimal
diagnostic.
This also now means we'll split an operator for
cases such as `foo(!/^/)` rather than treating it
as an unapplied infix operator.
rdar://92469917
This fixes:
* An issue where the diagnostic messages were leaked
* Diagnose at correct position inside the regex literal
To do this:
* Introduce 'Parse' SwiftCompiler module that is a bridging layer
between '_CompilerRegexParser' and C++ libParse
* Move libswiftParseRegexLiteral and libswiftLexRegexLiteral to 'Parse'
Also this change makes 'SwiftCompilerSources/Package.swift' be configured
by CMake so it can actually be built with 'swift-build'.
rdar://92187284
Queue up diagnostics when lexing, waiting until
`Lexer::lex` is called before emitting them. This
allows us to re-lex without having to deal with
previously invalid tokens.
As the _MatchingEngine module no longer contains the matching engine, this patch renames this module to describe its role more accurately. Because this module primarily contains the AST and the regex parsing logic, I propose we rename it to "_RegexParser".
Also renames the ExperimentalRegex module in SwiftCompilerSources to _RegexParser for consistency. This would prevent errors if sources in _RegexParser used qualified lookup with the module name.
Update the lexing code for the replacement of the
`'/.../'` and `'|...|'` delimiters with `#/.../#`
and `#|...|#` respectively, in addition to
allowing the `re'...'` delimiter.
This reverts commit a67a0436f7, reversing
changes made to 9965df76d0.
This commit or the earlier commit this commit is based on (#40531) broke the
incremental bot.
Update the lexing implementation to defer to the
regex library, which will pass back the pointer
from to resume lexing, and update the emission to
call the new `Regex(_regexString:version:)`
overload, that will accept the regex string with
delimiters.
Because this uses the library's lexing
implementation, the delimiters are now `'/.../'`
and `'|...|'` instead of plain `'...'`.
With `-enable-experimental-string-processing`,
start lexing `'` delimiters as regex literals (this
is just a placeholder delimiter for now). The
contents of which gets passed to the libswift
library, which can return an error string to be
emitted, or null for success.
The libswift side isn't yet hooked up to the Swift
regex parser, so for now just emit a dummy
diagnostic for regexes starting with quantifiers.
If successful, build an AST node which will be
emitted as an implicit call to an
`init(_regexString:)` initializer of an in-scope
`Regex` decl (which will eventually be a known
stdlib decl).
These modules are part of the experimental declarative string processing feature. If accepted to the Standard Library, _StringProcessing will be available via implicit import just like _Concurrency, though _MatchingEngine will still be hidden as an implementation detail.
`_MatchingEngine` will contain the general-purpose pattern matching engine ISA, bytecode, and executor. `_StringProcessing` will contain regular expression and pattern matching APIs whose implementation depends on the matching engine..
Also consolidates frontend flag `-enable-experimental-regex` as `-enable-experimental-string-processing`.
Resolves rdar://85478647.
Previously, when we reached the maximum nesting level, we changed the current token’s kind to an EOF token. A lot of places in the parser are not set up to expect this token change. The intended workaround was to check whether pushing a structure marker failed (which would change the token kind) and bail out parsing if this happened. This was fragile and caused assertion failures in assert builds.
Instead of changing the current token’s kind, and failing to push the structure marker, let the lexer know that it should cut off lexing, essentially making the input buffer stop at the current position. The parser will continue to consume its current token (`Parser.Tok`) and the next token that’s already lexed in the lexer (`Lexer.NextToken`) before reaching the emulated EOF token. Thus two more tokens are parsed than before, but that shouldn’t make much of a difference.
This hidden frontend option lets us be more lax
when type-checking in the presence of editor
placeholders by treating them as holes during
constraint solving.
This cast causes overflows on platforms that use unsigned char as default and at least on linux-aarch64 causes false negatives in the switch statements.
