Otherwise, the closure discriminator will be incremented by one when the closure witht he code completion token is parsed a second time during the second pass, and thus it would receive a different discriminator during the second pass.
This will be necessary to make cursor info completion like to inspect the just parsed source file because the callback from parsing the code completion token won’t be called.
In the Swift grammar, the top-level of a source file is a mix of three
different kinds of "items": declarations, statements, and expressions.
However, the existing parser forces all of these into declarations at
parse time, wrapping statements and expressions in TopLevelCodeDecls,
so the primary API for getting the top-level entities in source files
is based on getting declarations.
Start generalizing the representation by storing ASTNode instances at
the top level, rather than declaration pointers, updating many (but
not all!) uses of this API. The walk over declarations is a (cached)
filter to pick out all of the declarations. Existing parsed files are
unaffected (the parser still creates top-level code declarations), but
the new "macro expansion" source file kind skips creating top-level
code declarations so we get the pure parse tree. Additionally, some
generalized clients (like ASTScope lookup) will now look at the list
of items, so they'll be able to walk into statements and expressions
without the intervening TopLevelCodeDecl.
Over time, I'd like to phase out `getTopLevelDecls()` entirely,
relying on the new `getTopLevelItems()` for parsed content. We can
introduce TopLevelCodeDecls more lazily for semantic walks.
Introduce a new source file kind to describe source files for macro
expansions, and include the macro expression that they expand. This
establishes a "parent" relationship
Also track every kind of auxiliary source file---whether for macro
expansions or other reasons---that is introduced into a module, adding
an operation that allows us to find the source file that contains a
given source location.
Refactors `parseSingleAttrOption()` to create a helper that can parse a single arbitrary `Identifier`. This simplifies the handling of `SwiftNativeObjCRuntimeBaseAttr`, `ObjCRuntimeNameAttr`, and `ProjectedValuePropertyAttr`.
Introduce support for parsing declaration attributes that occur within
example:
#if hasAttribute(frozen)
@frozen
#endif
public struct X { ... }
will apply to "frozen" attribute to the struct `X`, but only when the
compiler supports the "frozen" attribute.
Correctly determining whether a particular `#if` block contains
attributes to be associated with the following declaration vs.
starting a new declaration requires arbitrary lookahead. The parser
will ensure that at least one of the branches of the `#if` contains an
attribute, and that none of the branches contains something that does
not fit the attribute grammar, before committing to parsing the `#if`
clause as part of the declaration attributes. This lookahead does
occur at the top level (e.g., in the parsing of top-level declarations
and code), but should only need to scan past the first `#if` line to
the following token in the common case.
Unlike other `#if` when used to wrap statements or declarations, we
make no attempt to record the `#if` not taken anywhere in the AST.
This reflects a change in attitude in the design of the AST, because
we have found that trying to represent this information there (e.g.,
via `IfConfigDecl`) complicates clients while providing little value.
This information is best kept in the syntax tree, only.
When recovering from a parser error in an expression, we resumed parsing at a '{'. I assume this was because we wanted to continue inside e.g. an if-body if parsing the condition failed, but it's actually causing more issue because when parsing e.g.
```swift
expr + has - error +
functionTakesClosure {
}
```
we continue parsing at the `{` of the trailing closure, which is a completely garbage location to continue parsing.
The motivating example for this change was (in a result builder)
```swift
Text("\(island.#^COMPLETE^#)")
takeTrailingClosure {}
```
Here `Text(…)` has an error (because it contains a code completion token) and thus we skip `takeTrailingClosure`, effectively parsing
```swift
Text(….) {}
```
which the type checker wasn’t very happy with and thus refused to provide code completion. With this change, we completely drop `takeTrailingClosure {}`. The type checker is a lot happier with that.
Previously, when we reached the maximum nesting level, we changed the current token’s kind to an EOF token. A lot of places in the parser are not set up to expect this token change. The intended workaround was to check whether pushing a structure marker failed (which would change the token kind) and bail out parsing if this happened. This was fragile and caused assertion failures in assert builds.
Instead of changing the current token’s kind, and failing to push the structure marker, let the lexer know that it should cut off lexing, essentially making the input buffer stop at the current position. The parser will continue to consume its current token (`Parser.Tok`) and the next token that’s already lexed in the lexer (`Lexer.NextToken`) before reaching the emulated EOF token. Thus two more tokens are parsed than before, but that shouldn’t make much of a difference.
* Fix unnecessary one-time recompile of stdlib with -enable-ossa-flag
This includes a bit in the module format to represent if the module was
compiled with -enable-ossa-modules flag. When compiling a client module
with -enable-ossa-modules flag, all dependent modules are checked for this bit,
if not on, recompilation is triggered with -enable-ossa-modules.
* Updated tests
Fixe a couple of bugs in libSyntax parsing found by enabling `-verify-syntax-tree` for `%target-build-swift`:
- Fix parsing of the `actor` contextual keyword in actor decls
- Don't build a libSyntax tree when parsing the availability macro
- The availability macro is not part of the source code and doesn't form a valid Swift file, thus creation of a libSyntax tree is completely pointless and will fail
- Add support for parsing `@_originallyDefinedIn` attributes.
- Add support for parsing `#sourceLocation` in member decl lists
- Add support for effectful properties (throwing/async getters/setters)
- Add support for optional types as the base of a key path (e.g. `\TestOptional2?.something`)
- Allow platform restrictions without a version (e.g. `_iOS13Aligned`)
In particular, we were unconditionally dropping argument labels on accessors; now we only do that for property accessors, not subscript accessors.
Doing this unconditionally causes ClangImporter failures when a method parameter is called “subscript” (really!), so this behavior is enabled only by the caller’s request.
For backtracking scopes that are never cancelled, we can completely disable the SyntaxParsingContext, avoiding the creation of deferred nodes which will never get recorded.
Small peformance improvement: Token is larger than a pointer and not
modified in the performance-criticial TokenReceivers, so we can pass
it by reference.
Instead, only reference count the SyntaxArena that the RawSyntax nodes
live in. The user of RawSyntax nodes must guarantee that the SyntaxArena
stays alive as long as the RawSyntax nodes are being accessed.
During parse time, the SyntaxTreeCreator holds on to the SyntaxArena
in which it creates RawSyntax nodes. When inspecting a syntax tree,
the root SyntaxData node keeps the SyntaxArena alive. The change should
be mostly invisible to the users of the public libSyntax API.
This change significantly decreases the overall reference-counting
overhead. Since we were not able to free individual RawSyntax nodes
anyway, performing the reference-counting on the level of the
SyntaxArena feels natural.
Do the same thing that we are already doing for trivia: Since RawSyntax
nodes always live inside a SyntaxArena, we don't need to tail-allocate
an OwnedString to store the token's text. Instead we can just copy it
to the SyntaxArena. If we copy the entire source buffer to the syntax
arena at the start of parsing, this means that no more copies are
required later on. Plus we also avoid ref-counting the OwnedString which
should also increase performance.
This is again a transitional state before SyntaxParsingContext hands
the responsibility over to SyntaxTreeCreator and from there to
SyntaxParseActions.
This is an intermediate state in which the lexer delegates the
responsibility for trivia lexing to the parser. Later, the parser will
delegate this responsibility to SyntaxParsingContext which will hand it
over to SyntaxParseAction, which will only lex the pieces if it is
really necessary to do so.
Code completion used to avoid forming single expression closures/function
bodies when the single expression contained the code completion expression
because a contextual type mismatch could result in types not being applied
to the AST, giving no completions.
Completions that have been migrated to the new solver-based completion
mechanism don't need this behavior, however. Rather than trying to guess
whether the type of completion we're going to end up performing is one of
the ones that haven't been migrated to the solver yet when parsing, instead
just always form single-expression closures/function bodies (like we do for
regular compilation) and undo the transformation if and when we know we're
going to perform a completion kind we haven't migrated yet.
Once all completion kinds are migrated, the undo-ing code can be removed.
For example, the completion below would trigger error recovery within the
closure, which we recover from by skipping to the first inner closure's right
brace. The fact that we recovered though, was not recorded. The closure is
treated as still being an error, triggering another recovery after it that
skips over the 'Thing' token, giving a lone closure expression, rather than a
call.
CreateThings {
Thing { point in
print("hello")
point.#^HERE^#
}
Thing { _ in }
}
This isn't an issue for code completion when the outer closure is a regular
closure, but when it's a function builder, invalid elements result in no types
being applied (no valid solutions) and we end up with no completion results.
The fix here is removing the error status from the parser result after the
initial parser recovery.
We need ClangImporterOptions to be persistent for several scenarios: (1)
when creating a sub-ASTContext to build Swift modules from interfaces; and
(2) when creating a new Clang instance to invoke Clang dependencies scanner.
This change is NFC.
Currently when parsing a SourceFile, the parser
gets handed pointers so that it can write the
interface hash and collected tokens directly into
the file. It can also call `setSyntaxRoot` at
the end of parsing to set the syntax tree.
In preparation for the removal of
`performParseOnly`, this commit formalizes these
values as outputs of `ParseSourceFileRequest`,
ensuring that the file gets parsed when the
interface hash, collected tokens, or syntax tree
is queried.
Sink the `BuildSyntaxTree` and
`CollectParsedTokens` bits into
`SourceFile::ParsingFlags`, with a static method
to get the parsing options from the lang opts.
Also add a parsing flag for enabling the interface
hash, which can be used instead of calling
`enableInterfaceHash`.
