Use `ExportedSourceFile.sourceLocationConverter.lineTable.virtualFiles`
to populate the information in `swift::SourceManger` and
`swift::SourceFile` when "parsing" with ASTGen
The Clang importer was directly calling into the parser to parse the
attribute (or modifier) within swift_attr. Aside from being gross, this
isn't possible with ASTGen.
Instead, teach ParseSourceFileRequest to deal with modifiers in the
same way that the Clang importer was hardcoding, and have the Clang
importer pull the attributes/modifiers off of the "missing"
declaration introduced by the request.
One benefit of this approach is that we're only parsing each
swift_attr source buffer once, then cloning the attributes each time
it's used, so we should be doing less work overall.
Fixes rdar://139119159.
Now that ASTGen should be able to generate most Swift code. Let's
remove "legacy parser" call-in, and remove the unhealthy cyclic
dependency between lib/Parse and ASTGen.
Introduce a number of fixes to allow us to fully use declarations that
are produced by applying a peer macro to an imported declarations.
These changes include:
* Ensuring that we have the right set of imports in the source file
containing the macro expansion, because it depends only on the module
it comes from
* Ensuring that name lookup looks in that file even when the
DeclContext hierarchy doesn't contain the source file (because it's
based on the Clang module structure)
Expand testing to be sure that we're getting the right calls,
diagnostics, and generated IR symbols.
Previously, they were being parsed as top-level code, which would cause
errors because there are no definitions. Introduce a new
GeneratedSourceInfo kind to mark the purpose of these buffers so the
parser can handle them appropriately.
This started out as a crash, where an expression macro could not be
defined in terms of one of the builtin macros (e.g., `#line`), because
we were expecting a macro expansion expression but didn't get one.
Easy fix.
However, this uncovered a second bug, which is that we couldn't handle
an expression macro expansino to `#line`. This is because we were
parsing the macro expansion buffer as "top level items", which treats
`#line` at the start of a line as a deprecated alias of
`#sourceLocation`. Switch over to parsing a single expression in these
contexts, and fix up an issue where `#isolation` didn't even have that
expression.
Fixes rdar://139372780.
* Make ExportedSourceFile hold any Syntax as the root node
* Move `ExportedSourceFileRequest::evaluate()` to `ParseRequests.cpp`
* Pass the decl context and `GeneatedSourceFileInfo::Kind` to
`swift_ASTGen_parseSourceFile()` to customize the parsing
* Make `ExportedSourceFile` to hold an arbitrary Syntax node
* Move round-trip checking into `ExportedSourceFileRequest::evaluate()`
* Split `parseSourceFileViaASTGen` completely from C++ parsing logic
(in `ParseSourceFileRequest::evaluate()`)
* Remove 'ParserDiagnostics' experimental feature: Now that we have
ParserASTGen mode which includes the swift-syntax parser diagnostics.
Closures appearing in freestanding macro arguments don't have
discriminators assigned, since we don't actually emit them.
Similarly we skip recording opaque return types that appear in macro
arguments, since they don't get emitted.
However this logic didn't take delayed parsing into account, which must
save and restore the InFreestandingMacroArgument bit correctly.
As a result, if the freestanding macro argument contained a closure
which contained a local function with a declaration that has an
opaque return type, we would crash in serialization from attempting
to mangle an opaque return type nested inside of a closure without a
discriminator.
Fixes rdar://135445004
The "buffer ID" in a SourceFile, which is used to find the source file's
contents in the SourceManager, has always been optional. However, the
effectively every SourceFile actually does have a buffer ID, and the
vast majority of accesses to this information dereference the optional
without checking.
Update the handful of call sites that provided `nullopt` as the buffer
ID to provide a proper buffer instead. These were mostly unit tests
and testing programs, with a few places that passed a never-empty
optional through to the SourceFile constructor.
Then, remove optionality from the representation and accessors. It is
now the case that every SourceFile has a buffer ID, simplying a bunch
of code.
Although I don't plan to bring over new assertions wholesale
into the current qualification branch, it's entirely possible
that various minor changes in main will use the new assertions;
having this basic support in the release branch will simplify that.
(This is why I'm adding the includes as a separate pass from
rewriting the individual assertions)
Test shadowed variable of same type
Fully type check caller side macro expansion
Skip macro default arg caller side expr at decl primary
Test macro expand more complex expressions
Set synthesized expression as implicit
Add test case for with argument, not compiling currently
Test with swiftinterface
Always use the string representation of the default argument
Now works across module boundary
Check works for multiple files
Make default argument expression work in single file
Use expected-error
Disallow expression macro as default argument
Using as a sub expression in default argument still allowed as expression macros behave the same as built-in magic literals
Now that the macro expansion machinery is wrapping the contents of a
body macro in curly braces, parse it as a single brace statement. This
has the advantage of giving us real locations for `{` and `}` rather
than synthesizing them, and simplifies the implementation of macro
body expansion somewhat.
Function body macros allow one to introduce a function body for a
particular function, either providing a body for a function that
doesn't have one, or wholesale replacing the body of a function that
was written with a new one.
Function bodies are skipped during typechecking when one of the
-experimental-skip-*-function-bodies flags is passed to the frontend. This was
implemented by setting the "body kind" of an `AbstractFunctionDecl` during decl
checking in `TypeCheckDeclPrimary`. This approach had a couple of issues:
- It is incompatible with skipping function bodies during lazy typechecking,
since the skipping is only evaluated during a phase of eager typechecking.
- It prevents skipped function bodies from being parsed on-demand ("skipped" is
a state that is distinct from "parsed", when they ought to be orthogonal).
This needlessly prevented complete module interfaces from being emitted with
-experimental-skip-all-function-bodies.
Storing the skipped status of a function separately from body kind and
requestifying the determination of whether to skip a function solves these
problems.
Resolves rdar://116020403
This is phase-1 of switching from llvm::Optional to std::optional in the
next rebranch. llvm::Optional was removed from upstream LLVM, so we need
to migrate off rather soon. On Darwin, std::optional, and llvm::Optional
have the same layout, so we don't need to be as concerned about ABI
beyond the name mangling. `llvm::Optional` is only returned from one
function in
```
getStandardTypeSubst(StringRef TypeName,
bool allowConcurrencyManglings);
```
It's the return value, so it should not impact the mangling of the
function, and the layout is the same as `std::optional`, so it should be
mostly okay. This function doesn't appear to have users, and the ABI was
already broken 2 years ago for concurrency and no one seemed to notice
so this should be "okay".
I'm doing the migration incrementally so that folks working on main can
cherry-pick back to the release/5.9 branch. Once 5.9 is done and locked
away, then we can go through and finish the replacement. Since `None`
and `Optional` show up in contexts where they are not `llvm::None` and
`llvm::Optional`, I'm preparing the work now by going through and
removing the namespace unwrapping and making the `llvm` namespace
explicit. This should make it fairly mechanical to go through and
replace llvm::Optional with std::optional, and llvm::None with
std::nullopt. It's also a change that can be brought onto the
release/5.9 with minimal impact. This should be an NFC change.
Allow freestanding macros to be used at top-level.
- Parse top-level `#…` as `MacroExpansionDecl` when we are not in scripting mode.
- Add macro expansion decls to the source lookup cache with name-driven lazy expansion. Not supporting arbitrary name yet.
- Experimental support for script mode and brace-level declaration macro expansions: When type-checking a `MacroExpansionExpr`, assign it a substitute `MacroExpansionDecl` if the macro reference resolves to a declaration macro. This doesn’t work quite fully yet and will be enabled in a future fix.
Global peer macro expansions are not injected into the AST. Instead, they
are visited as "auxiliary declarations" when needed, such as in the decl
checker and during SILGen. This is the same mechanism used for local property
wrappers and local lazy variables.
Once an accessor macro has produced accessors, parse them and wire them
into the AST so the rest of the compiler will see them. First
end-to-end test case!
Each macro expansion buffer was getting parsed twice: once by
ParseSourceFileRequest (which is used by unqualified name lookup) and
once to parse the expression when type-checking the expanded macro.
This meant that the same code had two ASTs. Hilarity ensures.
Stop directly invoking the parser on macro-expanded code. Instead, go
through ParseSourceFileRequest *as is always the right way*, and dig
out the expression we want.
Replace the use of the "consistency check" vended by swift-syntax with
an ASTGen-implemented operation that emits diagnostics from the new parser
via the normal diagnostic engine. This eliminates our last dependency
on SwiftCompilerSupport, so stop linking it.
In the Swift grammar, the top-level of a source file is a mix of three
different kinds of "items": declarations, statements, and expressions.
However, the existing parser forces all of these into declarations at
parse time, wrapping statements and expressions in TopLevelCodeDecls,
so the primary API for getting the top-level entities in source files
is based on getting declarations.
Start generalizing the representation by storing ASTNode instances at
the top level, rather than declaration pointers, updating many (but
not all!) uses of this API. The walk over declarations is a (cached)
filter to pick out all of the declarations. Existing parsed files are
unaffected (the parser still creates top-level code declarations), but
the new "macro expansion" source file kind skips creating top-level
code declarations so we get the pure parse tree. Additionally, some
generalized clients (like ASTScope lookup) will now look at the list
of items, so they'll be able to walk into statements and expressions
without the intervening TopLevelCodeDecl.
Over time, I'd like to phase out `getTopLevelDecls()` entirely,
relying on the new `getTopLevelItems()` for parsed content. We can
introduce TopLevelCodeDecls more lazily for semantic walks.
Allow user-defined macros to be loaded from dynamic libraries and evaluated.
- Introduce a _CompilerPluginSupport module installed into the toolchain. Its `_CompilerPlugin` protocol acts as a stable interface between the compiler and user-defined macros.
- Introduce a `-load-plugin-library <path>` attribute which allows users to specify dynamic libraries to be loaded into the compiler.
A macro library must declare a public top-level computed property `public var allMacros: [Any.Type]` and be compiled to a dynamic library. The compiler will call the getter of this property to obtain and register all macros.
Known issues:
- We current do not have a way to strip out unnecessary symbols from the plugin dylib, i.e. produce a plugin library that does not contain SwiftSyntax symbols that will collide with the compiler itself.
- `MacroExpansionExpr`'s type is hard-coded as `(Int, String)`. It should instead be specified by the macro via protocol requirements such as `signature` and `genericSignature`. We need more protocol requirements in `_CompilerPlugin` to handle this.
- `dlopen` is not secure and is only for prototyping use here.
Friend PR: apple/swift-syntax#1022
If we had a structural overflow, don't perform round-trip testing. The
new parser currently fails, and will need to deal with the problem in
a more appropriate manner than the existing one, which cuts off lexing entirely.
Remove the CMake configuration option `SWIFT_SWIFT_PARSER_ROUNDTRIP`.
Instead, whenever the "early" SwiftSyntax module is built, link the
Swift Swift parser into the compiler and related tools.
Introduce a new experimental feature `ParserRoundTrip` that can be
enabled to perform round-trip testing.
When enabled, compile in support for round-trip testing the new
SwiftSyntax-provided Swift parser alongside the existing parser. Right
now, this means parsing every source file with the new parser and
ensuring that the resulting syntax tree can reproduce the input source
precisely. Over time, this is expected to grow.
Opt in to this behavior by passing the following to build-script:
build-script --early-swiftsyntax --extra-cmake-options=-DSWIFT_SWIFT_PARSER_MODE:STRING=ROUNDTRIP
