For a `@Testable` import in program source, if a Swift interface dependency is discovered, and has an adjacent binary `.swiftmodule`, open up the module, and pull in its optional dependencies. If an optional dependency cannot be resolved on the filesystem, fail silently without raising a diagnostic.
The macro name resolution in the source lookup cache was only looking at
macros in the current module, meaning that any names introduced by peer
or declaration macros declared in one module but used in another would
not be found by name lookup.
Switch the source lookup cache over to using the same
`forEachPotentialResolvedMacro` API that is used by lookup within
types, so we have consistent name-lookup-level macro resolution in both
places.
... except that would be horribly cyclic, of course, so introduce name
lookup flags to ignore top-level declarations introduced by macro
expansions. This is semantically correct because macro expansions are
not allowed to introduce new macros anyway, because that would have
been a terrible idea.
Fixes rdar://107321469. Peer and declaration macros at module scope
should work a whole lot better now.
Add a private discriminator to the mangling of an outermost-private `MacroExpansionDecl` so that declaration macros in different files won't have colliding macro expansion buffer names.
rdar://107462515
A @testable import allows a client to call internal decls which may
refer to non-public dependencies. To support such a use case, load
non-public transitive dependencies of a module when it's imported
@testable from the main module.
This replaces the previous behavior where we loaded those dependencies
for any modules built for testing. This was risky as we would load more
module for any debug build, opening the door to a different behavior
between debug and release builds. In contrast, applying this logic to
@testable clients will only change the behavior of test targets.
rdar://107329303
If a module was first read using the adjacent swiftmodule and then
reloaded using the swiftinterface, we would do an up to date check on
the adjacent module but write out the unit using the swiftinterface.
This would cause the same modules to be indexed repeatedly for the first
invocation using a new SDK. On the next run we would instead raad the
swiftmodule from the cache and thus the out of date check would match
up.
The impact of this varies depending on the size of the module graph in
the initial compilation and the number of jobs started at the same time.
Each SDK dependency is re-indexed *and* reloaded, which is a drain on
both CPU and memory. Thus, if many jobs are initially started and
they're all going down this path, it can cause the system to run out of
memory very quickly.
Resolves rdar://103119964.
Introduce a new flag `-export-as` to specify a name used to identify the
target module in swiftinterfaces. This provides an analoguous feature
for Swift module as Clang's `export_as` feature.
In practice it should be used when a lower level module `MyKitCore` is
desired to be shown publicly as a downstream module `MyKit`. This should
be used in conjunction with `@_exported import MyKitCore` from `MyKit`
that allows clients to refer to all services as being part of `MyKit`,
while the new `-export-as MyKit` from `MyKitCore` will ensure that the
clients swiftinterfaces also use the `MyKit` name for all services.
In the current implementation, the export-as name is used in the
module's clients and not in the declarer's swiftinterface (e.g.
`MyKitCore`'s swiftinterface still uses the `MyKitCore` module name).
This way the module swiftinterface can be verified. In the future, we
may want a similar behavior for other modules in between `MyKitCore` and
`MyKit` as verifying a swiftinterface referencing `MyKit` without it
being imported would fail.
rdar://103888618
This changes the scanner's behavior to "resolve" a discovered module's dependencies to a set of Module IDs: module name + module kind (swift textual, swift binary, clang, etc.).
The 'ModuleDependencyInfo' objects that are stored in the dependency scanner's cache now carry a set of kind-qualified ModuleIDs for their dependencies, in addition to unqualified imported module names of their dependencies.
Previously, the scanner's internal state would cache a module dependnecy as having its own set of dependencies which were stored as names of imported modules. This led to a design where any time we needed to process the dependency downstream from its discovery (e.g. cycle detection, graph construction), we had to query the ASTContext to resolve this dependency's imports, which shouldn't be necessary. Now, upon discovery, we "resolve" a discovered dependency by executing a lookup for each of its imported module names (this operation happens regardless of this patch) and store a fully-resolved set of dependencies in the dependency module info.
Moreover, looking up a given module dependency by name (via `ASTContext`'s `getModuleDependencies`) would result in iterating over the scanner's module "loaders" and querying each for the module name. The corresponding modules would then check the scanner's cache for a respective discovered module, and if no such module is found the "loader" would search the filesystem.
This meant that in practice, we searched the filesystem on many occasions where we actually had cached the required dependency, as follows:
Suppose we had previously discovered a Clang module "foo" and cached its dependency info.
-> ASTContext.getModuleDependencies("foo")
--> (1) Swift Module "Loader" checks caches for a Swift module "foo" and doesn't find one, so it searches the filesystem for "foo" and fails to find one.
--> (2) Clang Module "Loader" checks caches for a Clang module "foo", finds one and returns it to the client.
This means that we were always searching the filesystem in (1) even if we knew that to be futile.
With this change, queries to `ASTContext`'s `getModuleDependencies` will always check all the caches first, and only delegate to the scanner "loaders" if no cached dependency is found. The loaders are then no longer in the business of checking the cached contents.
To handle cases in the scanner where we must only lookup either a Swift-only module or a Clang-only module, this patch splits 'getModuleDependencies' into an alrady-existing 'getSwiftModuleDependencies' and a newly-added 'getClangModuleDependencies'.
Previously, when evaluating a `#if canImport(Module, _version: 42)` directive the compiler could diagnose and ignore the directive under the following conditions:
- The associated binary module is corrupt/bogus.
- The .tbd for an underlying Clang module is missing a current-version field.
This behavior is surprising when there is a valid `.swiftinterface` available and it only becomes apparent when building against an SDK with an old enough version of the module that the version in the `.swiftinterface` is too low, making this failure easy to miss. Some modules have different versioning systems for their Swift and Clang modules and it can also be intentional for a distributed binary `.swiftmodule` to contain bogus data (to force the compiler to recompile the `.swiftinterface`) so we need to handle both of these cases gracefully and predictably.
Now the compiler will enumerate all module loaders, ask each of them to attempt to parse the module version and then consistently use the parsed version from a single source. The `.swiftinterface` is preferred if present, then the binary module if present, and then finally the `.tbd`. The `.tbd` is still always used exclusively for the `_underlyingVersion` variant of `canImport()`.
Resolves rdar://88723492
The ObjCMethodLookupTable for protocols was not being serialized and rebuilt on load, so NominalTypeDecl::lookupDirect() on selectors was not working correctly for deserialized types. Correct this oversight.
We noticed some Swift clients rely on the serialized search paths in the module to
find dependencies and droping these paths altogether can lead to build failures like
rdar://85840921.
This change teaches the serialization to obfuscate the search paths and the deserialization
to recover them. This allows clients to keep accessing these paths without exposing
them when shipping the module to other users.
* Fix unnecessary one-time recompile of stdlib with -enable-ossa-flag
This includes a bit in the module format to represent if the module was
compiled with -enable-ossa-modules flag. When compiling a client module
with -enable-ossa-modules flag, all dependent modules are checked for this bit,
if not on, recompilation is triggered with -enable-ossa-modules.
* Updated tests
We should hold off actually building the binary module file until it is imported.
`canImport` queries can happen, for example, during dependency scanning, when we do not wish to have the scanner tool execute any module builds.
Resolves rdar://82603098
The MemoryBuffer loader is used by LLDB during debugging to import binary Swift
modules from .swift_ast sections. Modules imported from .swift_ast sections are
never produced from textual interfaces. By disabling resilience the expression
evaluator in the debugger can directly access private members.
rdar://79462915
It's a known issue that we are using arm64e interfaces contents for the arm64 target,
meaning the encoded module flags are specifying -target arm64e-x-x instead of
-target arm64-x-x. Fortunately, we can tell the target arch from the interface file
name, so we could sanitize the target to use by inferring arch from the file name.
This mechanism allows the compiler to use a backup interface file to build into a binary module when
a corresponding interface file from the SDK is failing for whatever reasons. This mechansim should be entirely opaque
to end users except several diagnostic messages communicating backup interfaces are used.
Part of rdar://77676064
For config condition `canImport(Foo, version: N)`, this patch teaches the compiler to check N
against the version of the Swift module Foo on disk. It returns true if the module version on
disk is greater or equal to N and returns false otherwise.
Part of rdar://73992299
canImport should be able to take an additional parameter labeled by either version or
underlyingVersion. We need underlyingVersion for clang modules with Swift overlays because they
have separate version numbers. The library users are usually interested in checking the importability
of the underlying clang module instead of its Swift overlay.
Part of rdar://73992299
The locations stored in .swiftsourceinfo included the presumed file,
line, and column. When a location is requested it would read these, open
the external file, create a line map, and find the offset corresponding
to that line/column.
The offset is known during serialization though, so output it as well to
avoid having to read the file and generate the line map.
Since the serialized location is returned from `Decl::getLoc()`, it
should not be the presumed location. Instead, also output the line
directives so that the presumed location can be built as per normal
locations.
Finally, move the cache out of `Decl` and into `ASTContext`, since very
few declarations will actually have their locations deserialized. Make
sure to actually write to that cache so it's used - the old cache was
never written to.
Cursor info for a constructor would previously give the cursor info for
the containing type only. It now also adds cursor info for the
constructor itself in a "secondary_symbols" field.
Refactor `passCursorInfoForDecl` to use a single allocator rather than
keeping track of positions in a buffer and assigning everything at the
end of the function.
Refactor the various available refactoring gathering functions to take a
SmallVectorImpl and to not copy strings where they don't need to.
Resolves rdar://75385556
To help consolidate our various types describing imports, this commit moves the following types and methods to Import.h:
* ImplicitImports
* ImplicitStdlibKind
* ImplicitImportInfo
* ModuleDecl::ImportedModule
* ModuleDecl::OrderImportedModules (as ImportedModule::Order)
* ModuleDecl::removeDuplicateImports() (as ImportedModule::removeDuplicates())
* SourceFile::ImportFlags
* SourceFile::ImportOptions
* SourceFile::ImportedModuleDesc
This commit is large and intentionally kept mechanical—nothing interesting to see here.
This refactoring allows us to drop ModuleInterfaceLoader when explicit modules
are enabled. Before this change, the dependencies scanner needs the loader to be
present to access functionalities like collecting prebuilt module candidates.
In the fast dependency scanner, depending on whether a module intrface was found via the import search path or framework search path, encode into the dependency graph Swift module details, whether a given module is a framework.
In order to avoid accidentally implicitly loading modules that are expected but were not provided as explicit inputs.
- Use either SerializedModuleLoader or ExplicitSwiftModuleLoader for loading of partial modules, depending on whether we are in Explicit Module Build or Implicit Module Build mode.
-compile-module-from-interface action now takes arguments of -candidate-module-file.
If one of the candidate module files is up-to-date, the action emits a forwarding
module pointing to the candidate module instead of building a binary module.
Instead of replacing an interface file with its up-to-date compile module,
the dep-scanner should report potentially up-to-date module candidates either adjacent to
the interface file or in the prebuilt module cache. swift-driver should later pass down
these candidates to -compile-module-from-interface invocation and the front-end job
will check if one of the candidates is ready to use. The front-end job then either emits a forwarding
module to an up-to-date candidate or a binary module.
The difference with `ModuleFile` is that `ModuleFileSharedCore` provides immutable data and is independent of a particular ASTContext.
It is designed to be able to be shared across multiple `ModuleFile`s of different `ASTContext`s in a thread-safe manner.
For the explicit module mode, swift-driver uses -compile-module-from-interface to
generate modules from interfaces found by the dependency scanner. However, we don't
need to build the binary module if up-to-date modules are available, either adjacent
to the interface file or in the prebuilt module cache directory. This patch teaches
dependencies scanner to report these ready-to-use binary modules.
