This was used a long time ago for a design of a scanner which could rely on the client to specify that some modules *will be* present at a given location but are not yet during the scan. We have long ago determined that the scanner must have all modules available to it at the time of scan for soundness. This code has been stale for a couple of years and it is time to simplify things a bit by deleting it.
The algorithm already performs pairwise checks on module dependencies brought into compilation per-source-file. Previously, the algorithm considered the entire sub-graph of a given source file. Actual source compiles do not consider the full transitive module dependency set for cross-import-overlay lookup, but rather only directly-imported modules in a given source file, and '@_exported import' Swift transitive dependencies.
This change adds tracking of whether a given import statement is 'exported' to the dependency scanner and then refines the cross-import overlay lookup logic to only consider transitive modules that are exported by directly-imported dependencies.
In https://github.com/swiftlang/swift/pull/77156, normalization was introduced
for -target-variant triples. That PR also caused -target-variant arguments to
be inherited from the main compilation options whenever building dependency
modules from their interfaces, which is incorrect. The -target-variant option
must only be specified when compiling a "zippered" module, but the dependencies
of zippered modules are not necessarily zippered themselves and
indiscriminantly propagating the option can cause miscompilation.
The new, more targeted approach to normalizing arm64e triples simply uses the
arch and subarch of the -target argument of the main compile to decide whether
the subarch of both the -target and -target-variant arguments of a dependency
need adjustment.
Resolves rdar://135322077 and rdar://141640919.
As per #65930, the Clang importer's Clang instance may be configured with a different (higher) OS version than the compilation target itself in order to be able to load pre-compiled Clang modules that are aligned with the broader SDK, and match the SDK deployment target against which Swift modules are also built. Code-generation, however, must use the actual compilation target triple. This matches how Swift itself loads Swift module dependencies as well: dependency '.swiftinterface' files are type-checked against the availability epoch and code-generated against the actual compilation triple.
Resolves rdar://113712186
Instead, only add the overlay itself, and let it refer to its own dependencies, which will still get recorded in the overall output.
Resolves rdar://117010118
The code, previously, only properly handled such dependencies being a distinct category for Swift source and Swift textual dependency infos. Swift binary module dependencies must handle this similarly and this change adds the missing support for them. Recent refactor of the scanner also means that now Swift binary dependencies with Swift overlay dependencies may crash the scanner, and this change resolves this as well.
Resolves rdar://117088840
It is possible that import resolution failed because we are attempting to resolve a module which can only be brought in via a modulemap of a different Clang module dependency which is not otherwise on the current search paths. For example, suppose we are scanning a '.swiftinterface' for module 'Foo', which contains:
'''
@_exported import Foo
import Bar
...
Where 'Foo' is the underlying Framework clang module whose '.modulemap' defines an auxiliary module 'Bar'. Because 'Foo' is a framework, its modulemap is under '<some_framework_search_path>/Foo.framework/Modules/module.modulemap'. Which means that lookup of `Bar` alone from Swift will not be able to locate the module in it. However, the lookup of Foo will itself bring in the auxiliary module becuase the Clang scanner instance scanning for clang module Foo will be able to find it in the corresponding framework module's modulemap and register it as a dependency which means it will be registered with the scanner's cache in the step above. To handle such cases, we first add all successfully-resolved modules and (for Clang modules) their transitive dependencies to the cache, and then attempt to re-query imports for which resolution originally failed from the cache. If this fails, then the scanner genuinely failed to resolve this dependency.
There is a special case that already exists in 'ClangImporter' for implicit module loading:
Import of a "submodule" named "Foo.Private" is treated as a top-level module named "Foo_Private".
Clang has special support for this.
Resolves rdar://108287140
Instead of being a part of 'directDependencies' on a module dependency info, make them a separate array of dependency IDs for Swift Source and Textual modules.
This will allow clients to still distinguish direct module dependencies imported from a given module, versus dependencies added because direct/transitive Clang module dependencies have Swift overlays.
This change does *not* remove overlay dependencies from 'directDependencies' yet, just adds them as a separate field on the module details info. A followup change will remove overlay and bridging header dependencies from 'directDependencies' once the clients have had a chance to adopt to this change.
For example, when scanning a source module `Foo`, which, when depending on module `Bar` causes a cross-import overlay `_Foo_Bar` to be added, do not add this cross-import overlay when scanning `Foo` itself. For example, if `Foo` adds a dependency on `Bar` itself in its own dependency graph.
The `__future__` we relied on is now, where the 3 specific things are
all included [since Python 3.0](https://docs.python.org/3/library/__future__.html):
* absolute_import
* print_function
* unicode_literals
* division
These import statements are no-ops and are no longer necessary.
We should hold off actually building the binary module file until it is imported.
`canImport` queries can happen, for example, during dependency scanning, when we do not wish to have the scanner tool execute any module builds.
Resolves rdar://82603098
The dependency scanner's cache persists across different queries and answering a subsequent query's module lookup with a module not in the query's search path is not correct.
For example, suppose we are looking for a Swift module `Foo` with a set of search paths `SP`.
And dependency scanner cache already contains a module `Foo`, for which we found an interface file at location `L`. If `L`∉`SP`, then we cannot re-use the cached entry because we’d be resolving the scanning query to a filesystem location that the current scanning context is not aware of.
Resolves rdar://81175942
Before this change, we always use the Swift target triple to instantiate the internal
Clang instance. When loading a Swift module from the textual interface, we may pick up
a lower target triple to use to build the Swift module because the target is hard-coded
in the textual interface file. This implies we may end up building multiple versions of the
same Clang module, one for each target triple of the loading Swift module.
This change adds a new frontend flag -clang-target to allow clients to specify a
consistent clang target to use across the Swift module boundaries. This value won't change
because it's not part of .swiftinterface files.
swift-driver should pass down -clang-target for each frontend invocation, and its value should be
identical to -target.
Related to: rdar://72480261
When outputting strings for things like filenames, using `write_escaped` will result in Unicode characters being outputted with a full 3-character-octal or hex escape. Clients which expect a UTF-8 JSON output will not be able to parse such escape sequences.
When building a set of command-line options required to build a Clang module, also add `NonPathCommandLine` from Clang's `ModuleDeps`. These flags are mandatory in order to build modules discovered by the scanner.
Resolves rdar://70212660
This matches the behavior of the current client (`swift-driver`) and reduces ambiguity in how the nodes in the graph are to be treated. Swift dependencies with a textual interface, for example, must be built into a binary module by clients. Swift dependencies without a textual interface, with only a binary module, are to be used directly, without any up-to-date checks.
Note, this is distinct from Swift dependencies that have a textual interface, for which we also detect potential pre-build binary module candidates. Those are still reported in the `details` field of textual Swift dependencies as `prebuiltModuleCandidates`.
This ensures that when the dependency scanner is invoked with additional clang (`-Xcc`) options, the Clang scanner is correctly configured using these options.
To help solving rdar://67079780, this change allows swift-driver to configure scanner using additional
arguments passed down via the batch input JSON file for each module under scanning.
In the fast dependency scanner, depending on whether a module intrface was found via the import search path or framework search path, encode into the dependency graph Swift module details, whether a given module is a framework.
Instead of replacing an interface file with its up-to-date compile module,
the dep-scanner should report potentially up-to-date module candidates either adjacent to
the interface file or in the prebuilt module cache. swift-driver should later pass down
these candidates to -compile-module-from-interface invocation and the front-end job
will check if one of the candidates is ready to use. The front-end job then either emits a forwarding
module to an up-to-date candidate or a binary module.
For the explicit module mode, swift-driver uses -compile-module-from-interface to
generate modules from interfaces found by the dependency scanner. However, we don't
need to build the binary module if up-to-date modules are available, either adjacent
to the interface file or in the prebuilt module cache directory. This patch teaches
dependencies scanner to report these ready-to-use binary modules.
Building each Swift module explicitly requires dependency PCMs to be built
with the exactly same deployment target version. This means we may need to
build a Clang module multiple times with different target triples.
This patch removes the -target arguments from the reported PCM build
arguments and inserts extraPcmArgs fields to each Swift module.
swift-driver can combine the generic PCM arguments with these extra arguments
to get the command suitable for building a PCM specifically for
that loading Swift module.
To support -disable-implicit-swift-modules, the explicitly built modules
are passed down as compiler arguments. We need this new module loader to
handle these modules.
This patch also stops ModuleInterfaceLoader from building module from interface
when -disable-implicit-swift-modules is set.
Running shell commands using Swift in simulator tests is hard. We change the mechanism so that
BuildModulesFromGraph.swift prints command line arguments and a simple python script
picks these arguments and actually runs the command.
