When planning for a swift source module, it should not get build
commands for its module dependencies. Those dependencies should be
planned and added by swift-driver.
This is another regression from #76700 that causes unnecessary increase
of build command-line size.
rdar://141843125
Currently, `-direct-clang-cc1-module-build` and `-only-use-extra-clang-opts`
have to be passed together for clang importer creation to succeed.
Missing either will result in error. Simplified the swift-frontend flags
by removing `-only-use-extra-clang-opts` and let
`-direct-clang-cc1-module-build` to do both.
The code of `ScanDependencies.cpp` was creating invalid JSON since #66031
because in the case of having `extraPcmArgs` and `swiftOverlayDependencies`,
but not `bridgingHeader`, a comma will not be added at the end of
`extraPcmArgs`, creating an invalid JSON file. Additionally that same PR
added a trailing comma at the end of the `swiftOverlayDependencies`, which
valid JSON does not allow, but that bug was removed in #66366.
Both problems are, however, present in the 5.9 branch, because #66936
included #66031, but not #66366.
Besides fixing the problem in `ScanDependencies.cpp` I modified every test
that uses `--scan-dependencies` to pass the produced JSON through
Python's `json.tool` in order to validate proper JSON is produced. In
most cases I was able to pipe the output of the tool into `FileCheck`,
but in some cases the validation is done by itself because the checks
depend on the exact format generated by `--scan-dependencies`. In
a couple of tests I added a call to `FileCheck` that seemed to be
missing.
Without these changes, two tests seems to be generating invalid JSON in
my machine:
- `ScanDependencies/local_cache_consistency.swift` (which outputs `Expecting ',' delimiter: line 525 column 11 (char 22799)`)
- `ScanDependencies/placholder_overlay_deps.swift`
This changes the scanner's behavior to "resolve" a discovered module's dependencies to a set of Module IDs: module name + module kind (swift textual, swift binary, clang, etc.).
The 'ModuleDependencyInfo' objects that are stored in the dependency scanner's cache now carry a set of kind-qualified ModuleIDs for their dependencies, in addition to unqualified imported module names of their dependencies.
Previously, the scanner's internal state would cache a module dependnecy as having its own set of dependencies which were stored as names of imported modules. This led to a design where any time we needed to process the dependency downstream from its discovery (e.g. cycle detection, graph construction), we had to query the ASTContext to resolve this dependency's imports, which shouldn't be necessary. Now, upon discovery, we "resolve" a discovered dependency by executing a lookup for each of its imported module names (this operation happens regardless of this patch) and store a fully-resolved set of dependencies in the dependency module info.
Moreover, looking up a given module dependency by name (via `ASTContext`'s `getModuleDependencies`) would result in iterating over the scanner's module "loaders" and querying each for the module name. The corresponding modules would then check the scanner's cache for a respective discovered module, and if no such module is found the "loader" would search the filesystem.
This meant that in practice, we searched the filesystem on many occasions where we actually had cached the required dependency, as follows:
Suppose we had previously discovered a Clang module "foo" and cached its dependency info.
-> ASTContext.getModuleDependencies("foo")
--> (1) Swift Module "Loader" checks caches for a Swift module "foo" and doesn't find one, so it searches the filesystem for "foo" and fails to find one.
--> (2) Clang Module "Loader" checks caches for a Clang module "foo", finds one and returns it to the client.
This means that we were always searching the filesystem in (1) even if we knew that to be futile.
With this change, queries to `ASTContext`'s `getModuleDependencies` will always check all the caches first, and only delegate to the scanner "loaders" if no cached dependency is found. The loaders are then no longer in the business of checking the cached contents.
To handle cases in the scanner where we must only lookup either a Swift-only module or a Clang-only module, this patch splits 'getModuleDependencies' into an alrady-existing 'getSwiftModuleDependencies' and a newly-added 'getClangModuleDependencies'.
The clang importer sets the clang resource directory to
`RuntimeResourcePath`/clang (usually "lib/swift/clang", a symbolic link
into "lib/clang/<version>"). This was inadvertently being removed in a
check to remove the clang *executable* path.
Modify that check to only the first argument so the resource directory
is properly passed through.
Doing so will allow clients to know which Swift-specific PCM arguments are already captured from the scan that first discovered this module.
SwiftDriver, in particular, will be able to use this information to avoid re-scanning a given Clang module if the initial scan was sufficient for all possible sets of PCM arguments on Swift modules that depend on said Clang module.
Swift module building commands need -sdk because dependencies sometimes use
sdk-relative paths (prebuilt modules for example). Without -sdk, the command
will not be able to local these dependencies, leading to unnecessary
rebuilding from textual interfaces.
rdar://81177968
https://github.com/apple/swift/pull/37774 (related to rdar://72480261) has made it so that the target of built clang modules (even downstream from Swift interface dependencies) can be consistent with that of the main module. This means that when building transitive Clang dependencies of the main module, they will always have a matching triple to the main module itself (ensured with `-clang-target`). This means we no longer have to report the target triple in the set of `extraPCMArgs` which encode an interface-specific requirement for building its dependencies.
Clang deduces its installation directory from the `argv[0]` parameter (see clang/lib/Frontend/CreateInvocationFromCommandLine.cpp), and the default include search paths are computed based on the installation directory.
This change allows compiling Swift code that imports the C++ stdlib without having to manually specify the include search path of `std` headers.
We were previously forgetting about these.
On top of that, make sure we treat accumulated dependencies as a set, because adding clang overlay dependencies may duplicate existing already-added dependencies for some module.
When building a set of command-line options required to build a Clang module, also add `NonPathCommandLine` from Clang's `ModuleDeps`. These flags are mandatory in order to build modules discovered by the scanner.
Resolves rdar://70212660
This ensures that when the dependency scanner is invoked with additional clang (`-Xcc`) options, the Clang scanner is correctly configured using these options.
Unlike Swift modules, building Clang PCMs requires search path options like -F and -I to
look for transitive header includes because in module maps, we can only find top-level headers.
rdar://67589328
Some implicitly imported modules aren't printed in the textual interface file as explicit import,
e.g. SwiftOnoneSupport. We should check implicit imports and add them to the dependency graph.
For the explicit module mode, swift-driver uses -compile-module-from-interface to
generate modules from interfaces found by the dependency scanner. However, we don't
need to build the binary module if up-to-date modules are available, either adjacent
to the interface file or in the prebuilt module cache directory. This patch teaches
dependencies scanner to report these ready-to-use binary modules.
Swift interface files may specify the effective language version to use. When building
a PCM loadable for these textual interface files, we should respect the language
version. This patch moves -fapinotes-swift-version from the generic PCM
commands to the extra PCM arguments owned by each loading Swift module.
Building each Swift module explicitly requires dependency PCMs to be built
with the exactly same deployment target version. This means we may need to
build a Clang module multiple times with different target triples.
This patch removes the -target arguments from the reported PCM build
arguments and inserts extraPcmArgs fields to each Swift module.
swift-driver can combine the generic PCM arguments with these extra arguments
to get the command suitable for building a PCM specifically for
that loading Swift module.
To support -disable-implicit-swift-modules, the explicitly built modules
are passed down as compiler arguments. We need this new module loader to
handle these modules.
This patch also stops ModuleInterfaceLoader from building module from interface
when -disable-implicit-swift-modules is set.
When resolving direct dependencies for a given Swift module, we go over all Clang module dependencies and add, as additional dependencies, their Swift overlays. We find overlays by querying `ASTContext::getModuleDependencies` with the Clang module's name. If the Clang module in question is a dependency of a Swift module with the same name, we will end up adding the Swift module as its own dependence.
e.g.
- Swift A depends on Clang A
- Add Clang A to dependencies of Swift A
- We look for overlays of Clang A, by name, and find Swift A
- Add Swift A to dependencies of Swift A
From what I can tell, the logic upstream is sound, and `getModuleDependencies` is doing the right thing, so this change is simply restricting what gets added when we are looking for overlays.
Resolves rdar://problem/63731428
Running shell commands using Swift in simulator tests is hard. We change the mechanism so that
BuildModulesFromGraph.swift prints command line arguments and a simple python script
picks these arguments and actually runs the command.
