For swift binary module deps, it's cache key and output are setup during
dependency scanning time. When loading the module dependency from the
incremental cache, need to validate the CAS key and output for swift
binary module both exist in the CAS, otherwise, it needs to be
invalidated so a rescan will re-ingest the binary modules into the CAS.
rdar://173647646
Compute and propagate the library level (api/spi/ipi) of each module
dependency through the dependency scanner so that the compiler can
correctly enforce private module import diagnostics in CAS mode, where
path-based SPI detection fails because CAS abstracts file paths to
content IDs.
Swift modules:
- Detect library level from the module interface path using
libraryLevelFromPath() during scanning, for both textual (.swiftinterface)
and binary (.swiftmodule) Swift modules.
Clang modules:
- Expose ModuleMapIsPrivate from clang::Module in ModuleDeps via the
dependency scanning infrastructure.
- Set library level for clang modules in bridgeClangModuleDependency()
using ModuleMapIsPrivate (catches module.private.modulemap in any
SDK location) and libraryLevelFromPath() on the module map file
(catches modules under PrivateFrameworks directories).
The library level is:
- Stored in ModuleDependencyInfo and serialized in the module dependency
cache (format version bumped to v8).
- Exposed through the swiftscan C API via a new
swiftscan_module_info_get_library_level() function (API minor version
bumped to 3).
- Emitted in the dependency scanner JSON output as "libraryLevel" for
all module kinds (Swift textual, Swift binary, Clang, and main module).
- Parsed from the explicit module map JSON by ExplicitModuleMapParser
for both Swift (ExplicitSwiftModuleInputInfo) and Clang
(ExplicitClangModuleInputInfo) modules.
- Looked up in ModuleLibraryLevelRequest via
ASTContext::getExplicitModuleLibraryLevel(name, isClang), which
consults the appropriate map (Swift or Clang) based on module kind.
rdar://172693314
Assisted-By: Claude
Fix a module build regression in PR86881 that causes an increase of
swift interface compilation tasks due to including the
const-gather-protocol list files into the build dependencies for
swiftinterface files. This causes targets cannot sharing module
dependencies.
This make sures the CASFS system only captures the const-gather-protocol
list in the tasks for main module, not for any of its module
dependencies.
rdar://171506799
During explicit module build, teach dependency scanner to emit the
module trace file instead of each following compile job command. This
reduces the duplicated info, and allows supporting fully cached build
that only loads module from CAS thus cannot produce the path to the
original module file on disk.
rdar://170007480
Dependency Scanning is recursive over discovered Swift module overlays and cross-import overlays. In the main 'resolveImportedModuleDependencies' routine, after all Swift and Clang dependencies of the main module are discovered, the scanner looks up Swift overlays for discovered Clang modules. For each such Swift overlay, the scanner will then proceed to call 'resolveImportedModuleDependencies' on the overlay module.
On these subsequent recursive calls to 'resolveImportedModuleDependencies', Clang dependency resolution will re-query all imports of the overlay module which do not resolve to Swift modules, even if they had been queried previously in a parent invocation. This change adds the ability to re-use previously-queried-and-cached Clang module dependency information by having the dependency cache also store the set of visible modules which resulted from each by-name lookup.
<!--
If this pull request is targeting a release branch, please fill out the
following form:
https://github.com/swiftlang/.github/blob/main/PULL_REQUEST_TEMPLATE/release.md?plain=1
Otherwise, replace this comment with a description of your changes and
rationale. Provide links to external references/discussions if
appropriate.
If this pull request resolves any GitHub issues, link them like so:
Resolves <link to issue>, resolves <link to another issue>.
For more information about linking a pull request to an issue, see:
https://docs.github.com/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue
-->
<!--
Before merging this pull request, you must run the Swift continuous
integration tests.
For information about triggering CI builds via @swift-ci, see:
https://github.com/apple/swift/blob/main/docs/ContinuousIntegration.md#swift-ci
Thank you for your contribution to Swift!
-->
This PR teaches Swift to take advantage of the new clang dependency
scanning API to use a single clang compiler instance per dependency
scanning worker to perform by-name queries.
rdar://136303612
This change adds collection of three metrics to the scanner:
- number of Swift module lookups
- number of named Clang module lookups
- recorded number of Clang modules which were imported into a Swift module by name
It introduces '-Rdependency-scan', which acts as a super-set flag to the existing '-Rdependency-scan-cache' and adds emission of the above metrics as remarks when this flag is enabled. Followup changes will add further remarks about dependency scanner progress.
For clients, such as the debugger, who do not have access the full
output of the dependency scanner, it is a huger performance and
correctness improvement if each explicitly built Swift module not just
serialized all its Clang .pcm dependencies (via the serialized Clang
compiler invocation) but also its direct Swift module dependencies.
This patch changes the Swift module format to store the absolute path
or cas cache key for each dependency in the INPUT block, and makes
sure the deserialization makes these available to the ESML.
rdar://150969755
To support distributed caching for targets that need to use legacy
layout file, the path of legacy layout needs to be remapped.
Current handling of the legacy layout file is to ingest all layout files
to the CAS FileSystem as part of the compiler resource files. But it
cannot convey remapped legacy layout file path to the swift-frontend
when distributed caching is enabled. In order to properly support path
remapping, the legacy layout file is ingested on demand (thus it doesn't
need to be ingested for module compilation), and the remapped path is
communicated to swift-front via frontend flag.
rdar://162793678
This change introduces a thread-safe version of the 'SerializedDiagnosticConsumer' and refactors scanning compilation instance creation code to ensure this consumer gets added when the scanner query configuration command-line includes '-serialized-diagnostics-path' option.
Previously, frequently-used methods like 'getAllDependencies' and 'getAllClangDependencies' had to aggregate (copy) multiple collections stored in a 'ModuleDependencyInfo' into a new result array to present to the client. These methods have been refactored to instead return an iterable joined view of the constituent collections.
This is required in order to always have computed the set of visible Clang modules for each Swift module in the graph. Otherwise when some Clang module gets cached as a transitive dependency from a query and is later looked up as a direct dependency, there will not be any computed visible modules set.
The note will point the user to where the "other" module with the same name is located and mention whether it is an SDK module. This is nice to have in various circumstances where developers attempt to define a module with the same name as a Swift module that already exists on their search paths, for example in the SDK.
Move per-query state out of ScanningService. There is still a check to
make sure the CASOptions are matching between queries because of the
requirement on clang scanner. Otherwise, the scanning service should
contain no per-query information anymore.
Resolves: https://github.com/swiftlang/swift/issues/82490
While this made sense in the distant past where the scanning service provided backing storage for the dependency cache, it no longer does so and now makes for awkard layering where clients get at the service via the cache. Now the cache is a simple data structure while all the clients that need access to the scanning service will get it explicitly.
- 'SwiftModuleScanner' will now be owned directly by the 'ModuleDependencyScanningWorker' and will contain all the necessary custom logic, instead of being instantiated by the module interface loader for each query
- Moves ownership over module output path and sdk module output path directly into the scanning worker, instead of the cache
This was used a long time ago for a design of a scanner which could rely on the client to specify that some modules *will be* present at a given location but are not yet during the scan. We have long ago determined that the scanner must have all modules available to it at the time of scan for soundness. This code has been stale for a couple of years and it is time to simplify things a bit by deleting it.
Otherwise querying this clang module, e.g. from the corresponding Swift overlay's underlying module import, will fail, since no such module exists.
Resolves rdar://151718115
Using IncludeTree::FileList to concat the include tree file systems that
are passed on the command-line. This significantly reduce the
command-line size, and also makes the cache key computation a lot
faster.
rdar://148752988
When we discover a textual module dependency which is a module which was not originally built from source using C++ interop (specifying '-formal-cxx-interoperability-mode=off'), avoid looking up the C++ standard library Swift overlay for it. This is required for the case of the 'Darwin' module, for example, which includes headers which map to C++ stdlib headers when the compiler is operating in C++ interop mode, but the C++ standard library Swift overlay module itself depends on 'Darwin', which results in a cycle. To resolve such situations, we can rely on the fact that Swift textual interfaces of modules which were not built with C++ interop must be able to build without importing the C++ standard library Swift overlay, so we avoid specifying it as a dependency for such modules.
The primary source module, as well as Swift textual module dependencies which *were* built with C++ interop will continue getting a direct depedency of the 'CxxStdlib' Swift module.
Resolves rdar://150222155
Improve diagnostics message for swift caching build by trying to emit
the diagnostics early when there is more context to differentiate the
different kind of problems.
After the improvement, CAS Error should be more closer to when there is
functional problem with the CAS, rather than mixing in other kinds of
problem (like scanning dependency failures) when operating with a CAS.
rdar://145676736
For the main source module, provide info on which dependencies are directly imported into the user program, explicitly ('import' statement) or implicitly (e.g. stdlib). Thist list does not include Swift overlay dependencies, cross-import dependencies, bridging header dependencies.
Currently, the macro plugin options are included as cache key and the
absolute path of the plugin executable and library will affect cache
hit, even the plugin itself is identical.
Using the new option `-resolved-plugin-validation` flag, the macro
plugin paths are remapped just like the other paths during dependency
scanning. `swift-frontend` will unmap to its original path during the
compilation, make sure the content hasn't changed, and load the plugin.
It also hands few other corner cases for macro plugins:
* Make sure the plugin options in the swift module is prefix mapped.
* Make sure the remarks of the macro loading is not cached, as the
mesasge includes the absolute path of the plugin, and is not
cacheable.
rdar://148465899
With '-sdk-module-cache-path', Swift textual interfaces found in the SDK will be built into a separate SDK-specific module cache.
Clang modules are not yet affected by this change, pending addition of the required API.
Make `-enable-deterministic-check` a driver option and teach dependency
scanner to propagate the option to explicit module build commmands. This
allows to the option to check every build output from the compiler is
deterministic.
https://github.com/swiftlang/swift/pull/79297 implemented current working directory pruning but left some unnecessary code
that computes Swift interface module output path prematurely. This PR removes the code that computes the output path too
early. The `ExplicitModuleDependencyResolver` now adds the path to the command line after it can correctly compute it.
Context: https://github.com/swiftlang/swift/pull/79297/files#r1955314542
Da-hye0
Argyrios Kyrtzidis
Steven Wu
Artem Chikin
Hiroshi Yamauchi
Qiongsi Wu
Adrian Prantl
Qiongsi Wu