This refines how bridging header chaining is done, mainly fixing two
problems:
* On Windows, `#import` statement is not supported. Instead of relying
on #import to workaround bridging header missing header guard, do a
simple deduplication in header generation.
* The __has_include check can failed the dependency scanner if the search
path hits `permission_denied` error. In that case, we cannot rely on
the clang dependeny scanner to check if the bridging header exists or
not. Swift dependeny scanner need to know how to reverse mapping and
check before generating the header.
rdar://175196897
All `ModuleDependencyScanningWorker`s previously shared the parent scanner's
`SourceManager`. `SourceManager` contains mutable caches that are not
thread-safe, so concurrent worker access allowed for a data race that could
corrupt the heap and crash in unrelated code (observed as an abort in
`json::Value::destroy()` during `parseDarwinSDKInfo` on a worker thread).
Give each worker a private `SourceManager` initialized from the same
(thread-safe, refcounted) VFS.
Speculatively resolves rdar://174752231
Make sure the CAS instances are not mixed during dependency scanning,
especially when used from libSwiftScan C APIs.
For clang scanning service and file system, it should always be created
from the global CAS instance.
For CAS instance inside the swift instance when performing dependency
scanning, it reuse the CAS instance from global as well if that is
already initialized.
rdar://173703843
Compute and propagate the library level (api/spi/ipi) of each module
dependency through the dependency scanner so that the compiler can
correctly enforce private module import diagnostics in CAS mode, where
path-based SPI detection fails because CAS abstracts file paths to
content IDs.
Swift modules:
- Detect library level from the module interface path using
libraryLevelFromPath() during scanning, for both textual (.swiftinterface)
and binary (.swiftmodule) Swift modules.
Clang modules:
- Expose ModuleMapIsPrivate from clang::Module in ModuleDeps via the
dependency scanning infrastructure.
- Set library level for clang modules in bridgeClangModuleDependency()
using ModuleMapIsPrivate (catches module.private.modulemap in any
SDK location) and libraryLevelFromPath() on the module map file
(catches modules under PrivateFrameworks directories).
The library level is:
- Stored in ModuleDependencyInfo and serialized in the module dependency
cache (format version bumped to v8).
- Exposed through the swiftscan C API via a new
swiftscan_module_info_get_library_level() function (API minor version
bumped to 3).
- Emitted in the dependency scanner JSON output as "libraryLevel" for
all module kinds (Swift textual, Swift binary, Clang, and main module).
- Parsed from the explicit module map JSON by ExplicitModuleMapParser
for both Swift (ExplicitSwiftModuleInputInfo) and Clang
(ExplicitClangModuleInputInfo) modules.
- Looked up in ModuleLibraryLevelRequest via
ASTContext::getExplicitModuleLibraryLevel(name, isClang), which
consults the appropriate map (Swift or Clang) based on module kind.
rdar://172693314
Assisted-By: Claude
Dependency Scanning is recursive over discovered Swift module overlays and cross-import overlays. In the main 'resolveImportedModuleDependencies' routine, after all Swift and Clang dependencies of the main module are discovered, the scanner looks up Swift overlays for discovered Clang modules. For each such Swift overlay, the scanner will then proceed to call 'resolveImportedModuleDependencies' on the overlay module.
On these subsequent recursive calls to 'resolveImportedModuleDependencies', Clang dependency resolution will re-query all imports of the overlay module which do not resolve to Swift modules, even if they had been queried previously in a parent invocation. This change adds the ability to re-use previously-queried-and-cached Clang module dependency information by having the dependency cache also store the set of visible modules which resulted from each by-name lookup.
<!--
If this pull request is targeting a release branch, please fill out the
following form:
https://github.com/swiftlang/.github/blob/main/PULL_REQUEST_TEMPLATE/release.md?plain=1
Otherwise, replace this comment with a description of your changes and
rationale. Provide links to external references/discussions if
appropriate.
If this pull request resolves any GitHub issues, link them like so:
Resolves <link to issue>, resolves <link to another issue>.
For more information about linking a pull request to an issue, see:
https://docs.github.com/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue
-->
<!--
Before merging this pull request, you must run the Swift continuous
integration tests.
For information about triggering CI builds via @swift-ci, see:
https://github.com/apple/swift/blob/main/docs/ContinuousIntegration.md#swift-ci
Thank you for your contribution to Swift!
-->
This PR teaches Swift to take advantage of the new clang dependency
scanning API to use a single clang compiler instance per dependency
scanning worker to perform by-name queries.
rdar://136303612
This change adds collection of three metrics to the scanner:
- number of Swift module lookups
- number of named Clang module lookups
- recorded number of Clang modules which were imported into a Swift module by name
It introduces '-Rdependency-scan', which acts as a super-set flag to the existing '-Rdependency-scan-cache' and adds emission of the above metrics as remarks when this flag is enabled. Followup changes will add further remarks about dependency scanner progress.
In 'resolveSwiftOverlayDependencies', instead of calling into 'resolveImportedModuleDependencies' for every newly-discovered Swift overlay dependency, gather all collecected overlay dependencies under one umbrella dummy query module and execute a single call into 'resolveImportedModuleDependencies' which will cover all of them.
With this change, we not only lookup all the various Swift module overlays' imports in parallel, but also all of their aggregate respective Clang dependencies are queried in one shot as well.
Previously, with the change to bridge Clang dependency scanning results on-demand, the scanner would execute Clang dependency scanning queries for each unresolved import, in parallel, and aggregate all of the results to post-process (including on-demand bridging) later. As a consequence of that change, all of the Clang scanner queries' results ('TranslationUnitDeps') got aggregated during a scan and had their lifetimes extended until a later point when they got processed and added to the scanner's cache.
This change refactors the Clang dependency scanner invocation to, upon query completion, accumulate only the 'ModuleDeps' nodes which have not been registered by a prior scan, discarding the rest of the 'TranslationUnitDeps' graph. The arrgegated 'ModuleDeps' objects are still bridged on-demand downstream.
This change further splits up the 'resolveAllClangModuleDependencies' method's functionality to improve readability and maintainability, into:
- 'gatherUnresolvedImports' method which collects all of collected Swift dependents' imports which did not get resolved to Swift dependencies
- 'performParallelClangModuleLookup' which actually executes the parallel queries and includes the new logic described above
- 'cacheComputedClangModuleLookupResults' method which takes the result of the parallel Clang scanner query and records in in the Swift scanner cache
- 'reQueryMissedModulesFromCache' method which covers the scenario where Clang scanner query returned no result because either the dependency can only be found transitively, or the query is for a dependency previously-queried.
To support distributed caching for targets that need to use legacy
layout file, the path of legacy layout needs to be remapped.
Current handling of the legacy layout file is to ingest all layout files
to the CAS FileSystem as part of the compiler resource files. But it
cannot convey remapped legacy layout file path to the swift-frontend
when distributed caching is enabled. In order to properly support path
remapping, the legacy layout file is ingested on demand (thus it doesn't
need to be ingested for module compilation), and the remapped path is
communicated to swift-front via frontend flag.
rdar://162793678
Otherwise, when multiple workers encounter a diagnostic simultaneously we can encounter races which lead to corrupted diagnostic data or crashes
Resolves rdar://159598539
This moves the functionality of 'bridgeClangModuleDependency' into a utility in the main scanner class because it relies on various objects whose lifetime is already tied to the scanner itself.
Previously, frequently-used methods like 'getAllDependencies' and 'getAllClangDependencies' had to aggregate (copy) multiple collections stored in a 'ModuleDependencyInfo' into a new result array to present to the client. These methods have been refactored to instead return an iterable joined view of the constituent collections.
Previously Swift overlay lookup was performed for every directly and transitively-imported Clang module.
https://github.com/llvm/llvm-project/pull/147969 introduced the concept of "visible" Clang modules from a given named Clang dependency scanner query which closely maps to the set of modules for which Swift will attempt to load a Swift overlay. This change switches overlay querying to apply only to the set of such visible modules.
Resolves rdar://144797648
When querying a Swift module, the scanner now also keeps track of all discovered candidate binary modules which are not compatible with current compilation.
- If a Swift dependency is successfully resolved to a compatible binary module or a textual interface, a warning is emitted for every incompatible binary Swift module discovered along the way.
- If a Swift dependency is not resolved, but incompatible module candidates were found, an error is emitted - while it is likely that the scan would fail downstream, it is also possible that an underlying Clang module dependency (with the same name) is successfuly resolved and the Swift lookup failure is ignored, which is still going to lead to failures most of the time if the client code assumes the presence of the Swift overlay module in this scenario.
This change refactors common error reporting by the scanner into a 'ModuleDependencyIssueReporter' class, which also keeps track of all diagnosed failed lookups to avoid repeating diagnostics.
Move per-query state out of ScanningService. There is still a check to
make sure the CASOptions are matching between queries because of the
requirement on clang scanner. Otherwise, the scanning service should
contain no per-query information anymore.
Resolves: https://github.com/swiftlang/swift/issues/82490
While this made sense in the distant past where the scanning service provided backing storage for the dependency cache, it no longer does so and now makes for awkard layering where clients get at the service via the cache. Now the cache is a simple data structure while all the clients that need access to the scanning service will get it explicitly.
- 'SwiftModuleScanner' will now be owned directly by the 'ModuleDependencyScanningWorker' and will contain all the necessary custom logic, instead of being instantiated by the module interface loader for each query
- Moves ownership over module output path and sdk module output path directly into the scanning worker, instead of the cache
Unlike with implicitly-built modules (prior to Swift 6 mode), explicitly-built modules require that all search paths be specified explicitly and no longer inherit search paths serialized into discovered Swift binary modules. This behavior was never intentional and is considered a bug. This change adds a diagnostic note to a scan failure: for each binary Swift module dependency, the scanner will attempt to execute a dependency scanning query for each serialized search path inside that module. If such diagnostic query returns a result, a diagnostic will be emitted to inform the user that the dependency may be found in the search path configuration of another Swift binary module dependency, specifying which search path contains the "missing" module, and stating that such search paths are not automatically inherited by the current compilation.
Otherwise querying this clang module, e.g. from the corresponding Swift overlay's underlying module import, will fail, since no such module exists.
Resolves rdar://151718115
With '-sdk-module-cache-path', Swift textual interfaces found in the SDK will be built into a separate SDK-specific module cache.
Clang modules are not yet affected by this change, pending addition of the required API.
Previous implementation took the entire transitive dependency set and cross-referenced all of its members to determine which ones introduce requried cross-import overlays. That implementation differed from the cross-import overlay loading logic during source compilation, where a corrsponding cross-import overlay module is only requested if the two constituent modules are reachable via direct 'import's from *the same source file*. Meaning the dependency scanner before this change would report cross-import overlay dependencies which never got loaded by the corresponding client source compile.
This change implements a new implementation of cross-import overlay discovery which first computes sub-graphs of module dependencies directly reachable by 'import's for each source file of the module under scan and then performs pairwise cross-import overlay query per each such sub-graph.
Resolves rdar://145157171
`llvm::StdThreadPool` is only available when LLVM_ENABLE_THREADS is enabled.
LLVM defines `DefaultThreadPool` type alias, which is always available
regardless of the threads enabled. This change allows building the library
without threads enabled, especially for Emscripten toolchain.
```
#if LLVM_ENABLE_THREADS
using DefaultThreadPool = StdThreadPool;
#else
using DefaultThreadPool = SingleThreadExecutor;
#endif
```
Add ability to automatically chaining the bridging headers discovered from all
dependencies module when doing swift caching build. This will eliminate all
implicit bridging header imports from the build and make the bridging header
importing behavior much more reliable, while keep the compatibility at maximum.
For example, if the current module A depends on module B and C, and both B and
C are binary modules that uses bridging header, when building module A,
dependency scanner will construct a new header that chains three bridging
headers together with the option to build a PCH from it. This will make all
importing errors more obvious while improving the performance.
This set, belonging to 'ModuleDependenciesCache', is only updated in a critical section behind a lock in the scanner. However, it is queried unsynchronized inside the Clang scanner itself. If an update causes a re-hash to happen, chaose can ensue with concurrent lookups.
Since this set only affects the produced set of results from teh Clang scanning query, we should simply pass in an immutable copy to scanning queries and rely on downstream de-duplication of scanning results.
With this, we can avoid passing in the reference to `ModuleDependenciesCache` to the 'scanFilesystemFor*ModuleDependency' altogether.
Resolves rdar://139414443
This change refactors the top-level dependency scanning flow to follow the following procedure:
Scan():
1. From the source target under scan, query all imported module identifiers for a *Swift* module. Leave unresolved identifiers unresolved. Proceed transitively to build a *Swift* module dependency graph.
2. Take every unresolved import identifier in the graph from (1) and, assuming that it must be a Clang module, dispatch all of them to be queried in-parallel by the scanner's worker pool.
3. Resolve bridging header Clang module dpendencies
4. Resolve all Swift overlay dependencies, relying on all Clang modules collected in (2) and (3)
5. For the source target under scan, use all of the above discovered module dependencies to resolve all cross-import overlay dependencies
Add support for cross import modules by ingesting swiftoverlay files for
the cross import into CAS file system.
The long-term better fix will be just passing the cross import
information from scanner to swift-frontend so frontend doesn't need to
read overlay files again to figure out the cross import module.
rdar://123839248
Otherwise they may have module dependencies of their own which will not be detected by the scanner and included in the list of explicit inputs for compilation.
'ModuleDependencyScanner' maintains a Thread Pool along with a pool of workers
which are capable of executing a filesystem lookup of a named module dependency.
When resolving imports of a given Swift module, each import's resolution
operation can be issued asunchronously.
Artem Chikin
Steven Wu
Qiongsi Wu
Yuta Saito
Ben Barham