clang's loadModule caches missing-module failures and silently
short-circuits subsequent lookups. The resulting empty messages
leaked past the existing "module 'X' not found" filter, producing
empty "clang dependency scanning failure:" errors, one per
duplicate consumer.
This refines how bridging header chaining is done, mainly fixing two
problems:
* On Windows, `#import` statement is not supported. Instead of relying
on #import to workaround bridging header missing header guard, do a
simple deduplication in header generation.
* The __has_include check can failed the dependency scanner if the search
path hits `permission_denied` error. In that case, we cannot rely on
the clang dependeny scanner to check if the bridging header exists or
not. Swift dependeny scanner need to know how to reverse mapping and
check before generating the header.
rdar://175196897
All `ModuleDependencyScanningWorker`s previously shared the parent scanner's
`SourceManager`. `SourceManager` contains mutable caches that are not
thread-safe, so concurrent worker access allowed for a data race that could
corrupt the heap and crash in unrelated code (observed as an abort in
`json::Value::destroy()` during `parseDarwinSDKInfo` on a worker thread).
Give each worker a private `SourceManager` initialized from the same
(thread-safe, refcounted) VFS.
Speculatively resolves rdar://174752231
The dependency scanner was not including -target in the Swift frontend
command line for Clang module PCM builds. This caused the frontend to
default to the host triple, producing incorrect "libc not found" warnings
when cross-compiling (e.g. for WebAssembly on Linux).
Also forward -clang-target when present so the frontend correctly
distinguishes Swift and Clang targets in zippered builds.
rdar://174674714
Make sure the CAS instances are not mixed during dependency scanning,
especially when used from libSwiftScan C APIs.
For clang scanning service and file system, it should always be created
from the global CAS instance.
For CAS instance inside the swift instance when performing dependency
scanning, it reuse the CAS instance from global as well if that is
already initialized.
rdar://173703843
Improve bridging header chaining when prefix mapping is used so it
matches the behavior of non-prefix-map and non-caching builds.
This also fixes a corner case where the same bridging header is used
by different modules in the dependency chain, preventing it from being
imported twice.
The improvements are:
* Fully utilize the clang scanner prefix mapping option, which can
restore prefix-mapped paths during scanning to find the real file on
the file system. This allows the generated bridging header to always
reference the header path without worrying about introducing path
dependencies.
* Move the header existence check from a file system access in the
Swift scanner to a `__has_include` check in the clang scanner. This
allows direct header import even when the header path is previously
prefix mapped.
* Use `#import` to chain bridging headers so the same header will not
be imported twice.
rdar://172870182
Compute and propagate the library level (api/spi/ipi) of each module
dependency through the dependency scanner so that the compiler can
correctly enforce private module import diagnostics in CAS mode, where
path-based SPI detection fails because CAS abstracts file paths to
content IDs.
Swift modules:
- Detect library level from the module interface path using
libraryLevelFromPath() during scanning, for both textual (.swiftinterface)
and binary (.swiftmodule) Swift modules.
Clang modules:
- Expose ModuleMapIsPrivate from clang::Module in ModuleDeps via the
dependency scanning infrastructure.
- Set library level for clang modules in bridgeClangModuleDependency()
using ModuleMapIsPrivate (catches module.private.modulemap in any
SDK location) and libraryLevelFromPath() on the module map file
(catches modules under PrivateFrameworks directories).
The library level is:
- Stored in ModuleDependencyInfo and serialized in the module dependency
cache (format version bumped to v8).
- Exposed through the swiftscan C API via a new
swiftscan_module_info_get_library_level() function (API minor version
bumped to 3).
- Emitted in the dependency scanner JSON output as "libraryLevel" for
all module kinds (Swift textual, Swift binary, Clang, and main module).
- Parsed from the explicit module map JSON by ExplicitModuleMapParser
for both Swift (ExplicitSwiftModuleInputInfo) and Clang
(ExplicitClangModuleInputInfo) modules.
- Looked up in ModuleLibraryLevelRequest via
ASTContext::getExplicitModuleLibraryLevel(name, isClang), which
consults the appropriate map (Swift or Clang) based on module kind.
rdar://172693314
Assisted-By: Claude
Migrate all callers of getOrCreateDatabases to either create their own CAS
instances using createDatabases() or to use the ones stored in the
CompilerInstance. Also forwards the instances into the ClangImporter and its
various clang::CompilerInstance instances.
Validated locally that this does not result in any additional calls to
createDatabases across all the swift tests. In fact, there are fewer due to an
improvement on the clange side, so I also checked that the overall reduction
doesn't hide any issues. Looking at individual cas paths that are opened, which
are per-test paths, there are none that open more than before.
We are seeing a huge slowdown in dependency scanning actions due to much more time spend parsing textual interfaces. I have narrowed this down to a large amount of duplicate interface parsing actions stemming from a lack of checking whether or not a given optional import needs to be queried based on whether or not it has already been queried before.
Resovles rdar://172199863
Fix a module build regression in PR86881 that causes an increase of
swift interface compilation tasks due to including the
const-gather-protocol list files into the build dependencies for
swiftinterface files. This causes targets cannot sharing module
dependencies.
This make sures the CASFS system only captures the const-gather-protocol
list in the tasks for main module, not for any of its module
dependencies.
rdar://171506799
Do not search for cross import when building swiftinterfaces. This is
uncessary because:
* The cross import modules are explicitly listed in the interface file
thus there is no need to walk the file system to discover more
cross imports.
* All cross imports are discovered by scanner already and the
dependencies are passed to the interface building job. There is no
need to search file system to find the module to consume.
This reduces the work needed when building interface and fix a bug that
the file system walk when caching is enabled can cause build failures on
windows platform.
During explicit module build, teach dependency scanner to emit the
module trace file instead of each following compile job command. This
reduces the duplicated info, and allows supporting fully cached build
that only loads module from CAS thus cannot produce the path to the
original module file on disk.
rdar://170007480
Partially revert https://github.com/swiftlang/swift/pull/86309. Keep the
SDKInfo parsing in the ASTContext separately from the search path
configuration. This allows the availablity checking to load SDKSettings
from the correct VFS.
rdar://169886913
Previously, const-values output is not supported correctly as it is not
captured correctly as a dependency. The load of the input JSON file is
loaded via file system directly, that can cause issues like:
* False cache hit when the file is changed
* Cannot be prefix mapped to allow distributed caching
Try to support the input file by:
* Correctly capture the input
* Create a new driver flag `-const-gather-protocols-list` that can do
path remapping correctly in swift-driver
rdar://169109358
https://github.com/swiftlang/swift/pull/85555 taught Swift to use a
shared clang compiler instance that is initialized once and reused for
multiple by-name lookups. This unfortunately led to a few internal
project build failures.
This PR reverts back to use clang dependency scanning tool's old by-name
lookup APIs while we investigate the failures.
rdar://168709914
Use it to avoid unnecessary re-queries of Swift dependencies during a given scanning action.
Refactor 'hasSwiftDependency' API into 'hasQueriedSwiftDependency' to instead report whether or not the cache has recorded a prior lookup of a given Swift dependency.
Some Darwin platforms like DriverKit use a system prefix on all of their search paths. Even though DriverKit isn't supported, add support to get the system prefix from SDKSettings when constructing the default search paths.
This requires the DarwinSDKInfo to be gotten earlier in CompilerInvocation, pass that down to ASTContext through CompilerInstance.
-platform-availability-inheritance-map-path is no longer needed to support visionOS in tests, remove that and its supporting code that gets an alternative DarwinSDKInfo.
rdar://166277280
This reverts commit e60ae24052 and fix
non-deterministic failures introduced by the commit.
Fix two issues when attempting to testing parallel scanning using
`swift-scan-test` tools:
* Make sure the BumpPtrAllocator in ScanningService is thread-safe so
there are no race condition when a new slab is allocated.
* Make sure the output of `swift-scan-test` only written from one
thread. This prevents some race conditions when writing to the same
raw_fd_ostream.
rdar://167760262
Fix a programming mistake when chaining bridging header for the main
module. Main module never has the binary module, thus it always need to
chain the bridging header content directly without falling back to
embedded binary module. Previously, the scanner was wrongly error out
because it failed to locate the binary module for main module.
rdar://167707148
Currently, dependency scanner is not reporting the redirecting files
that are baked inside swift-frontend for platform support. This causes
dependency scanner returns virtual path for those files, and
swift-driver/build-system will not be able to correct validate the files
on incremental build, causing incremental build to be almost clean
builds.
This behavior issue is caused by the dependency scanning file system
layer inside clang dependency scanner that caches stats. If the
redirecting files are created underneath the layer, the real path is
lost. This fixes the issue by moving the redirecting files above the
caching layer using `-ivfsoverlay` option.
In addition to that, this commit also unifies how clang importer and
clang dependency scanner initiate the VFS, making the logic much
simpler.
Dependency Scanning is recursive over discovered Swift module overlays and cross-import overlays. In the main 'resolveImportedModuleDependencies' routine, after all Swift and Clang dependencies of the main module are discovered, the scanner looks up Swift overlays for discovered Clang modules. For each such Swift overlay, the scanner will then proceed to call 'resolveImportedModuleDependencies' on the overlay module.
On these subsequent recursive calls to 'resolveImportedModuleDependencies', Clang dependency resolution will re-query all imports of the overlay module which do not resolve to Swift modules, even if they had been queried previously in a parent invocation. This change adds the ability to re-use previously-queried-and-cached Clang module dependency information by having the dependency cache also store the set of visible modules which resulted from each by-name lookup.
<!--
If this pull request is targeting a release branch, please fill out the
following form:
https://github.com/swiftlang/.github/blob/main/PULL_REQUEST_TEMPLATE/release.md?plain=1
Otherwise, replace this comment with a description of your changes and
rationale. Provide links to external references/discussions if
appropriate.
If this pull request resolves any GitHub issues, link them like so:
Resolves <link to issue>, resolves <link to another issue>.
For more information about linking a pull request to an issue, see:
https://docs.github.com/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue
-->
<!--
Before merging this pull request, you must run the Swift continuous
integration tests.
For information about triggering CI builds via @swift-ci, see:
https://github.com/apple/swift/blob/main/docs/ContinuousIntegration.md#swift-ci
Thank you for your contribution to Swift!
-->
This PR teaches Swift to take advantage of the new clang dependency
scanning API to use a single clang compiler instance per dependency
scanning worker to perform by-name queries.
rdar://136303612
This change adds collection of three metrics to the scanner:
- number of Swift module lookups
- number of named Clang module lookups
- recorded number of Clang modules which were imported into a Swift module by name
It introduces '-Rdependency-scan', which acts as a super-set flag to the existing '-Rdependency-scan-cache' and adds emission of the above metrics as remarks when this flag is enabled. Followup changes will add further remarks about dependency scanner progress.
In 'resolveSwiftOverlayDependencies', instead of calling into 'resolveImportedModuleDependencies' for every newly-discovered Swift overlay dependency, gather all collecected overlay dependencies under one umbrella dummy query module and execute a single call into 'resolveImportedModuleDependencies' which will cover all of them.
With this change, we not only lookup all the various Swift module overlays' imports in parallel, but also all of their aggregate respective Clang dependencies are queried in one shot as well.
We currently support static linking swiftCxxStdlib only on Windows. When
C++ interop is enabled but swiftCxxStdlib isn't actually used or only
when its underlying std module is used, we incorrectly emitted the
dynamic version of the lib name and caused the link error. This change
fixes it by always adding it to the dependency.
Issue https://github.com/swiftlang/swift/issues/85876
During parallel clang module dependency resolution, an unintended side-effect of https://github.com/swiftlang/swift/pull/84929 is that we stopped uniquing the module identifiers we query to the Clang dependency scanner.
This change ensures we do not query the same identifier more than once.
Resolves rdar://165133617
Explicit module builds currently fail on Windows because
direct-clang-cc1-module-build emit-pcm commands take overlaid system
module map files as inputs but miss the clang VFS overlay. This change
adds the overlay and fixes explicit module builds on Windows.
Previously, with the change to bridge Clang dependency scanning results on-demand, the scanner would execute Clang dependency scanning queries for each unresolved import, in parallel, and aggregate all of the results to post-process (including on-demand bridging) later. As a consequence of that change, all of the Clang scanner queries' results ('TranslationUnitDeps') got aggregated during a scan and had their lifetimes extended until a later point when they got processed and added to the scanner's cache.
This change refactors the Clang dependency scanner invocation to, upon query completion, accumulate only the 'ModuleDeps' nodes which have not been registered by a prior scan, discarding the rest of the 'TranslationUnitDeps' graph. The arrgegated 'ModuleDeps' objects are still bridged on-demand downstream.
This change further splits up the 'resolveAllClangModuleDependencies' method's functionality to improve readability and maintainability, into:
- 'gatherUnresolvedImports' method which collects all of collected Swift dependents' imports which did not get resolved to Swift dependencies
- 'performParallelClangModuleLookup' which actually executes the parallel queries and includes the new logic described above
- 'cacheComputedClangModuleLookupResults' method which takes the result of the parallel Clang scanner query and records in in the Swift scanner cache
- 'reQueryMissedModulesFromCache' method which covers the scenario where Clang scanner query returned no result because either the dependency can only be found transitively, or the query is for a dependency previously-queried.
To support distributed caching for targets that need to use legacy
layout file, the path of legacy layout needs to be remapped.
Current handling of the legacy layout file is to ingest all layout files
to the CAS FileSystem as part of the compiler resource files. But it
cannot convey remapped legacy layout file path to the swift-frontend
when distributed caching is enabled. In order to properly support path
remapping, the legacy layout file is ingested on demand (thus it doesn't
need to be ingested for module compilation), and the remapped path is
communicated to swift-front via frontend flag.
rdar://162793678
Qiongsi Wu
Ryan Mansfield
Artem Chikin
Steven Wu
Ben Langmuir
Hiroshi Yamauchi
Ian Anderson