During explicit module build, teach dependency scanner to emit the
module trace file instead of each following compile job command. This
reduces the duplicated info, and allows supporting fully cached build
that only loads module from CAS thus cannot produce the path to the
original module file on disk.
rdar://170007480
Partially revert https://github.com/swiftlang/swift/pull/86309. Keep the
SDKInfo parsing in the ASTContext separately from the search path
configuration. This allows the availablity checking to load SDKSettings
from the correct VFS.
rdar://169886913
Previously, const-values output is not supported correctly as it is not
captured correctly as a dependency. The load of the input JSON file is
loaded via file system directly, that can cause issues like:
* False cache hit when the file is changed
* Cannot be prefix mapped to allow distributed caching
Try to support the input file by:
* Correctly capture the input
* Create a new driver flag `-const-gather-protocols-list` that can do
path remapping correctly in swift-driver
rdar://169109358
https://github.com/swiftlang/swift/pull/85555 taught Swift to use a
shared clang compiler instance that is initialized once and reused for
multiple by-name lookups. This unfortunately led to a few internal
project build failures.
This PR reverts back to use clang dependency scanning tool's old by-name
lookup APIs while we investigate the failures.
rdar://168709914
Use it to avoid unnecessary re-queries of Swift dependencies during a given scanning action.
Refactor 'hasSwiftDependency' API into 'hasQueriedSwiftDependency' to instead report whether or not the cache has recorded a prior lookup of a given Swift dependency.
Some Darwin platforms like DriverKit use a system prefix on all of their search paths. Even though DriverKit isn't supported, add support to get the system prefix from SDKSettings when constructing the default search paths.
This requires the DarwinSDKInfo to be gotten earlier in CompilerInvocation, pass that down to ASTContext through CompilerInstance.
-platform-availability-inheritance-map-path is no longer needed to support visionOS in tests, remove that and its supporting code that gets an alternative DarwinSDKInfo.
rdar://166277280
This reverts commit e60ae24052 and fix
non-deterministic failures introduced by the commit.
Fix two issues when attempting to testing parallel scanning using
`swift-scan-test` tools:
* Make sure the BumpPtrAllocator in ScanningService is thread-safe so
there are no race condition when a new slab is allocated.
* Make sure the output of `swift-scan-test` only written from one
thread. This prevents some race conditions when writing to the same
raw_fd_ostream.
rdar://167760262
Fix a programming mistake when chaining bridging header for the main
module. Main module never has the binary module, thus it always need to
chain the bridging header content directly without falling back to
embedded binary module. Previously, the scanner was wrongly error out
because it failed to locate the binary module for main module.
rdar://167707148
Currently, dependency scanner is not reporting the redirecting files
that are baked inside swift-frontend for platform support. This causes
dependency scanner returns virtual path for those files, and
swift-driver/build-system will not be able to correct validate the files
on incremental build, causing incremental build to be almost clean
builds.
This behavior issue is caused by the dependency scanning file system
layer inside clang dependency scanner that caches stats. If the
redirecting files are created underneath the layer, the real path is
lost. This fixes the issue by moving the redirecting files above the
caching layer using `-ivfsoverlay` option.
In addition to that, this commit also unifies how clang importer and
clang dependency scanner initiate the VFS, making the logic much
simpler.
Dependency Scanning is recursive over discovered Swift module overlays and cross-import overlays. In the main 'resolveImportedModuleDependencies' routine, after all Swift and Clang dependencies of the main module are discovered, the scanner looks up Swift overlays for discovered Clang modules. For each such Swift overlay, the scanner will then proceed to call 'resolveImportedModuleDependencies' on the overlay module.
On these subsequent recursive calls to 'resolveImportedModuleDependencies', Clang dependency resolution will re-query all imports of the overlay module which do not resolve to Swift modules, even if they had been queried previously in a parent invocation. This change adds the ability to re-use previously-queried-and-cached Clang module dependency information by having the dependency cache also store the set of visible modules which resulted from each by-name lookup.
<!--
If this pull request is targeting a release branch, please fill out the
following form:
https://github.com/swiftlang/.github/blob/main/PULL_REQUEST_TEMPLATE/release.md?plain=1
Otherwise, replace this comment with a description of your changes and
rationale. Provide links to external references/discussions if
appropriate.
If this pull request resolves any GitHub issues, link them like so:
Resolves <link to issue>, resolves <link to another issue>.
For more information about linking a pull request to an issue, see:
https://docs.github.com/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue
-->
<!--
Before merging this pull request, you must run the Swift continuous
integration tests.
For information about triggering CI builds via @swift-ci, see:
https://github.com/apple/swift/blob/main/docs/ContinuousIntegration.md#swift-ci
Thank you for your contribution to Swift!
-->
This PR teaches Swift to take advantage of the new clang dependency
scanning API to use a single clang compiler instance per dependency
scanning worker to perform by-name queries.
rdar://136303612
This change adds collection of three metrics to the scanner:
- number of Swift module lookups
- number of named Clang module lookups
- recorded number of Clang modules which were imported into a Swift module by name
It introduces '-Rdependency-scan', which acts as a super-set flag to the existing '-Rdependency-scan-cache' and adds emission of the above metrics as remarks when this flag is enabled. Followup changes will add further remarks about dependency scanner progress.
In 'resolveSwiftOverlayDependencies', instead of calling into 'resolveImportedModuleDependencies' for every newly-discovered Swift overlay dependency, gather all collecected overlay dependencies under one umbrella dummy query module and execute a single call into 'resolveImportedModuleDependencies' which will cover all of them.
With this change, we not only lookup all the various Swift module overlays' imports in parallel, but also all of their aggregate respective Clang dependencies are queried in one shot as well.
We currently support static linking swiftCxxStdlib only on Windows. When
C++ interop is enabled but swiftCxxStdlib isn't actually used or only
when its underlying std module is used, we incorrectly emitted the
dynamic version of the lib name and caused the link error. This change
fixes it by always adding it to the dependency.
Issue https://github.com/swiftlang/swift/issues/85876
During parallel clang module dependency resolution, an unintended side-effect of https://github.com/swiftlang/swift/pull/84929 is that we stopped uniquing the module identifiers we query to the Clang dependency scanner.
This change ensures we do not query the same identifier more than once.
Resolves rdar://165133617
Explicit module builds currently fail on Windows because
direct-clang-cc1-module-build emit-pcm commands take overlaid system
module map files as inputs but miss the clang VFS overlay. This change
adds the overlay and fixes explicit module builds on Windows.
Previously, with the change to bridge Clang dependency scanning results on-demand, the scanner would execute Clang dependency scanning queries for each unresolved import, in parallel, and aggregate all of the results to post-process (including on-demand bridging) later. As a consequence of that change, all of the Clang scanner queries' results ('TranslationUnitDeps') got aggregated during a scan and had their lifetimes extended until a later point when they got processed and added to the scanner's cache.
This change refactors the Clang dependency scanner invocation to, upon query completion, accumulate only the 'ModuleDeps' nodes which have not been registered by a prior scan, discarding the rest of the 'TranslationUnitDeps' graph. The arrgegated 'ModuleDeps' objects are still bridged on-demand downstream.
This change further splits up the 'resolveAllClangModuleDependencies' method's functionality to improve readability and maintainability, into:
- 'gatherUnresolvedImports' method which collects all of collected Swift dependents' imports which did not get resolved to Swift dependencies
- 'performParallelClangModuleLookup' which actually executes the parallel queries and includes the new logic described above
- 'cacheComputedClangModuleLookupResults' method which takes the result of the parallel Clang scanner query and records in in the Swift scanner cache
- 'reQueryMissedModulesFromCache' method which covers the scenario where Clang scanner query returned no result because either the dependency can only be found transitively, or the query is for a dependency previously-queried.
To support distributed caching for targets that need to use legacy
layout file, the path of legacy layout needs to be remapped.
Current handling of the legacy layout file is to ingest all layout files
to the CAS FileSystem as part of the compiler resource files. But it
cannot convey remapped legacy layout file path to the swift-frontend
when distributed caching is enabled. In order to properly support path
remapping, the legacy layout file is ingested on demand (thus it doesn't
need to be ingested for module compilation), and the remapped path is
communicated to swift-front via frontend flag.
rdar://162793678
We have adopters who are relying on directly importing the underlying Clang module in the presence of incompatible Swift modules.
Resolves rdar://162549210
Adjust the rule for how bridging header chaining is performed to
increase compatibilities with existing project configuration.
Previously in #84442, bridging header chaining was done via content
concatenation, rather than `#include` via absolute path, to enable prefix
mapping to drop references to absolute path. This triggers an
incompatibility since the directory of the header file is also
considered as part of the search path when resolve the include
directives in the file. Simple concatenation will result in some
`#include` can no longer be resolved.
Now relax the rule to do header content concatenation only when prefix
map is used. This will keep the breakage minimal, and the breakage can
always be fixed by additional include paths when turning on prefix
mapping.
In the future, we should move towards internal bridging header
implementation to avoid the need of chaining completely.
rdar://161854282
To allow prefix mapping of the bridging header to achieve cache hit when
source files are located in different location, the generated chained
bridging header should not include absolute paths of the headers. Fix
the problem by concat the chained bridging header together.
Fixes: https://github.com/swiftlang/swift/issues/84088
Avoid Swift Overlay lookup for the underlying clang module of a known Swift module.
i.e. When computing set of Swift Overlay dependencies for module 'A', which depends on a Clang module 'A', ensure we don't lookup Swift module 'A' itself here - this can lead to bizarre interactions where the source module under scan is queried as a dependency of itself.
Resolves rdar://159706486
This change introduces a thread-safe version of the 'SerializedDiagnosticConsumer' and refactors scanning compilation instance creation code to ensure this consumer gets added when the scanner query configuration command-line includes '-serialized-diagnostics-path' option.
