<!--
If this pull request is targeting a release branch, please fill out the
following form:
https://github.com/swiftlang/.github/blob/main/PULL_REQUEST_TEMPLATE/release.md?plain=1
Otherwise, replace this comment with a description of your changes and
rationale. Provide links to external references/discussions if
appropriate.
If this pull request resolves any GitHub issues, link them like so:
Resolves <link to issue>, resolves <link to another issue>.
For more information about linking a pull request to an issue, see:
https://docs.github.com/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue
-->
<!--
Before merging this pull request, you must run the Swift continuous
integration tests.
For information about triggering CI builds via @swift-ci, see:
https://github.com/apple/swift/blob/main/docs/ContinuousIntegration.md#swift-ci
Thank you for your contribution to Swift!
-->
This PR teaches Swift to take advantage of the new clang dependency
scanning API to use a single clang compiler instance per dependency
scanning worker to perform by-name queries.
rdar://136303612
This change adds collection of three metrics to the scanner:
- number of Swift module lookups
- number of named Clang module lookups
- recorded number of Clang modules which were imported into a Swift module by name
It introduces '-Rdependency-scan', which acts as a super-set flag to the existing '-Rdependency-scan-cache' and adds emission of the above metrics as remarks when this flag is enabled. Followup changes will add further remarks about dependency scanner progress.
In 'resolveSwiftOverlayDependencies', instead of calling into 'resolveImportedModuleDependencies' for every newly-discovered Swift overlay dependency, gather all collecected overlay dependencies under one umbrella dummy query module and execute a single call into 'resolveImportedModuleDependencies' which will cover all of them.
With this change, we not only lookup all the various Swift module overlays' imports in parallel, but also all of their aggregate respective Clang dependencies are queried in one shot as well.
We currently support static linking swiftCxxStdlib only on Windows. When
C++ interop is enabled but swiftCxxStdlib isn't actually used or only
when its underlying std module is used, we incorrectly emitted the
dynamic version of the lib name and caused the link error. This change
fixes it by always adding it to the dependency.
Issue https://github.com/swiftlang/swift/issues/85876
During parallel clang module dependency resolution, an unintended side-effect of https://github.com/swiftlang/swift/pull/84929 is that we stopped uniquing the module identifiers we query to the Clang dependency scanner.
This change ensures we do not query the same identifier more than once.
Resolves rdar://165133617
Explicit module builds currently fail on Windows because
direct-clang-cc1-module-build emit-pcm commands take overlaid system
module map files as inputs but miss the clang VFS overlay. This change
adds the overlay and fixes explicit module builds on Windows.
Previously, with the change to bridge Clang dependency scanning results on-demand, the scanner would execute Clang dependency scanning queries for each unresolved import, in parallel, and aggregate all of the results to post-process (including on-demand bridging) later. As a consequence of that change, all of the Clang scanner queries' results ('TranslationUnitDeps') got aggregated during a scan and had their lifetimes extended until a later point when they got processed and added to the scanner's cache.
This change refactors the Clang dependency scanner invocation to, upon query completion, accumulate only the 'ModuleDeps' nodes which have not been registered by a prior scan, discarding the rest of the 'TranslationUnitDeps' graph. The arrgegated 'ModuleDeps' objects are still bridged on-demand downstream.
This change further splits up the 'resolveAllClangModuleDependencies' method's functionality to improve readability and maintainability, into:
- 'gatherUnresolvedImports' method which collects all of collected Swift dependents' imports which did not get resolved to Swift dependencies
- 'performParallelClangModuleLookup' which actually executes the parallel queries and includes the new logic described above
- 'cacheComputedClangModuleLookupResults' method which takes the result of the parallel Clang scanner query and records in in the Swift scanner cache
- 'reQueryMissedModulesFromCache' method which covers the scenario where Clang scanner query returned no result because either the dependency can only be found transitively, or the query is for a dependency previously-queried.
To support distributed caching for targets that need to use legacy
layout file, the path of legacy layout needs to be remapped.
Current handling of the legacy layout file is to ingest all layout files
to the CAS FileSystem as part of the compiler resource files. But it
cannot convey remapped legacy layout file path to the swift-frontend
when distributed caching is enabled. In order to properly support path
remapping, the legacy layout file is ingested on demand (thus it doesn't
need to be ingested for module compilation), and the remapped path is
communicated to swift-front via frontend flag.
rdar://162793678
We have adopters who are relying on directly importing the underlying Clang module in the presence of incompatible Swift modules.
Resolves rdar://162549210
Adjust the rule for how bridging header chaining is performed to
increase compatibilities with existing project configuration.
Previously in #84442, bridging header chaining was done via content
concatenation, rather than `#include` via absolute path, to enable prefix
mapping to drop references to absolute path. This triggers an
incompatibility since the directory of the header file is also
considered as part of the search path when resolve the include
directives in the file. Simple concatenation will result in some
`#include` can no longer be resolved.
Now relax the rule to do header content concatenation only when prefix
map is used. This will keep the breakage minimal, and the breakage can
always be fixed by additional include paths when turning on prefix
mapping.
In the future, we should move towards internal bridging header
implementation to avoid the need of chaining completely.
rdar://161854282
To allow prefix mapping of the bridging header to achieve cache hit when
source files are located in different location, the generated chained
bridging header should not include absolute paths of the headers. Fix
the problem by concat the chained bridging header together.
Fixes: https://github.com/swiftlang/swift/issues/84088
Avoid Swift Overlay lookup for the underlying clang module of a known Swift module.
i.e. When computing set of Swift Overlay dependencies for module 'A', which depends on a Clang module 'A', ensure we don't lookup Swift module 'A' itself here - this can lead to bizarre interactions where the source module under scan is queried as a dependency of itself.
Resolves rdar://159706486
This change introduces a thread-safe version of the 'SerializedDiagnosticConsumer' and refactors scanning compilation instance creation code to ensure this consumer gets added when the scanner query configuration command-line includes '-serialized-diagnostics-path' option.
Otherwise, when multiple workers encounter a diagnostic simultaneously we can encounter races which lead to corrupted diagnostic data or crashes
Resolves rdar://159598539
This change re-enables the new semantic of looking up Swift Overlay modules only for "visible" clang modules.
We have investigated the failures we were seeing with this change enabled and addressed them with project-side fixes.
SwiftVerifyEmittedModuleInterface job is configured to be built within
the SubInvocation and it needs to inherit the caching replay prefix map
in order to load macro plugin library correctly.
rdar://158692095
This moves the functionality of 'bridgeClangModuleDependency' into a utility in the main scanner class because it relies on various objects whose lifetime is already tied to the scanner itself.
Previously, frequently-used methods like 'getAllDependencies' and 'getAllClangDependencies' had to aggregate (copy) multiple collections stored in a 'ModuleDependencyInfo' into a new result array to present to the client. These methods have been refactored to instead return an iterable joined view of the constituent collections.
In addition to skipping it on textual Swift module dependencies which were built without C++ interop enabled, also skip it over similarly on binary Swift dependencies
We have identified scenarios where this is insufficient and compilation still relies on the presence of certain overlay modules this excludes. Reverting to prior lookup logic while we investigate.
Resolves rdar://157519278
This is required in order to always have computed the set of visible Clang modules for each Swift module in the graph. Otherwise when some Clang module gets cached as a transitive dependency from a query and is later looked up as a direct dependency, there will not be any computed visible modules set.
Previously Swift overlay lookup was performed for every directly and transitively-imported Clang module.
https://github.com/llvm/llvm-project/pull/147969 introduced the concept of "visible" Clang modules from a given named Clang dependency scanner query which closely maps to the set of modules for which Swift will attempt to load a Swift overlay. This change switches overlay querying to apply only to the set of such visible modules.
Resolves rdar://144797648
Previously this flag was only used to pass explicit dependencies to compilation tasks. This change adds support for the dependency scanner to also consider these inputs when resolving dependencies.
Resolves https://github.com/swiftlang/swift-driver/issues/1951
When querying a Swift module, the scanner now also keeps track of all discovered candidate binary modules which are not compatible with current compilation.
- If a Swift dependency is successfully resolved to a compatible binary module or a textual interface, a warning is emitted for every incompatible binary Swift module discovered along the way.
- If a Swift dependency is not resolved, but incompatible module candidates were found, an error is emitted - while it is likely that the scan would fail downstream, it is also possible that an underlying Clang module dependency (with the same name) is successfuly resolved and the Swift lookup failure is ignored, which is still going to lead to failures most of the time if the client code assumes the presence of the Swift overlay module in this scenario.
This change refactors common error reporting by the scanner into a 'ModuleDependencyIssueReporter' class, which also keeps track of all diagnosed failed lookups to avoid repeating diagnostics.
A prior change ensured that we forego this query when looking up Swift overlays for a textual interface which was built without C++ interop. This change introduced a bug where it also caused us to skip this lookup for the main source module. This commit resolves that by preserving the fix above but also ensuring we perform the lookup for the main source module under scan.
The note will point the user to where the "other" module with the same name is located and mention whether it is an SDK module. This is nice to have in various circumstances where developers attempt to define a module with the same name as a Swift module that already exists on their search paths, for example in the SDK.
