For clients, such as the debugger, who do not have access the full
output of the dependency scanner, it is a huger performance and
correctness improvement if each explicitly built Swift module not just
serialized all its Clang .pcm dependencies (via the serialized Clang
compiler invocation) but also its direct Swift module dependencies.
This patch changes the Swift module format to store the absolute path
or cas cache key for each dependency in the INPUT block, and makes
sure the deserialization makes these available to the ESML.
rdar://150969755
Previously, with the change to bridge Clang dependency scanning results on-demand, the scanner would execute Clang dependency scanning queries for each unresolved import, in parallel, and aggregate all of the results to post-process (including on-demand bridging) later. As a consequence of that change, all of the Clang scanner queries' results ('TranslationUnitDeps') got aggregated during a scan and had their lifetimes extended until a later point when they got processed and added to the scanner's cache.
This change refactors the Clang dependency scanner invocation to, upon query completion, accumulate only the 'ModuleDeps' nodes which have not been registered by a prior scan, discarding the rest of the 'TranslationUnitDeps' graph. The arrgegated 'ModuleDeps' objects are still bridged on-demand downstream.
This change further splits up the 'resolveAllClangModuleDependencies' method's functionality to improve readability and maintainability, into:
- 'gatherUnresolvedImports' method which collects all of collected Swift dependents' imports which did not get resolved to Swift dependencies
- 'performParallelClangModuleLookup' which actually executes the parallel queries and includes the new logic described above
- 'cacheComputedClangModuleLookupResults' method which takes the result of the parallel Clang scanner query and records in in the Swift scanner cache
- 'reQueryMissedModulesFromCache' method which covers the scenario where Clang scanner query returned no result because either the dependency can only be found transitively, or the query is for a dependency previously-queried.
To support distributed caching for targets that need to use legacy
layout file, the path of legacy layout needs to be remapped.
Current handling of the legacy layout file is to ingest all layout files
to the CAS FileSystem as part of the compiler resource files. But it
cannot convey remapped legacy layout file path to the swift-frontend
when distributed caching is enabled. In order to properly support path
remapping, the legacy layout file is ingested on demand (thus it doesn't
need to be ingested for module compilation), and the remapped path is
communicated to swift-front via frontend flag.
rdar://162793678
We have adopters who are relying on directly importing the underlying Clang module in the presence of incompatible Swift modules.
Resolves rdar://162549210
Adjust the rule for how bridging header chaining is performed to
increase compatibilities with existing project configuration.
Previously in #84442, bridging header chaining was done via content
concatenation, rather than `#include` via absolute path, to enable prefix
mapping to drop references to absolute path. This triggers an
incompatibility since the directory of the header file is also
considered as part of the search path when resolve the include
directives in the file. Simple concatenation will result in some
`#include` can no longer be resolved.
Now relax the rule to do header content concatenation only when prefix
map is used. This will keep the breakage minimal, and the breakage can
always be fixed by additional include paths when turning on prefix
mapping.
In the future, we should move towards internal bridging header
implementation to avoid the need of chaining completely.
rdar://161854282
To allow prefix mapping of the bridging header to achieve cache hit when
source files are located in different location, the generated chained
bridging header should not include absolute paths of the headers. Fix
the problem by concat the chained bridging header together.
Fixes: https://github.com/swiftlang/swift/issues/84088
Avoid Swift Overlay lookup for the underlying clang module of a known Swift module.
i.e. When computing set of Swift Overlay dependencies for module 'A', which depends on a Clang module 'A', ensure we don't lookup Swift module 'A' itself here - this can lead to bizarre interactions where the source module under scan is queried as a dependency of itself.
Resolves rdar://159706486
This change introduces a thread-safe version of the 'SerializedDiagnosticConsumer' and refactors scanning compilation instance creation code to ensure this consumer gets added when the scanner query configuration command-line includes '-serialized-diagnostics-path' option.
Otherwise, when multiple workers encounter a diagnostic simultaneously we can encounter races which lead to corrupted diagnostic data or crashes
Resolves rdar://159598539
This change re-enables the new semantic of looking up Swift Overlay modules only for "visible" clang modules.
We have investigated the failures we were seeing with this change enabled and addressed them with project-side fixes.
SwiftVerifyEmittedModuleInterface job is configured to be built within
the SubInvocation and it needs to inherit the caching replay prefix map
in order to load macro plugin library correctly.
rdar://158692095
This moves the functionality of 'bridgeClangModuleDependency' into a utility in the main scanner class because it relies on various objects whose lifetime is already tied to the scanner itself.
Previously, serialization of optional import statement node arrays erroneously used the node layout of a non-optional import node array while maintaining a separate index for these arrays which got serialized into each module's info node. This meant it got serialized out in a sequence of non-optional import arrays, and as a result, the consumer deserialized them as such, which broke ordering of which arrays of import nodes belong which module, causing mismatches between a module's actual import set and the import which got deserialized for it.
This change fixes the optional import serialization routine to use the purpose-made separate layout which gets read out according to the separate optional import array index.
Previously, frequently-used methods like 'getAllDependencies' and 'getAllClangDependencies' had to aggregate (copy) multiple collections stored in a 'ModuleDependencyInfo' into a new result array to present to the client. These methods have been refactored to instead return an iterable joined view of the constituent collections.
In addition to skipping it on textual Swift module dependencies which were built without C++ interop enabled, also skip it over similarly on binary Swift dependencies
We have identified scenarios where this is insufficient and compilation still relies on the presence of certain overlay modules this excludes. Reverting to prior lookup logic while we investigate.
Resolves rdar://157519278
This is required in order to always have computed the set of visible Clang modules for each Swift module in the graph. Otherwise when some Clang module gets cached as a transitive dependency from a query and is later looked up as a direct dependency, there will not be any computed visible modules set.
Previously Swift overlay lookup was performed for every directly and transitively-imported Clang module.
https://github.com/llvm/llvm-project/pull/147969 introduced the concept of "visible" Clang modules from a given named Clang dependency scanner query which closely maps to the set of modules for which Swift will attempt to load a Swift overlay. This change switches overlay querying to apply only to the set of such visible modules.
Resolves rdar://144797648
Previously this flag was only used to pass explicit dependencies to compilation tasks. This change adds support for the dependency scanner to also consider these inputs when resolving dependencies.
Resolves https://github.com/swiftlang/swift-driver/issues/1951
When querying a Swift module, the scanner now also keeps track of all discovered candidate binary modules which are not compatible with current compilation.
- If a Swift dependency is successfully resolved to a compatible binary module or a textual interface, a warning is emitted for every incompatible binary Swift module discovered along the way.
- If a Swift dependency is not resolved, but incompatible module candidates were found, an error is emitted - while it is likely that the scan would fail downstream, it is also possible that an underlying Clang module dependency (with the same name) is successfuly resolved and the Swift lookup failure is ignored, which is still going to lead to failures most of the time if the client code assumes the presence of the Swift overlay module in this scenario.
This change refactors common error reporting by the scanner into a 'ModuleDependencyIssueReporter' class, which also keeps track of all diagnosed failed lookups to avoid repeating diagnostics.
A prior change ensured that we forego this query when looking up Swift overlays for a textual interface which was built without C++ interop. This change introduced a bug where it also caused us to skip this lookup for the main source module. This commit resolves that by preserving the fix above but also ensuring we perform the lookup for the main source module under scan.
The note will point the user to where the "other" module with the same name is located and mention whether it is an SDK module. This is nice to have in various circumstances where developers attempt to define a module with the same name as a Swift module that already exists on their search paths, for example in the SDK.
