This change introduces a thread-safe version of the 'SerializedDiagnosticConsumer' and refactors scanning compilation instance creation code to ensure this consumer gets added when the scanner query configuration command-line includes '-serialized-diagnostics-path' option.
Otherwise, when multiple workers encounter a diagnostic simultaneously we can encounter races which lead to corrupted diagnostic data or crashes
Resolves rdar://159598539
This change re-enables the new semantic of looking up Swift Overlay modules only for "visible" clang modules.
We have investigated the failures we were seeing with this change enabled and addressed them with project-side fixes.
SwiftVerifyEmittedModuleInterface job is configured to be built within
the SubInvocation and it needs to inherit the caching replay prefix map
in order to load macro plugin library correctly.
rdar://158692095
This moves the functionality of 'bridgeClangModuleDependency' into a utility in the main scanner class because it relies on various objects whose lifetime is already tied to the scanner itself.
Previously, frequently-used methods like 'getAllDependencies' and 'getAllClangDependencies' had to aggregate (copy) multiple collections stored in a 'ModuleDependencyInfo' into a new result array to present to the client. These methods have been refactored to instead return an iterable joined view of the constituent collections.
In addition to skipping it on textual Swift module dependencies which were built without C++ interop enabled, also skip it over similarly on binary Swift dependencies
We have identified scenarios where this is insufficient and compilation still relies on the presence of certain overlay modules this excludes. Reverting to prior lookup logic while we investigate.
Resolves rdar://157519278
This is required in order to always have computed the set of visible Clang modules for each Swift module in the graph. Otherwise when some Clang module gets cached as a transitive dependency from a query and is later looked up as a direct dependency, there will not be any computed visible modules set.
Previously Swift overlay lookup was performed for every directly and transitively-imported Clang module.
https://github.com/llvm/llvm-project/pull/147969 introduced the concept of "visible" Clang modules from a given named Clang dependency scanner query which closely maps to the set of modules for which Swift will attempt to load a Swift overlay. This change switches overlay querying to apply only to the set of such visible modules.
Resolves rdar://144797648
Previously this flag was only used to pass explicit dependencies to compilation tasks. This change adds support for the dependency scanner to also consider these inputs when resolving dependencies.
Resolves https://github.com/swiftlang/swift-driver/issues/1951
When querying a Swift module, the scanner now also keeps track of all discovered candidate binary modules which are not compatible with current compilation.
- If a Swift dependency is successfully resolved to a compatible binary module or a textual interface, a warning is emitted for every incompatible binary Swift module discovered along the way.
- If a Swift dependency is not resolved, but incompatible module candidates were found, an error is emitted - while it is likely that the scan would fail downstream, it is also possible that an underlying Clang module dependency (with the same name) is successfuly resolved and the Swift lookup failure is ignored, which is still going to lead to failures most of the time if the client code assumes the presence of the Swift overlay module in this scenario.
This change refactors common error reporting by the scanner into a 'ModuleDependencyIssueReporter' class, which also keeps track of all diagnosed failed lookups to avoid repeating diagnostics.
A prior change ensured that we forego this query when looking up Swift overlays for a textual interface which was built without C++ interop. This change introduced a bug where it also caused us to skip this lookup for the main source module. This commit resolves that by preserving the fix above but also ensuring we perform the lookup for the main source module under scan.
The note will point the user to where the "other" module with the same name is located and mention whether it is an SDK module. This is nice to have in various circumstances where developers attempt to define a module with the same name as a Swift module that already exists on their search paths, for example in the SDK.
Move per-query state out of ScanningService. There is still a check to
make sure the CASOptions are matching between queries because of the
requirement on clang scanner. Otherwise, the scanning service should
contain no per-query information anymore.
Resolves: https://github.com/swiftlang/swift/issues/82490
While this made sense in the distant past where the scanning service provided backing storage for the dependency cache, it no longer does so and now makes for awkard layering where clients get at the service via the cache. Now the cache is a simple data structure while all the clients that need access to the scanning service will get it explicitly.
- 'SwiftModuleScanner' will now be owned directly by the 'ModuleDependencyScanningWorker' and will contain all the necessary custom logic, instead of being instantiated by the module interface loader for each query
- Moves ownership over module output path and sdk module output path directly into the scanning worker, instead of the cache
This was used a long time ago for a design of a scanner which could rely on the client to specify that some modules *will be* present at a given location but are not yet during the scan. We have long ago determined that the scanner must have all modules available to it at the time of scan for soundness. This code has been stale for a couple of years and it is time to simplify things a bit by deleting it.
Adds an access control field for each imported module identified. When multiple imports of the same module are found, this keeps track of the most "open" access specifier.
Unlike with implicitly-built modules (prior to Swift 6 mode), explicitly-built modules require that all search paths be specified explicitly and no longer inherit search paths serialized into discovered Swift binary modules. This behavior was never intentional and is considered a bug. This change adds a diagnostic note to a scan failure: for each binary Swift module dependency, the scanner will attempt to execute a dependency scanning query for each serialized search path inside that module. If such diagnostic query returns a result, a diagnostic will be emitted to inform the user that the dependency may be found in the search path configuration of another Swift binary module dependency, specifying which search path contains the "missing" module, and stating that such search paths are not automatically inherited by the current compilation.
Textual interfaces for 'Darwin' built with recent compilers specify that it is built witout C++ interop enabled. However, to ensure compatibility with versions of the 'Darwin' module built with older compilers, we hard-code this fact. This is required to break the module cycle that occurs when building the 'Darwin' module with C++ interop enabled, where the underlying 'Darwin' clang module depends on C++ standard library for which the compiler brings in the 'CxxStdlib' Swift overlay, which depends on 'Darwin'.
On creation, 'ClangImporter' adds overlay modulemap files for non-modular platform libraries (e.g. glibc, libstdc++), which allows Swift code to import and use those libraries.
This change adds the same filesystem overlay to dependency scanning queries by applying them to the filesystem instantiated for each depndency scanning worker. Without these overlays EBM builds cannot discover and use non-modular system libraries on non-Darwin platforms.
Resolves rdar://151780437
Otherwise querying this clang module, e.g. from the corresponding Swift overlay's underlying module import, will fail, since no such module exists.
Resolves rdar://151718115
After removing the CASFS implementation for clang modules, there is no
need to capture clang extra file that sets up the VFS for the clang
modules since all content imported by ClangImporter is dependency
scanned and available via include-tree. This saves more ClangImporter
instance when caching is enabled.
Update the test to check that clang content found via `-Xcc` VFS options
can currently work without capture the headermaps and vfs overlays.
When we discover a textual module dependency which is a module which was not originally built from source using C++ interop (specifying '-formal-cxx-interoperability-mode=off'), avoid looking up the C++ standard library Swift overlay for it. This is required for the case of the 'Darwin' module, for example, which includes headers which map to C++ stdlib headers when the compiler is operating in C++ interop mode, but the C++ standard library Swift overlay module itself depends on 'Darwin', which results in a cycle. To resolve such situations, we can rely on the fact that Swift textual interfaces of modules which were not built with C++ interop must be able to build without importing the C++ standard library Swift overlay, so we avoid specifying it as a dependency for such modules.
The primary source module, as well as Swift textual module dependencies which *were* built with C++ interop will continue getting a direct depedency of the 'CxxStdlib' Swift module.
Resolves rdar://150222155
In expectation, this should never happen. Such a situation means that within the same scanning action, Clang Dependency Scanner has produced two different variants of the same module. This is not supposed to happen, but we are currently hunting down the rare cases where it does, seemingly due to differences in Clang Scanner direct by-name queries and transitive header lookup queries.
Currently, the macro plugin options are included as cache key and the
absolute path of the plugin executable and library will affect cache
hit, even the plugin itself is identical.
Using the new option `-resolved-plugin-validation` flag, the macro
plugin paths are remapped just like the other paths during dependency
scanning. `swift-frontend` will unmap to its original path during the
compilation, make sure the content hasn't changed, and load the plugin.
It also hands few other corner cases for macro plugins:
* Make sure the plugin options in the swift module is prefix mapped.
* Make sure the remarks of the macro loading is not cached, as the
mesasge includes the absolute path of the plugin, and is not
cacheable.
rdar://148465899
With '-sdk-module-cache-path', Swift textual interfaces found in the SDK will be built into a separate SDK-specific module cache.
Clang modules are not yet affected by this change, pending addition of the required API.
