For clients, such as the debugger, who do not have access the full
output of the dependency scanner, it is a huger performance and
correctness improvement if each explicitly built Swift module not just
serialized all its Clang .pcm dependencies (via the serialized Clang
compiler invocation) but also its direct Swift module dependencies.
This patch changes the Swift module format to store the absolute path
or cas cache key for each dependency in the INPUT block, and makes
sure the deserialization makes these available to the ESML.
rdar://150969755
To support distributed caching for targets that need to use legacy
layout file, the path of legacy layout needs to be remapped.
Current handling of the legacy layout file is to ingest all layout files
to the CAS FileSystem as part of the compiler resource files. But it
cannot convey remapped legacy layout file path to the swift-frontend
when distributed caching is enabled. In order to properly support path
remapping, the legacy layout file is ingested on demand (thus it doesn't
need to be ingested for module compilation), and the remapped path is
communicated to swift-front via frontend flag.
rdar://162793678
This change introduces a thread-safe version of the 'SerializedDiagnosticConsumer' and refactors scanning compilation instance creation code to ensure this consumer gets added when the scanner query configuration command-line includes '-serialized-diagnostics-path' option.
Previously, frequently-used methods like 'getAllDependencies' and 'getAllClangDependencies' had to aggregate (copy) multiple collections stored in a 'ModuleDependencyInfo' into a new result array to present to the client. These methods have been refactored to instead return an iterable joined view of the constituent collections.
This is required in order to always have computed the set of visible Clang modules for each Swift module in the graph. Otherwise when some Clang module gets cached as a transitive dependency from a query and is later looked up as a direct dependency, there will not be any computed visible modules set.
The note will point the user to where the "other" module with the same name is located and mention whether it is an SDK module. This is nice to have in various circumstances where developers attempt to define a module with the same name as a Swift module that already exists on their search paths, for example in the SDK.
Move per-query state out of ScanningService. There is still a check to
make sure the CASOptions are matching between queries because of the
requirement on clang scanner. Otherwise, the scanning service should
contain no per-query information anymore.
Resolves: https://github.com/swiftlang/swift/issues/82490
While this made sense in the distant past where the scanning service provided backing storage for the dependency cache, it no longer does so and now makes for awkard layering where clients get at the service via the cache. Now the cache is a simple data structure while all the clients that need access to the scanning service will get it explicitly.
- 'SwiftModuleScanner' will now be owned directly by the 'ModuleDependencyScanningWorker' and will contain all the necessary custom logic, instead of being instantiated by the module interface loader for each query
- Moves ownership over module output path and sdk module output path directly into the scanning worker, instead of the cache
This was used a long time ago for a design of a scanner which could rely on the client to specify that some modules *will be* present at a given location but are not yet during the scan. We have long ago determined that the scanner must have all modules available to it at the time of scan for soundness. This code has been stale for a couple of years and it is time to simplify things a bit by deleting it.
Otherwise querying this clang module, e.g. from the corresponding Swift overlay's underlying module import, will fail, since no such module exists.
Resolves rdar://151718115
Using IncludeTree::FileList to concat the include tree file systems that
are passed on the command-line. This significantly reduce the
command-line size, and also makes the cache key computation a lot
faster.
rdar://148752988
When we discover a textual module dependency which is a module which was not originally built from source using C++ interop (specifying '-formal-cxx-interoperability-mode=off'), avoid looking up the C++ standard library Swift overlay for it. This is required for the case of the 'Darwin' module, for example, which includes headers which map to C++ stdlib headers when the compiler is operating in C++ interop mode, but the C++ standard library Swift overlay module itself depends on 'Darwin', which results in a cycle. To resolve such situations, we can rely on the fact that Swift textual interfaces of modules which were not built with C++ interop must be able to build without importing the C++ standard library Swift overlay, so we avoid specifying it as a dependency for such modules.
The primary source module, as well as Swift textual module dependencies which *were* built with C++ interop will continue getting a direct depedency of the 'CxxStdlib' Swift module.
Resolves rdar://150222155
Improve diagnostics message for swift caching build by trying to emit
the diagnostics early when there is more context to differentiate the
different kind of problems.
After the improvement, CAS Error should be more closer to when there is
functional problem with the CAS, rather than mixing in other kinds of
problem (like scanning dependency failures) when operating with a CAS.
rdar://145676736
For the main source module, provide info on which dependencies are directly imported into the user program, explicitly ('import' statement) or implicitly (e.g. stdlib). Thist list does not include Swift overlay dependencies, cross-import dependencies, bridging header dependencies.
Currently, the macro plugin options are included as cache key and the
absolute path of the plugin executable and library will affect cache
hit, even the plugin itself is identical.
Using the new option `-resolved-plugin-validation` flag, the macro
plugin paths are remapped just like the other paths during dependency
scanning. `swift-frontend` will unmap to its original path during the
compilation, make sure the content hasn't changed, and load the plugin.
It also hands few other corner cases for macro plugins:
* Make sure the plugin options in the swift module is prefix mapped.
* Make sure the remarks of the macro loading is not cached, as the
mesasge includes the absolute path of the plugin, and is not
cacheable.
rdar://148465899
With '-sdk-module-cache-path', Swift textual interfaces found in the SDK will be built into a separate SDK-specific module cache.
Clang modules are not yet affected by this change, pending addition of the required API.
Make `-enable-deterministic-check` a driver option and teach dependency
scanner to propagate the option to explicit module build commmands. This
allows to the option to check every build output from the compiler is
deterministic.
https://github.com/swiftlang/swift/pull/79297 implemented current working directory pruning but left some unnecessary code
that computes Swift interface module output path prematurely. This PR removes the code that computes the output path too
early. The `ExplicitModuleDependencyResolver` now adds the path to the command line after it can correctly compute it.
Context: https://github.com/swiftlang/swift/pull/79297/files#r1955314542
Add ability to automatically chaining the bridging headers discovered from all
dependencies module when doing swift caching build. This will eliminate all
implicit bridging header imports from the build and make the bridging header
importing behavior much more reliable, while keep the compatibility at maximum.
For example, if the current module A depends on module B and C, and both B and
C are binary modules that uses bridging header, when building module A,
dependency scanner will construct a new header that chains three bridging
headers together with the option to build a PCH from it. This will make all
importing errors more obvious while improving the performance.
Checking each module dependency info if it is up-to-date with respect to when the cache contents were serialized in a prior scan.
- Add a timestamp field to the serialization format for the dependency scanner cache
- Add a flag "-validate-prior-dependency-scan-cache" which, when combined with "-load-dependency-scan-cache" will have the scanner prune dependencies from the deserialized cache which have inputs that are newer than the prior scan itself
With the above in-place, the scan otherwise proceeds as-is, getting cache hits for entries still valid since the prior scan.
Batch dependency scanning was added as a mechanism to support multiple compilation contexts within a single module dependency graph.
The Swift compiler and the Explicitly-built modules model has long since abandoned this approach and this code has long been stale. It is time to remove it and its associated C API.
When planning for a swift source module, it should not get build
commands for its module dependencies. Those dependencies should be
planned and added by swift-driver.
This is another regression from #76700 that causes unnecessary increase
of build command-line size.
rdar://141843125
