While this made sense in the distant past where the scanning service provided backing storage for the dependency cache, it no longer does so and now makes for awkard layering where clients get at the service via the cache. Now the cache is a simple data structure while all the clients that need access to the scanning service will get it explicitly.
- 'SwiftModuleScanner' will now be owned directly by the 'ModuleDependencyScanningWorker' and will contain all the necessary custom logic, instead of being instantiated by the module interface loader for each query
- Moves ownership over module output path and sdk module output path directly into the scanning worker, instead of the cache
This was used a long time ago for a design of a scanner which could rely on the client to specify that some modules *will be* present at a given location but are not yet during the scan. We have long ago determined that the scanner must have all modules available to it at the time of scan for soundness. This code has been stale for a couple of years and it is time to simplify things a bit by deleting it.
Adds an access control field for each imported module identified. When multiple imports of the same module are found, this keeps track of the most "open" access specifier.
Unlike with implicitly-built modules (prior to Swift 6 mode), explicitly-built modules require that all search paths be specified explicitly and no longer inherit search paths serialized into discovered Swift binary modules. This behavior was never intentional and is considered a bug. This change adds a diagnostic note to a scan failure: for each binary Swift module dependency, the scanner will attempt to execute a dependency scanning query for each serialized search path inside that module. If such diagnostic query returns a result, a diagnostic will be emitted to inform the user that the dependency may be found in the search path configuration of another Swift binary module dependency, specifying which search path contains the "missing" module, and stating that such search paths are not automatically inherited by the current compilation.
On creation, 'ClangImporter' adds overlay modulemap files for non-modular platform libraries (e.g. glibc, libstdc++), which allows Swift code to import and use those libraries.
This change adds the same filesystem overlay to dependency scanning queries by applying them to the filesystem instantiated for each depndency scanning worker. Without these overlays EBM builds cannot discover and use non-modular system libraries on non-Darwin platforms.
Resolves rdar://151780437
Otherwise querying this clang module, e.g. from the corresponding Swift overlay's underlying module import, will fail, since no such module exists.
Resolves rdar://151718115
When we discover a textual module dependency which is a module which was not originally built from source using C++ interop (specifying '-formal-cxx-interoperability-mode=off'), avoid looking up the C++ standard library Swift overlay for it. This is required for the case of the 'Darwin' module, for example, which includes headers which map to C++ stdlib headers when the compiler is operating in C++ interop mode, but the C++ standard library Swift overlay module itself depends on 'Darwin', which results in a cycle. To resolve such situations, we can rely on the fact that Swift textual interfaces of modules which were not built with C++ interop must be able to build without importing the C++ standard library Swift overlay, so we avoid specifying it as a dependency for such modules.
The primary source module, as well as Swift textual module dependencies which *were* built with C++ interop will continue getting a direct depedency of the 'CxxStdlib' Swift module.
Resolves rdar://150222155
In expectation, this should never happen. Such a situation means that within the same scanning action, Clang Dependency Scanner has produced two different variants of the same module. This is not supposed to happen, but we are currently hunting down the rare cases where it does, seemingly due to differences in Clang Scanner direct by-name queries and transitive header lookup queries.
The field is only used to store information to be used in finalize stage, in caching builds. When loading scan results from the cache, the entries are finalized already and have the file info encoded in CASIDs already.
Resolves rdar://150307865
With '-sdk-module-cache-path', Swift textual interfaces found in the SDK will be built into a separate SDK-specific module cache.
Clang modules are not yet affected by this change, pending addition of the required API.
Add ability to automatically chaining the bridging headers discovered from all
dependencies module when doing swift caching build. This will eliminate all
implicit bridging header imports from the build and make the bridging header
importing behavior much more reliable, while keep the compatibility at maximum.
For example, if the current module A depends on module B and C, and both B and
C are binary modules that uses bridging header, when building module A,
dependency scanner will construct a new header that chains three bridging
headers together with the option to build a PCH from it. This will make all
importing errors more obvious while improving the performance.
Checking each module dependency info if it is up-to-date with respect to when the cache contents were serialized in a prior scan.
- Add a timestamp field to the serialization format for the dependency scanner cache
- Add a flag "-validate-prior-dependency-scan-cache" which, when combined with "-load-dependency-scan-cache" will have the scanner prune dependencies from the deserialized cache which have inputs that are newer than the prior scan itself
With the above in-place, the scan otherwise proceeds as-is, getting cache hits for entries still valid since the prior scan.
Instead, each scan's 'ModuleDependenciesCache' will hold all of the data corresponding to discovered module dependencies.
The initial design presumed the possibility of sharing a global scanning cache amongs different scanner invocations, possibly even different concurrent scanner invocations.
This change also deprecates two libSwiftScan entry-points: 'swiftscan_scanner_cache_load' and 'swiftscan_scanner_cache_serialize'. They never ended up getting used, and since this code has been largely stale, we are confident they have not otherwise had users, and they do not fit with this design.
A follow-up change will re-introduce moduele dependency cache serialization on a per-query basis and bring the binary format up-to-date.
This change refactors the top-level dependency scanning flow to follow the following procedure:
Scan():
1. From the source target under scan, query all imported module identifiers for a *Swift* module. Leave unresolved identifiers unresolved. Proceed transitively to build a *Swift* module dependency graph.
2. Take every unresolved import identifier in the graph from (1) and, assuming that it must be a Clang module, dispatch all of them to be queried in-parallel by the scanner's worker pool.
3. Resolve bridging header Clang module dpendencies
4. Resolve all Swift overlay dependencies, relying on all Clang modules collected in (2) and (3)
5. For the source target under scan, use all of the above discovered module dependencies to resolve all cross-import overlay dependencies
Use IncludeTreeFileList instead of full feature CASFS for swift
dependency filesystem. This allows smaller CAS based VFS that is smaller
and faster. This is enabled by the CAS enabled compilation does not
need to iterate file system.
rdar://136787368
Rewrite part of the dependency scanner into a class so it is easier to
keep track of the global states and avoid redundent work. This allows
folding the pruning unused VFS job into the same loop and reduce a lot
of repeated dependency cache updates.
Add function to handle all macro dependencies kinds in the scanner,
including taking care of the macro definitions in the module interface
for its client to use. The change involves:
* Encode the macro definition inside the binary module
* Resolve macro modules in the dependencies scanners, including those
declared inside the dependency modules.
* Propagate the macro defined from the direct dependencies to track
all the potentially available modules inside a module compilation.
When scanning swift modules and constructing their build commands, there
is no need to pass any external plugin search paths if there are no macro
dependencies for the module.
rdar://135221984
This makes sure that Swift respects `-Xcc -stdlib=libc++` flags.
Clang already has existing logic to discover the system-wide libc++ installation on Linux. We rely on that logic here.
Importing a Swift module that was built with a different C++ stdlib is not supported and emits an error.
The Cxx module can be imported when compiling with any C++ stdlib. The synthesized conformances, e.g. to CxxRandomAccessCollection also work. However, CxxStdlib currently cannot be imported when compiling with libc++, since on Linux it refers to symbols from libstdc++ which have different mangled names in libc++.
rdar://118357548 / https://github.com/swiftlang/swift/issues/69825
Rather than only protecting the insertion and non-const access to
`ContextSpecificCacheMap` in ScanningService, extend the mutex
protection to all accesses. Even a 'const' lookup in the cache map is
not thread safe because the `StringMap` could be in the process of being
rehashed.
rdar://127205953
Fix few issues from previous implementation from explicit module build
with macros and accurate macro dependency scanning in
https://github.com/swiftlang/swift/pull/73421.
First, there is a crash when propagating the macro dependencies. It
turns out that the current macro plugin implementation doesn't need the
downstream users to know about the plugin search path from the upstream
dependencies.
Secondly, fix a bug that the swiftinterface that has macro usage won't
build because the build command doesn't inherit the plugin search path
option.
Finally, add JSON output for macro dependencies so it is easier to
debug the macro dependencies.
rdar://131214106
When the dependency scanner picks a pre-built binary module candidate for a given dependency, it needs to be able to attempt to resolve its cross-import overlays relative to the textual interface that the binary module was built from. For example, if a collection of binary modules are located in, and resolved as dependencies from, a pre-built module directory, the scanner must lookup their corresponding cross-import overlays relative to the defining interface as read out from the binary module's MODULE_INTERFACE_PATH. https://github.com/swiftlang/swift/pull/70817 ensures that binary modules serialize the path to their defining textual interface.
Resolves rdar://130778577
Build an accurate macro dependency for swift caching. Specifically, do
not include not used macro plugins into the dependency, which might
cause false negatives for cache hits.
This also builds the foundation for future improvement when dependency
scanning will determine the macro plugin to load and swift-frontend do
not need to redo the work.
rdar://127116512
This change modifies the dependency scanner to keep track of source locations of each encountered 'import' statement, in order to be able to emit diagnostics with source locations if an import failed to resolve.
- Keep track of each 'import' statement's source buffer, line number, and column number when adding it. The dependency scanner utilizes separate compilation instances, and therefore separate Source Managers for scanning `import` statements of user sources and textual interfaces of Swift dependencies. Since import resolution may happen in the main scanner compilation instance while the `import` itself was found by an interface-scanning sub-instance, we cannot simply hold on to the import's `SourceLoc`.
- Add libSwiftScan API for diagnostics to carry above source locations to clients.
Teach dependency scanner to pass cross import overlay file to
swift-frontend for main module compilation. This allows swift-frontend
not to repeat the file system search for overlay files when loading
modules.
This also fixes the issue when caching is enabled, the cross import
doesn't work when the first module is a clang module because the module
built with caching using clang include tree does not preserve
DefinitionLoc which is used to inferred the modulemap location for cross
import overlay search.
rdar://127844120
