Improve diagnostics message for swift caching build by trying to emit
the diagnostics early when there is more context to differentiate the
different kind of problems.
After the improvement, CAS Error should be more closer to when there is
functional problem with the CAS, rather than mixing in other kinds of
problem (like scanning dependency failures) when operating with a CAS.
rdar://145676736
(cherry picked from commit 226552bf23)
With '-sdk-module-cache-path', Swift textual interfaces found in the SDK will be built into a separate SDK-specific module cache.
Clang modules are not yet affected by this change, pending addition of the required API.
The algorithm already performs pairwise checks on module dependencies brought into compilation per-source-file. Previously, the algorithm considered the entire sub-graph of a given source file. Actual source compiles do not consider the full transitive module dependency set for cross-import-overlay lookup, but rather only directly-imported modules in a given source file, and '@_exported import' Swift transitive dependencies.
This change adds tracking of whether a given import statement is 'exported' to the dependency scanner and then refines the cross-import overlay lookup logic to only consider transitive modules that are exported by directly-imported dependencies.
Add ability to automatically chaining the bridging headers discovered from all
dependencies module when doing swift caching build. This will eliminate all
implicit bridging header imports from the build and make the bridging header
importing behavior much more reliable, while keep the compatibility at maximum.
For example, if the current module A depends on module B and C, and both B and
C are binary modules that uses bridging header, when building module A,
dependency scanner will construct a new header that chains three bridging
headers together with the option to build a PCH from it. This will make all
importing errors more obvious while improving the performance.
Checking each module dependency info if it is up-to-date with respect to when the cache contents were serialized in a prior scan.
- Add a timestamp field to the serialization format for the dependency scanner cache
- Add a flag "-validate-prior-dependency-scan-cache" which, when combined with "-load-dependency-scan-cache" will have the scanner prune dependencies from the deserialized cache which have inputs that are newer than the prior scan itself
With the above in-place, the scan otherwise proceeds as-is, getting cache hits for entries still valid since the prior scan.
Instead, each scan's 'ModuleDependenciesCache' will hold all of the data corresponding to discovered module dependencies.
The initial design presumed the possibility of sharing a global scanning cache amongs different scanner invocations, possibly even different concurrent scanner invocations.
This change also deprecates two libSwiftScan entry-points: 'swiftscan_scanner_cache_load' and 'swiftscan_scanner_cache_serialize'. They never ended up getting used, and since this code has been largely stale, we are confident they have not otherwise had users, and they do not fit with this design.
A follow-up change will re-introduce moduele dependency cache serialization on a per-query basis and bring the binary format up-to-date.
This change refactors the top-level dependency scanning flow to follow the following procedure:
Scan():
1. From the source target under scan, query all imported module identifiers for a *Swift* module. Leave unresolved identifiers unresolved. Proceed transitively to build a *Swift* module dependency graph.
2. Take every unresolved import identifier in the graph from (1) and, assuming that it must be a Clang module, dispatch all of them to be queried in-parallel by the scanner's worker pool.
3. Resolve bridging header Clang module dpendencies
4. Resolve all Swift overlay dependencies, relying on all Clang modules collected in (2) and (3)
5. For the source target under scan, use all of the above discovered module dependencies to resolve all cross-import overlay dependencies
Use IncludeTreeFileList instead of full feature CASFS for swift
dependency filesystem. This allows smaller CAS based VFS that is smaller
and faster. This is enabled by the CAS enabled compilation does not
need to iterate file system.
rdar://136787368
Add function to handle all macro dependencies kinds in the scanner,
including taking care of the macro definitions in the module interface
for its client to use. The change involves:
* Encode the macro definition inside the binary module
* Resolve macro modules in the dependencies scanners, including those
declared inside the dependency modules.
* Propagate the macro defined from the direct dependencies to track
all the potentially available modules inside a module compilation.
When scanning swift modules and constructing their build commands, there
is no need to pass any external plugin search paths if there are no macro
dependencies for the module.
rdar://135221984
This makes sure that Swift respects `-Xcc -stdlib=libc++` flags.
Clang already has existing logic to discover the system-wide libc++ installation on Linux. We rely on that logic here.
Importing a Swift module that was built with a different C++ stdlib is not supported and emits an error.
The Cxx module can be imported when compiling with any C++ stdlib. The synthesized conformances, e.g. to CxxRandomAccessCollection also work. However, CxxStdlib currently cannot be imported when compiling with libc++, since on Linux it refers to symbols from libstdc++ which have different mangled names in libc++.
rdar://118357548 / https://github.com/swiftlang/swift/issues/69825
Rather than only protecting the insertion and non-const access to
`ContextSpecificCacheMap` in ScanningService, extend the mutex
protection to all accesses. Even a 'const' lookup in the cache map is
not thread safe because the `StringMap` could be in the process of being
rehashed.
rdar://127205953
Build an accurate macro dependency for swift caching. Specifically, do
not include not used macro plugins into the dependency, which might
cause false negatives for cache hits.
This also builds the foundation for future improvement when dependency
scanning will determine the macro plugin to load and swift-frontend do
not need to redo the work.
rdar://127116512
Although I don't plan to bring over new assertions wholesale
into the current qualification branch, it's entirely possible
that various minor changes in main will use the new assertions;
having this basic support in the release branch will simplify that.
(This is why I'm adding the includes as a separate pass from
rewriting the individual assertions)
This change modifies the dependency scanner to keep track of source locations of each encountered 'import' statement, in order to be able to emit diagnostics with source locations if an import failed to resolve.
- Keep track of each 'import' statement's source buffer, line number, and column number when adding it. The dependency scanner utilizes separate compilation instances, and therefore separate Source Managers for scanning `import` statements of user sources and textual interfaces of Swift dependencies. Since import resolution may happen in the main scanner compilation instance while the `import` itself was found by an interface-scanning sub-instance, we cannot simply hold on to the import's `SourceLoc`.
- Add libSwiftScan API for diagnostics to carry above source locations to clients.
Preliminary caching support for macro:
* Inserting the plugin into the CASFS
* Lookup plugin via physical file system
For future better support, we should teach dependency scanner to resolve
macros and return the resolved plugins to swift-frontend.
rdar://121873571
Improve swift dependency scanner by validating and selecting dependency
module into scanner. This provides benefits that:
* Build system does not need to schedule interface compilation task if
the candidate module is picked, it can just use the candidate module
directly.
* There is no need for forwarding module in the explicit module build.
Since the build system is coordinating the build, there is no need for
the forwarding module in the module cache to avoid duplicated work,
* This also correctly supports all the module loading modes in the
dependency scanner.
This is achieved by only adding validate and up-to-date binary module as
the candidate module for swift interface module dependency. This allows
caching build to construct the correct dependency in the CAS. If there
is a candidate module for the interface module, dependency scanner will
return a binary module dependency in the dependency graph.
The legacy behavior is mostly preserved with a hidden frontend flag
`-no-scanner-module-validation`, while the scanner output is mostly
interchangeable with new scanner behavior with `prefer-interface` module
loading mode except the candidate module will not be returned.
rdar://123711823
Support `-vfsoverlay` swift option for explicit module build (including
caching build). Previously, if the interface file is discovered from a
location that is remapped by overlay, module cannot be built correctly.
Make sure the overlay options are passed down to all interface
compilaiton command.
For caching build, need to make sure the overlay itself is part of the CAS
file system so the downstream compilation can discover that.
rdar://123655183
Otherwise they may have module dependencies of their own which will not be detected by the scanner and included in the list of explicit inputs for compilation.
Guard 'resetCache' and 'resetDiagnostics' as critical sections using 'DependencyScanningToolStateLock'. Otherwise there's a chance that one thread in the scanner is doing a reset on the diagnostic consumer, while some other thread is adding this diagnostic consumer to another scan instance which may also be populating said consumer at the time. Similarly, for resetting 'ScanningService' though it is much more unlikely to be reset while in-use by other scanning actions.
SE-0364 was implemented to discourage "retroactive" conformances that might
conflict with conformances that could be introduced by other modules in the
future. These diagnostics should not apply to conformances that involve types
and protocols imported from the underlying clang module of a Swift module since
the two modules are assumed to be developed in tandem by the same owners,
despite technically being separate modules from the perspective of the
compiler.
The diagnostics implemented in https://github.com/apple/swift/pull/36068 were
designed to take underlying clang modules into account. However, the
implementation assumed that `ModuleDecl::getUnderlyingModuleIfOverlay()` would
behave as expected when called on the Swift module being compiled.
Unfortunately, it would always return `nullptr` and thus conformances involving
the underlying clang module are being diagnosed unexpectedly.
The fix is to make `ModuleDecl::getUnderlyingModuleIfOverlay()` behave as
expected when it is made up of `SourceFile`s.
Resolves rdar://121478556
