Otherwise, when multiple workers encounter a diagnostic simultaneously we can encounter races which lead to corrupted diagnostic data or crashes
Resolves rdar://159598539
Adds an access control field for each imported module identified. When multiple imports of the same module are found, this keeps track of the most "open" access specifier.
Unlike with implicitly-built modules (prior to Swift 6 mode), explicitly-built modules require that all search paths be specified explicitly and no longer inherit search paths serialized into discovered Swift binary modules. This behavior was never intentional and is considered a bug. This change adds a diagnostic note to a scan failure: for each binary Swift module dependency, the scanner will attempt to execute a dependency scanning query for each serialized search path inside that module. If such diagnostic query returns a result, a diagnostic will be emitted to inform the user that the dependency may be found in the search path configuration of another Swift binary module dependency, specifying which search path contains the "missing" module, and stating that such search paths are not automatically inherited by the current compilation.
Otherwise querying this clang module, e.g. from the corresponding Swift overlay's underlying module import, will fail, since no such module exists.
Resolves rdar://151718115
Since we enabled parallel dependency scanning by-default, each individual scan needs a diagnostic consumer that is safe to use across many threads. Deprecate the 'Locking' sub-class, making its behavior the default in the base class.
This hash is also used for the dependency scanning hash. In both cases, PCH contents may differ based on whether a certain module they depend on is found in a system or non-system search path. In dependency scanning, systemness should cause a full change of scanning context requiring a from-scratch scan.
Resolves rdar://150334077
The field is only used to store information to be used in finalize stage, in caching builds. When loading scan results from the cache, the entries are finalized already and have the file info encoded in CASIDs already.
Resolves rdar://150307865
For the main source module, provide info on which dependencies are directly imported into the user program, explicitly ('import' statement) or implicitly (e.g. stdlib). Thist list does not include Swift overlay dependencies, cross-import dependencies, bridging header dependencies.
With '-sdk-module-cache-path', Swift textual interfaces found in the SDK will be built into a separate SDK-specific module cache.
Clang modules are not yet affected by this change, pending addition of the required API.
The algorithm already performs pairwise checks on module dependencies brought into compilation per-source-file. Previously, the algorithm considered the entire sub-graph of a given source file. Actual source compiles do not consider the full transitive module dependency set for cross-import-overlay lookup, but rather only directly-imported modules in a given source file, and '@_exported import' Swift transitive dependencies.
This change adds tracking of whether a given import statement is 'exported' to the dependency scanner and then refines the cross-import overlay lookup logic to only consider transitive modules that are exported by directly-imported dependencies.
Previous implementation took the entire transitive dependency set and cross-referenced all of its members to determine which ones introduce requried cross-import overlays. That implementation differed from the cross-import overlay loading logic during source compilation, where a corrsponding cross-import overlay module is only requested if the two constituent modules are reachable via direct 'import's from *the same source file*. Meaning the dependency scanner before this change would report cross-import overlay dependencies which never got loaded by the corresponding client source compile.
This change implements a new implementation of cross-import overlay discovery which first computes sub-graphs of module dependencies directly reachable by 'import's for each source file of the module under scan and then performs pairwise cross-import overlay query per each such sub-graph.
Resolves rdar://145157171
`llvm::StdThreadPool` is only available when LLVM_ENABLE_THREADS is enabled.
LLVM defines `DefaultThreadPool` type alias, which is always available
regardless of the threads enabled. This change allows building the library
without threads enabled, especially for Emscripten toolchain.
```
#if LLVM_ENABLE_THREADS
using DefaultThreadPool = StdThreadPool;
#else
using DefaultThreadPool = SingleThreadExecutor;
#endif
```
Add ability to automatically chaining the bridging headers discovered from all
dependencies module when doing swift caching build. This will eliminate all
implicit bridging header imports from the build and make the bridging header
importing behavior much more reliable, while keep the compatibility at maximum.
For example, if the current module A depends on module B and C, and both B and
C are binary modules that uses bridging header, when building module A,
dependency scanner will construct a new header that chains three bridging
headers together with the option to build a PCH from it. This will make all
importing errors more obvious while improving the performance.
Checking each module dependency info if it is up-to-date with respect to when the cache contents were serialized in a prior scan.
- Add a timestamp field to the serialization format for the dependency scanner cache
- Add a flag "-validate-prior-dependency-scan-cache" which, when combined with "-load-dependency-scan-cache" will have the scanner prune dependencies from the deserialized cache which have inputs that are newer than the prior scan itself
With the above in-place, the scan otherwise proceeds as-is, getting cache hits for entries still valid since the prior scan.
Batch dependency scanning was added as a mechanism to support multiple compilation contexts within a single module dependency graph.
The Swift compiler and the Explicitly-built modules model has long since abandoned this approach and this code has long been stale. It is time to remove it and its associated C API.
Instead, each scan's 'ModuleDependenciesCache' will hold all of the data corresponding to discovered module dependencies.
The initial design presumed the possibility of sharing a global scanning cache amongs different scanner invocations, possibly even different concurrent scanner invocations.
This change also deprecates two libSwiftScan entry-points: 'swiftscan_scanner_cache_load' and 'swiftscan_scanner_cache_serialize'. They never ended up getting used, and since this code has been largely stale, we are confident they have not otherwise had users, and they do not fit with this design.
A follow-up change will re-introduce moduele dependency cache serialization on a per-query basis and bring the binary format up-to-date.
This set, belonging to 'ModuleDependenciesCache', is only updated in a critical section behind a lock in the scanner. However, it is queried unsynchronized inside the Clang scanner itself. If an update causes a re-hash to happen, chaose can ensue with concurrent lookups.
Since this set only affects the produced set of results from teh Clang scanning query, we should simply pass in an immutable copy to scanning queries and rely on downstream de-duplication of scanning results.
With this, we can avoid passing in the reference to `ModuleDependenciesCache` to the 'scanFilesystemFor*ModuleDependency' altogether.
Resolves rdar://139414443
This change refactors the top-level dependency scanning flow to follow the following procedure:
Scan():
1. From the source target under scan, query all imported module identifiers for a *Swift* module. Leave unresolved identifiers unresolved. Proceed transitively to build a *Swift* module dependency graph.
2. Take every unresolved import identifier in the graph from (1) and, assuming that it must be a Clang module, dispatch all of them to be queried in-parallel by the scanner's worker pool.
3. Resolve bridging header Clang module dpendencies
4. Resolve all Swift overlay dependencies, relying on all Clang modules collected in (2) and (3)
5. For the source target under scan, use all of the above discovered module dependencies to resolve all cross-import overlay dependencies
Add function to handle all macro dependencies kinds in the scanner,
including taking care of the macro definitions in the module interface
for its client to use. The change involves:
* Encode the macro definition inside the binary module
* Resolve macro modules in the dependencies scanners, including those
declared inside the dependency modules.
* Propagate the macro defined from the direct dependencies to track
all the potentially available modules inside a module compilation.
Conflicts:
- `test/Interop/Cxx/class/method/methods-this-and-indirect-return-irgen-itanium.swift`
previously fixed on rebranch, now fixed on main (slightly differently).
Fix few issues from previous implementation from explicit module build
with macros and accurate macro dependency scanning in
https://github.com/swiftlang/swift/pull/73421.
First, there is a crash when propagating the macro dependencies. It
turns out that the current macro plugin implementation doesn't need the
downstream users to know about the plugin search path from the upstream
dependencies.
Secondly, fix a bug that the swiftinterface that has macro usage won't
build because the build command doesn't inherit the plugin search path
option.
Finally, add JSON output for macro dependencies so it is easier to
debug the macro dependencies.
rdar://131214106
