Create a path that swift-frontend can execute an uncached job from
modules built with CAS based explicit module build. The new flag
-import-module-from-cas will allow an uncached build to load module
from CAS, and combined with source file from real file system to build
the current module. This allows quick iterations that bypasses CAS,
without full dependency scanning every time in between.
rdar://152441866
Use `resetBenignCodeGenOptions()` from clang dependency scanner to clear
the swift explicit module build cc1 arguments. This fixes the problem
that CurrentWorkingDirectory is leaking through
`-fcoverage-compilation-dir` that can cause extra module variants when
caching is enabled. This also avoid the duplicating the logics for
clearing CodeGen options inside Swift.
rdar://151395300
(cherry picked from commit 531227d4b9)
After removing the CASFS implementation for clang modules, there is no
need to capture clang extra file that sets up the VFS for the clang
modules since all content imported by ClangImporter is dependency
scanned and available via include-tree. This saves more ClangImporter
instance when caching is enabled.
Update the test to check that clang content found via `-Xcc` VFS options
can currently work without capture the headermaps and vfs overlays.
(cherry picked from commit 1506a0d495)
Using IncludeTree::FileList to concat the include tree file systems that
are passed on the command-line. This significantly reduce the
command-line size, and also makes the cache key computation a lot
faster.
rdar://148752988
(cherry picked from commit 201e4faea7)
Currently, the macro plugin options are included as cache key and the
absolute path of the plugin executable and library will affect cache
hit, even the plugin itself is identical.
Using the new option `-resolved-plugin-validation` flag, the macro
plugin paths are remapped just like the other paths during dependency
scanning. `swift-frontend` will unmap to its original path during the
compilation, make sure the content hasn't changed, and load the plugin.
It also hands few other corner cases for macro plugins:
* Make sure the plugin options in the swift module is prefix mapped.
* Make sure the remarks of the macro loading is not cached, as the
mesasge includes the absolute path of the plugin, and is not
cacheable.
rdar://148465899
(cherry picked from commit 3d38d0dd56)
Suppose module 'Foo' exists in the search paths and specifies user module version '1.0'.
If the first encountered 'canImport' query is unversioned:
...
Followed by a versioned one:
...
The success of the first check will record an unversioned successful canImport, which will cause the second check to evaluate to 'true', which is incorrect.
This change causes even unversioned 'canImport' checks to track and record the discovered user module version.
Make `-enable-deterministic-check` a driver option and teach dependency
scanner to propagate the option to explicit module build commmands. This
allows to the option to check every build output from the compiler is
deterministic.
https://github.com/swiftlang/swift/pull/79270 taught the dependency scanner to ignore `-file-compilation-dir` when caching is
_not_ in effect but did not make the corresponding change when caching is in effect. This PR teaches the scanner to ignore `-file-compliation-dir` when caching is in effect.
rdar://146025100
Previous implementation took the entire transitive dependency set and cross-referenced all of its members to determine which ones introduce requried cross-import overlays. That implementation differed from the cross-import overlay loading logic during source compilation, where a corrsponding cross-import overlay module is only requested if the two constituent modules are reachable via direct 'import's from *the same source file*. Meaning the dependency scanner before this change would report cross-import overlay dependencies which never got loaded by the corresponding client source compile.
This change implements a new implementation of cross-import overlay discovery which first computes sub-graphs of module dependencies directly reachable by 'import's for each source file of the module under scan and then performs pairwise cross-import overlay query per each such sub-graph.
Resolves rdar://145157171
Fix a bug when emitting cached diagnostics from nested macros. When
building a new source manager for cached diagnostics, there is a bug
when creating mapping between FileIDs between two different source
managers. If the newly created file in source manager requires another
file to be mapped, e.g. when emitting a diagnostic from nested macros,
the returning FileID can be updated by the second request before
returning. Fix the bug by making sure two different FileIDs are returned
in this case.
rdar://144810862
Add ability to automatically chaining the bridging headers discovered from all
dependencies module when doing swift caching build. This will eliminate all
implicit bridging header imports from the build and make the bridging header
importing behavior much more reliable, while keep the compatibility at maximum.
For example, if the current module A depends on module B and C, and both B and
C are binary modules that uses bridging header, when building module A,
dependency scanner will construct a new header that chains three bridging
headers together with the option to build a PCH from it. This will make all
importing errors more obvious while improving the performance.
Checking each module dependency info if it is up-to-date with respect to when the cache contents were serialized in a prior scan.
- Add a timestamp field to the serialization format for the dependency scanner cache
- Add a flag "-validate-prior-dependency-scan-cache" which, when combined with "-load-dependency-scan-cache" will have the scanner prune dependencies from the deserialized cache which have inputs that are newer than the prior scan itself
With the above in-place, the scan otherwise proceeds as-is, getting cache hits for entries still valid since the prior scan.
Fix the typo in GenerateExplicitModuleMap.py where the modulePath is
mis-spelled. Use the module path from the dependency output json file
directly instead, which means the script can be used to generate
non-caching build module map as well since the module path actually
matters for those builds.
Fix a bug that an empty filename for diagnostics can trigger an error
in the cached diagnostics. The empty filename can be a virtual file
synthesized, for example, to hold an implicit attribute from clang
importer.
The cached diagnostics can correctly handle such filename now and
cached/replay those diagnostics.
rdar://141961161
When planning for a swift source module, it should not get build
commands for its module dependencies. Those dependencies should be
planned and added by swift-driver.
This is another regression from #76700 that causes unnecessary increase
of build command-line size.
rdar://141843125
When cloning the clang instance in ClangImporter to perform tasks like
PCM/PCH compilation, make two compiler instance to share the same
CASOptions so they can share the CAS ObjectStore and life time.
This fixes a case where clang emits a diagnostics in a source buffer that
gets mapped in via CAS, and mapped into swift source manager. While CAS has
the ownship from CompilerInvocation -> CASOptions -> CAS, if the
CompilerInvocation is deleted when the cloned instance is deleted, it
left an invalid buffer in the swift source manager.
rdar://141284501
The symbol graph output from a module can contain an arbitrary number of
files, depending on what extensions it contains, so cache a list of
symbol graph files with their base name and contents so that they can be
replayed.
rdar://140286819
Find all the usages of `--enable-experimental-feature` or
`--enable-upcoming-feature` in the tests and replace some of the
`REQUIRES: asserts` to use `REQUIRES: swift-feature-Foo` instead, which
should correctly apply to depending on the asserts/noasserts mode of the
toolchain for each feature.
Remove some comments that talked about enabling asserts since they don't
apply anymore (but I might had miss some).
All this was done with an automated script, so some formatting weirdness
might happen, but I hope I fixed most of those.
There might be some tests that were `REQUIRES: asserts` that might run
in `noasserts` toolchains now. This will normally be because their
feature went from experimental to upcoming/base and the tests were not
updated.
Teach dependency scanner to construct build commands using resolved
plugin search path option. This ensures the modules that do not have
access to the macro plugins will not have a different variant due to
different plugin search path.
rdar://136682810
This change refactors the top-level dependency scanning flow to follow the following procedure:
Scan():
1. From the source target under scan, query all imported module identifiers for a *Swift* module. Leave unresolved identifiers unresolved. Proceed transitively to build a *Swift* module dependency graph.
2. Take every unresolved import identifier in the graph from (1) and, assuming that it must be a Clang module, dispatch all of them to be queried in-parallel by the scanner's worker pool.
3. Resolve bridging header Clang module dpendencies
4. Resolve all Swift overlay dependencies, relying on all Clang modules collected in (2) and (3)
5. For the source target under scan, use all of the above discovered module dependencies to resolve all cross-import overlay dependencies
Do not add CAS configurations into cache key computation. This ensure if
the cache keys are identical if two functionally equivalent plugins are
used. This is safe to do because if any of the CAS configurations
changes how the cache key is computed, they are going to be directly
reflected in the cache key without adding its configuration into the
key.
rdar://137091843
Use IncludeTreeFileList instead of full feature CASFS for swift
dependency filesystem. This allows smaller CAS based VFS that is smaller
and faster. This is enabled by the CAS enabled compilation does not
need to iterate file system.
rdar://136787368
Fully support make-style `.d` dependencies file output by making
following improvements:
* All correct dependency file render when cache hit for a different
output file location. The dependency file should list the correct
output path, not the stale output path for the initial compilation
* When enable a path prefix mapper to canonicalize the path, the
dependency file should render the input file correctly as the input
file path on disk.
rdar://132250067
Make sure blocklists are remapped for interface compilation. The main
module blocklist remap is done by swift-driver and do not need to be
handled in scanner.
Make sure block-list file is read through VFS so CASFS can be used to
read the configuration to ensure sound caching, and also the path of the
blocklist can be canonicalized via path remapping.
Add function to handle all macro dependencies kinds in the scanner,
including taking care of the macro definitions in the module interface
for its client to use. The change involves:
* Encode the macro definition inside the binary module
* Resolve macro modules in the dependencies scanners, including those
declared inside the dependency modules.
* Propagate the macro defined from the direct dependencies to track
all the potentially available modules inside a module compilation.
When scanning swift modules and constructing their build commands, there
is no need to pass any external plugin search paths if there are no macro
dependencies for the module.
rdar://135221984
Fix a bug that swift clang importer is not setup correctly when prefix
map is used. There are two separate issues:
* CC1 args used to setup clang import when building swift (interface and
sources) are not correctly remapped.
* When loading SDKInfo from SDK path, the SDKSettings.json is not
loading from VFS, thus the file cannot be loaded from remapped path.
rdar://134458611
To materialize the object file correctly on a cache hit when MCCAS is
enabled, if lib_InternalSwiftScan.dylib is used to detect the cache hit,
we need to add MCCAS support to the replay code in SwiftCaching.cpp
When direct cc1 mode is used, the cc1 arguments are generated from extra
clang targets so there is no need to add this extra argument.
Furthermore, the `--target=` is a clang driver argument and is not valid
under cc1 mode.
Needed for: rdar://134090397
Don't try to figure out the executable names during replay from
libSwiftScan.dylib. The actual executable path for the process actually
doesn't matter in this case to reconstruct the invocation and might
actually be misleading.
Just use `swift-frontend` as a placeheader executable name for
in-process cache replay.
rdar://132758308
