Make swift caching remarks more friendly to use by:
* Allow environmental variable `SWIFT_ENABLE_COMPILE_CACHE_REMARK` to
control if the remarks are emitted or not. This avoids updating and
reconfiguring build flags just to see the caching remarks.
* Add a new cache hit remark on top of the existing cache replay output
remarks. Cache output replay remark is one per output, while cache hit
remark is one per invocation. This allows better cache hit/miss rate
calculation just based on the number of remarks emitted.
rdar://178563432
For cached output replay or storing compiler produced output, the CAS
should perform remote lookup/store if possible. This allows distributed
caching from plain `swiftc` invocation and unlocks distributed caching
from build systems like `cmake`.
If there is no remote CAS configured, this is NFC.
rdar://175436790
Fix that the `file` fields in ConstExtract JSON output are all prefix
mapped path when caching and prefix mapping are all enabled. This can
cause the consumers of the output failed to find the source file.
Now the JSON file is correct updated with the correct source file
location when caching is on. This allows both un-prefix-mapped source
location and relocatable source file location. As a bonus, now CAS
stores the compact version of the JSON file to save space without
affecting the pretty formatted final output file.
rdar://173736094
Make sure cache replay matches the same behavior as normal compilation
that the output is not written (timestamp preserved) when replaying
cache hits. This make sure the downstream dependencies are not
invalidated on a cache hit but the output doesn't change.
With the new option, when doing caching we write the hash that we already
computed for the main output file to an extended attribute (xattr) on the file.
This is equivalent to clang's -fwrite-output-hash-xattr option.
The format of the xattr is
name: com.apple.clang.cas_output_hash
data:
* Null-terminated hash schema name, e.g. llvm.builtin.v2[BLAKE3].
* Hash length (4 bytes, little-endian).
* Hash bytes
rdar://171185394
Fix the debug info emitted for bridging header PCH when compilation
caching is used. This includes:
* Bridging header should have the correct dwo_name that uses CASID
* The search_path for PCH should be the cache key that can load the PCH
just like other modules.
Using IncludeTree::FileList to concat the include tree file systems that
are passed on the command-line. This significantly reduce the
command-line size, and also makes the cache key computation a lot
faster.
rdar://148752988
Improve diagnostics message for swift caching build by trying to emit
the diagnostics early when there is more context to differentiate the
different kind of problems.
After the improvement, CAS Error should be more closer to when there is
functional problem with the CAS, rather than mixing in other kinds of
problem (like scanning dependency failures) when operating with a CAS.
rdar://145676736
There were two cache replay code exists, one for cache replay from
swift-frontend, the other for replay using C API from libSwiftScan. It
is easy to forget to update one copy when new specialized cache replay
logic is added for some output kinds. Now unify the replay logics to a
single location to avoid confusion. This is a rewrite of the existing
logic and NFCI.
The symbol graph output from a module can contain an arbitrary number of
files, depending on what extensions it contains, so cache a list of
symbol graph files with their base name and contents so that they can be
replayed.
rdar://140286819
If the output loading failed after cache key lookup, treat that as a
warning and resume as if that is a cache miss. This is not a valid
configuration for builtin CAS but can happen for a remote CAS service
that failed to serve the output. Instead of failing, we should continue
to compile to avoid disruptive failures.
rdar://140822432
Use IncludeTreeFileList instead of full feature CASFS for swift
dependency filesystem. This allows smaller CAS based VFS that is smaller
and faster. This is enabled by the CAS enabled compilation does not
need to iterate file system.
rdar://136787368
Fully support make-style `.d` dependencies file output by making
following improvements:
* All correct dependency file render when cache hit for a different
output file location. The dependency file should list the correct
output path, not the stale output path for the initial compilation
* When enable a path prefix mapper to canonicalize the path, the
dependency file should render the input file correctly as the input
file path on disk.
rdar://132250067
This patch adds support for MCCAS when a cache hit is encountered when
trying to replay a compilation, and uses the MCCAS serialization code
to materialize the object file that is the main output of the
compilation.
Although I don't plan to bring over new assertions wholesale
into the current qualification branch, it's entirely possible
that various minor changes in main will use the new assertions;
having this basic support in the release branch will simplify that.
(This is why I'm adding the includes as a separate pass from
rewriting the individual assertions)
Avoid path encoding difference (for example, real_path vs. path from
symlink) by eliminating the path from cache key. Cache key is now
encoded with the index of the input file from all the input files from
the command-line, reguardless if those inputs will produce output or
not. This is to ensure stable ordering even the batching is different.
Add a new cache computation API that is preferred for using input index
directly. Old API for cache key is deprecated but still updated to
fallback to real_path comparsion if needed.
As a result of swift scan API change, rename the feature in JSON file to
avoid version confusion between swift-driver and libSwiftScan.
rdar://119387650
Teach swift-frontend to correctly replay cache when using
batch-mode/primary-file. Currently, the cache replay logic will try to
look up cache key for all the inputs and only replay if all lookups are
successful. In batch-mode, not all inputs are producing outputs, which
causes lookup for those keys to be a miss and causing a false miss for
the compilation.
Add new APIs libSwiftScan that can be used for cache query and cache
replay. This enables swift-driver or build system to query the cache and
replay the compilation results without invocation swift-frontend for
better scheduling.
Change how cached diagnostics are stored inside the CAS. It used to be
stored as a standalone entry for a frontend invocation in the cache and
now it is switched to be associated with input files, stored together
with other outputs like object files, etc.
This enables cleaner Cache Replay APIs and future cached diagnostics
that can be splitted up by file contribution.
Older compilers don't have the copy-elision of newer C++ compilers,
resulting in an error message when returning a non-copyable type. Move
llvm::Error instead of copying it.
Update swift cache key computation mechanism from one cache key per
output, to one cache key per primary input file (for all outputs that
associated with that input).
The new schema allows fewer cache lookups while still preserving most of
the flexibility for batch mode and incremental mode.
Allow DependencyScanner to canonicalize path using a prefix map. When
option `-scanner-prefix-map` option is used, dependency scanner will
remap all the input paths in following:
* all the paths in the CAS file system or clang include tree
* all the paths related to input on the command-line returned by scanner
This allows all the input paths to be canonicalized so cache key can be
computed reguardless of the exact on disk path.
The sourceFile field is not remapped so build system can track the exact
file as on the local file system.
Rename `-enable-cas` to `-compile-cache-job` to align with clang option
names and promote that to a new driver only flag.
Few other additions to driver flag for caching behaviors:
* `-compile-cache-remarks`: now cache hit/miss remarks are guarded behind
this flag
* `-compile-cache-skip`: skip replaying from the cache. Useful as a
debugging tool to do the compilation using CAS inputs even the output
is a hit from the cache.
Teach swift dependency scanner to use CAS to capture the full dependencies for a build and construct build commands with immutable inputs from CAS.
This allows swift compilation caching using CAS.
Steven Wu
Ben Langmuir
Shubham Sandeep Rastogi
Tim Kientzle
Ben Barham
Evan Wilde