For cached output replay or storing compiler produced output, the CAS
should perform remote lookup/store if possible. This allows distributed
caching from plain `swiftc` invocation and unlocks distributed caching
from build systems like `cmake`.
If there is no remote CAS configured, this is NFC.
rdar://175436790
Fix that the `file` fields in ConstExtract JSON output are all prefix
mapped path when caching and prefix mapping are all enabled. This can
cause the consumers of the output failed to find the source file.
Now the JSON file is correct updated with the correct source file
location when caching is on. This allows both un-prefix-mapped source
location and relocatable source file location. As a bonus, now CAS
stores the compact version of the JSON file to save space without
affecting the pretty formatted final output file.
rdar://173736094
With the new option, when doing caching we write the hash that we already
computed for the main output file to an extended attribute (xattr) on the file.
This is equivalent to clang's -fwrite-output-hash-xattr option.
The format of the xattr is
name: com.apple.clang.cas_output_hash
data:
* Null-terminated hash schema name, e.g. llvm.builtin.v2[BLAKE3].
* Hash length (4 bytes, little-endian).
* Hash bytes
rdar://171185394
Temporarily disable replaying optimization remarks when cache hit.
Optimization remarks file are not setup to be cached because:
* the write of such files are not going through output backend
* when optimization remarks files are requested, it produces files that
are not intended to be produced in SupplementoryOutputs, which
confuses compiler when storing and loading compilation results, and
causes cache misses all the time when optimization remark file is
reuqested.
Disable optimization remarks for cache replay so it is currently only
produced when compiler actually did the compile locally.
It is technically possible for different threads trying to write output
files at the same time, e.g. multi-threaded WMO. It can cause a race
condition to update the output status in the CAS output backend. Use a
lock to protect the update and finalization of CAS entries.
Temporarily make action cache update non-fatal and not failing the job.
The action cache update failure can happen due to cache poisoning from
non-deterministic compiler output. Assume the errors from the
non-determinism is benign and can be ignored till all issues are
resolved.
rdar://146780363
The symbol graph output from a module can contain an arbitrary number of
files, depending on what extensions it contains, so cache a list of
symbol graph files with their base name and contents so that they can be
replayed.
rdar://140286819
Fully support make-style `.d` dependencies file output by making
following improvements:
* All correct dependency file render when cache hit for a different
output file location. The dependency file should list the correct
output path, not the stale output path for the initial compilation
* When enable a path prefix mapper to canonicalize the path, the
dependency file should render the input file correctly as the input
file path on disk.
rdar://132250067
Although I don't plan to bring over new assertions wholesale
into the current qualification branch, it's entirely possible
that various minor changes in main will use the new assertions;
having this basic support in the release branch will simplify that.
(This is why I'm adding the includes as a separate pass from
rewriting the individual assertions)
This is a regression causing lots of cached diagnostics tests not
functioning since the cached diagnostics processors are not initialized
for those tests which are supposed to test diagnostics caching.
The regression is caused by the fix that the typecheck module interface
job need to run a typecheck job in the sub-invocation. Now the typecheck
module interface job is correctly setup to avoid diagnostics about
unsupported file system error.
Avoid path encoding difference (for example, real_path vs. path from
symlink) by eliminating the path from cache key. Cache key is now
encoded with the index of the input file from all the input files from
the command-line, reguardless if those inputs will produce output or
not. This is to ensure stable ordering even the batching is different.
Add a new cache computation API that is preferred for using input index
directly. Old API for cache key is deprecated but still updated to
fallback to real_path comparsion if needed.
As a result of swift scan API change, rename the feature in JSON file to
avoid version confusion between swift-driver and libSwiftScan.
rdar://119387650
Change how cached diagnostics are stored inside the CAS. It used to be
stored as a standalone entry for a frontend invocation in the cache and
now it is switched to be associated with input files, stored together
with other outputs like object files, etc.
This enables cleaner Cache Replay APIs and future cached diagnostics
that can be splitted up by file contribution.
Update swift cache key computation mechanism from one cache key per
output, to one cache key per primary input file (for all outputs that
associated with that input).
The new schema allows fewer cache lookups while still preserving most of
the flexibility for batch mode and incremental mode.
Argyrios Kyrtzidis
Steven Wu
Ben Langmuir
Anthony Latsis
Shubham Sandeep Rastogi
Tim Kientzle
Ben Barham