When Swift passes search paths to clang, it does so directly into the HeaderSearch. That means that those paths get ordered inconsistently compared to the equivalent clang flag, and causes inconsistencies when building clang modules with clang and with Swift. Instead of touching the HeaderSearch directly, pass Swift search paths as driver flags, just do them after the -Xcc ones.
Swift doesn't have a way to pass a search path to clang as -isystem, only as -I which usually isn't the right flag. Add an -Isystem Swift flag so that those paths can be passed to clang as -isystem.
rdar://93951328
Previous behavior had the scanner simply proceed if the header input of a binary Swift module dependency could not be resolved on the filesystem
Resolves rdar://139736789
This change refactors the top-level dependency scanning flow to follow the following procedure:
Scan():
1. From the source target under scan, query all imported module identifiers for a *Swift* module. Leave unresolved identifiers unresolved. Proceed transitively to build a *Swift* module dependency graph.
2. Take every unresolved import identifier in the graph from (1) and, assuming that it must be a Clang module, dispatch all of them to be queried in-parallel by the scanner's worker pool.
3. Resolve bridging header Clang module dpendencies
4. Resolve all Swift overlay dependencies, relying on all Clang modules collected in (2) and (3)
5. For the source target under scan, use all of the above discovered module dependencies to resolve all cross-import overlay dependencies
Use IncludeTreeFileList instead of full feature CASFS for swift
dependency filesystem. This allows smaller CAS based VFS that is smaller
and faster. This is enabled by the CAS enabled compilation does not
need to iterate file system.
rdar://136787368
Use `-fsyntax-only` action to scan clang module dependencies instead of
`-c` option. This fixes a non-deterministic output on windows from
scan-dependency output because `-c` implies it needs a temporary object
file in the cc1 arguments that makes the pcm compilation command
different every run. This can also make the `-Xcc` commands for PCM
compilation simpler and more likely to be deduplicated by build system.
rdar://135319536
Avoid constructing tasks with `-Xcc` options that references clang
CASOptions. This is going to make the cache hit/miss to be dependent on
CAS location/configuraitons.
Instead, only give swift CASOptions to constructed tasks and propagate
the configurations to underlying clang importer.
rdar://132255889
Although I don't plan to bring over new assertions wholesale
into the current qualification branch, it's entirely possible
that various minor changes in main will use the new assertions;
having this basic support in the release branch will simplify that.
(This is why I'm adding the includes as a separate pass from
rewriting the individual assertions)
There is no need to pass output path of the PCM compilation twice, once
as swift `-o` flag and other time as `-Xcc -o` flag to clang importer.
This can also cause swift caching to miss when output path is different
because `-Xcc` options are not modeled in caching model to understand
`-o` output path doesn't affect compilation output content.
rdar://128650954
This fixes -gmodules when caching by adopting the new clang -cc1 option
-finclude-tree-preserve-pch-path. -gmodules is required to make
debugging work when examining types that come from clang modules or
bridging headers, but was previously being disabled by clang's caching
support.
rdar://126370706
Support `-vfsoverlay` swift option for explicit module build (including
caching build). Previously, if the interface file is discovered from a
location that is remapped by overlay, module cannot be built correctly.
Make sure the overlay options are passed down to all interface
compilaiton command.
For caching build, need to make sure the overlay itself is part of the CAS
file system so the downstream compilation can discover that.
rdar://123655183
LLVM is presumably moving towards `std::string_view` -
`StringRef::startswith` is deprecated on tip. `SmallString::startswith`
was just renamed there (maybe with some small deprecation inbetween, but
if so, we've missed it).
The `SmallString::startswith` references were moved to
`.str().starts_with()`, rather than adding the `starts_with` on
`stable/20230725` as we only had a few of them. Open to switching that
over if anyone feels strongly though.
Otherwise they may have module dependencies of their own which will not be detected by the scanner and included in the list of explicit inputs for compilation.
When prefix mapping paths that are used in clang, ensure we are
consistently using the same prefix mapper from clang. This prevents
mismatches that could cause modules to fail to load.
rdar://123324072
Fix the bridging header dependencies calculation for explicit module
build, especially for caching which needs an accurate list of deps for
compute cache key correctly.
Previously, the bridging header deps are computed from `ModuleGraph`
from the clang dependency scanner, which can be affected by already seen
modules. It causes the dependencies to be missing for bridging header if
the module is seen by main swift source module.
Now report only the directly module dependencies from bridging header,
then compute all the transitive dependencies before calculating all the
cache keys.
rdar://123156636
Currently, `-direct-clang-cc1-module-build` and `-only-use-extra-clang-opts`
have to be passed together for clang importer creation to succeed.
Missing either will result in error. Simplified the swift-frontend flags
by removing `-only-use-extra-clang-opts` and let
`-direct-clang-cc1-module-build` to do both.
Re-write and clean up how clang-importer is created from clang
arguments. Previously, it is unclear if `getClangArguments` will return
CC1 args or driver args and the logic is unnecessarily compilicated when
creating clang invocation. Now clang invocation is always created from
cc1 arguments, which can be directly provided via direct-cc1-mode or
converted from driver args.
There is no functional changes in this patch, other than
`-dump-clang-diagnostics` now will always print cc1 args, and also
driver args if that is applicable.
Switch to use clang-include-tree by default for clang module
building/caching when using a CAS. This is the default mode for clang
module and has less issues than CAS file system based implementation.
Allow DependencyScanner to canonicalize path using a prefix map. When
option `-scanner-prefix-map` option is used, dependency scanner will
remap all the input paths in following:
* all the paths in the CAS file system or clang include tree
* all the paths related to input on the command-line returned by scanner
This allows all the input paths to be canonicalized so cache key can be
computed reguardless of the exact on disk path.
The sourceFile field is not remapped so build system can track the exact
file as on the local file system.
From being a scattered collection of 'static' methods in ScanDependencies.cpp
and member methods of ASTContext. This makes 'ScanDependencies.cpp' much easier
to read, and abstracts the actual scanning logic away to a place with common
state which will make it easier to reason about in the future.
Instead of the code querying the compiler's built-in Clang instance, refactor the
dependency scanner to explicitly keep track of module output path. It is still
set according to '-module-cache-path' as it has been prior to this change, but
now the scanner can use a different module cache for scanning PCMs, as specified
with '-clang-scanner-module-cache-path', without affecting module output path.
Resolves rdar://113222853
Reformatting everything now that we have `llvm` namespaces. I've
separated this from the main commit to help manage merge-conflicts and
for making it a bit easier to read the mega-patch.
This is phase-1 of switching from llvm::Optional to std::optional in the
next rebranch. llvm::Optional was removed from upstream LLVM, so we need
to migrate off rather soon. On Darwin, std::optional, and llvm::Optional
have the same layout, so we don't need to be as concerned about ABI
beyond the name mangling. `llvm::Optional` is only returned from one
function in
```
getStandardTypeSubst(StringRef TypeName,
bool allowConcurrencyManglings);
```
It's the return value, so it should not impact the mangling of the
function, and the layout is the same as `std::optional`, so it should be
mostly okay. This function doesn't appear to have users, and the ABI was
already broken 2 years ago for concurrency and no one seemed to notice
so this should be "okay".
I'm doing the migration incrementally so that folks working on main can
cherry-pick back to the release/5.9 branch. Once 5.9 is done and locked
away, then we can go through and finish the replacement. Since `None`
and `Optional` show up in contexts where they are not `llvm::None` and
`llvm::Optional`, I'm preparing the work now by going through and
removing the namespace unwrapping and making the `llvm` namespace
explicit. This should make it fairly mechanical to go through and
replace llvm::Optional with std::optional, and llvm::None with
std::nullopt. It's also a change that can be brought onto the
release/5.9 with minimal impact. This should be an NFC change.
Teach swift dependency scanner to use CAS to capture the full dependencies for a build and construct build commands with immutable inputs from CAS.
This allows swift compilation caching using CAS.
The Swift compiler does not have a concept of a working directory. It is instead handled by the Swift driver by resolving relative paths according to the driver's working directory argument. On the other hand, Clang does have a concept working directory which may be specified on this Clang invocation with '-working-directory'. If so, it is crucial that we use this directory as an argument to the Clang scanner API. Otherwiswe, we risk having a mismatch between the working directory specified on the scanner's Clang invocation and the one use from the scanner API entry-points, which leads to downstream inconsistencies and errors.
This was originally fixed for the main by-name module dependencies query in https://github.com/apple/swift/pull/61025 (03136e06aa), but the Bridging Header dependencies code has continued to incorrectly expect the Swift ASTContext to both have the working directory set *and* be consistent with that of the Clang scanner instance.
Resolves rdar://108464467
For a `@Testable` import in program source, if a Swift interface dependency is discovered, and has an adjacent binary `.swiftmodule`, open up the module, and pull in its optional dependencies. If an optional dependency cannot be resolved on the filesystem, fail silently without raising a diagnostic.
This is currently always done when instantiating `ClangImporter` by manually setting the option on the Clang invocation with:
```
Invocation->getCodeGenOpts().DebugTypeExtRefs = true
```
Now also add it to `importer::addCommonInvocationArguments` so that the dependency scanner always generates command-lines with the required for Swift Clang flags.
Resolves rdar://107570568
