Reformatting everything now that we have `llvm` namespaces. I've
separated this from the main commit to help manage merge-conflicts and
for making it a bit easier to read the mega-patch.
This is phase-1 of switching from llvm::Optional to std::optional in the
next rebranch. llvm::Optional was removed from upstream LLVM, so we need
to migrate off rather soon. On Darwin, std::optional, and llvm::Optional
have the same layout, so we don't need to be as concerned about ABI
beyond the name mangling. `llvm::Optional` is only returned from one
function in
```
getStandardTypeSubst(StringRef TypeName,
bool allowConcurrencyManglings);
```
It's the return value, so it should not impact the mangling of the
function, and the layout is the same as `std::optional`, so it should be
mostly okay. This function doesn't appear to have users, and the ABI was
already broken 2 years ago for concurrency and no one seemed to notice
so this should be "okay".
I'm doing the migration incrementally so that folks working on main can
cherry-pick back to the release/5.9 branch. Once 5.9 is done and locked
away, then we can go through and finish the replacement. Since `None`
and `Optional` show up in contexts where they are not `llvm::None` and
`llvm::Optional`, I'm preparing the work now by going through and
removing the namespace unwrapping and making the `llvm` namespace
explicit. This should make it fairly mechanical to go through and
replace llvm::Optional with std::optional, and llvm::None with
std::nullopt. It's also a change that can be brought onto the
release/5.9 with minimal impact. This should be an NFC change.
Teach swift dependency scanner to use CAS to capture the full dependencies for a build and construct build commands with immutable inputs from CAS.
This allows swift compilation caching using CAS.
The Swift compiler does not have a concept of a working directory. It is instead handled by the Swift driver by resolving relative paths according to the driver's working directory argument. On the other hand, Clang does have a concept working directory which may be specified on this Clang invocation with '-working-directory'. If so, it is crucial that we use this directory as an argument to the Clang scanner API. Otherwiswe, we risk having a mismatch between the working directory specified on the scanner's Clang invocation and the one use from the scanner API entry-points, which leads to downstream inconsistencies and errors.
This was originally fixed for the main by-name module dependencies query in https://github.com/apple/swift/pull/61025 (03136e06aa), but the Bridging Header dependencies code has continued to incorrectly expect the Swift ASTContext to both have the working directory set *and* be consistent with that of the Clang scanner instance.
Resolves rdar://108464467
For a `@Testable` import in program source, if a Swift interface dependency is discovered, and has an adjacent binary `.swiftmodule`, open up the module, and pull in its optional dependencies. If an optional dependency cannot be resolved on the filesystem, fail silently without raising a diagnostic.
This is currently always done when instantiating `ClangImporter` by manually setting the option on the Clang invocation with:
```
Invocation->getCodeGenOpts().DebugTypeExtRefs = true
```
Now also add it to `importer::addCommonInvocationArguments` so that the dependency scanner always generates command-lines with the required for Swift Clang flags.
Resolves rdar://107570568
- '-o <output_path>'
- '-disable-implicit-swift-modules'
- '-Xcc -fno-implicit-modules' and '-Xcc -fno-implicit-module-maps'
- '-candidate-module-file'
These were previously supplied by the driver. Instead, they will now be ready to be run directly from the dependency scanner's output.
This changes the scanner's behavior to "resolve" a discovered module's dependencies to a set of Module IDs: module name + module kind (swift textual, swift binary, clang, etc.).
The 'ModuleDependencyInfo' objects that are stored in the dependency scanner's cache now carry a set of kind-qualified ModuleIDs for their dependencies, in addition to unqualified imported module names of their dependencies.
Previously, the scanner's internal state would cache a module dependnecy as having its own set of dependencies which were stored as names of imported modules. This led to a design where any time we needed to process the dependency downstream from its discovery (e.g. cycle detection, graph construction), we had to query the ASTContext to resolve this dependency's imports, which shouldn't be necessary. Now, upon discovery, we "resolve" a discovered dependency by executing a lookup for each of its imported module names (this operation happens regardless of this patch) and store a fully-resolved set of dependencies in the dependency module info.
Moreover, looking up a given module dependency by name (via `ASTContext`'s `getModuleDependencies`) would result in iterating over the scanner's module "loaders" and querying each for the module name. The corresponding modules would then check the scanner's cache for a respective discovered module, and if no such module is found the "loader" would search the filesystem.
This meant that in practice, we searched the filesystem on many occasions where we actually had cached the required dependency, as follows:
Suppose we had previously discovered a Clang module "foo" and cached its dependency info.
-> ASTContext.getModuleDependencies("foo")
--> (1) Swift Module "Loader" checks caches for a Swift module "foo" and doesn't find one, so it searches the filesystem for "foo" and fails to find one.
--> (2) Clang Module "Loader" checks caches for a Clang module "foo", finds one and returns it to the client.
This means that we were always searching the filesystem in (1) even if we knew that to be futile.
With this change, queries to `ASTContext`'s `getModuleDependencies` will always check all the caches first, and only delegate to the scanner "loaders" if no cached dependency is found. The loaders are then no longer in the business of checking the cached contents.
To handle cases in the scanner where we must only lookup either a Swift-only module or a Clang-only module, this patch splits 'getModuleDependencies' into an alrady-existing 'getSwiftModuleDependencies' and a newly-added 'getClangModuleDependencies'.
`getValue` -> `value`
`getValueOr` -> `value_or`
`hasValue` -> `has_value`
`map` -> `transform`
The old API will be deprecated in the rebranch.
To avoid merge conflicts, use the new API already in the main branch.
rdar://102362022
Move clangScanningTool and clangScanningService to be parts of 'ModuleDependenciesCache' state, getting rid of the 'ClangModuleDependenciesCacheImpl', which is no-longer needed since we moved moved to by-name lookup of Clang modules.
The Swift compiler does not have a concept of a working directory. It is instead handled by the Swift driver by resolving relative paths according to the driver's working directory argument. On the other hand, Clang does have a concept working directory which may be specified on this Clang invocation with '-working-directory'. If so, it is crucial that we use this directory as an argument to the Clang scanner API. Otherwiswe, we risk having a mismatch between the working directory specified on the scanner's Clang invocation and the one use from the scanner API entry-points, which leads to downstream inconsistencies and errors.
apple/llvm-project 000a90ab63ce0595d442b3e7e8adc9c0506d2525 removed
`getAdditionalArgsWithoutModulePaths`.
092b2ddf7a swapped to using
`getCommandLineWithoutModulePaths` instead, but this is (almost) all cc1
args used to build the module - which is very different to what was
being looped over previously.
Inline `getAdditionalArgsWithoutModulePaths` instead.
Resolves rdar://94545725.
apple/llvm-project 000a90ab63ce0595d442b3e7e8adc9c0506d2525 removed
`getAdditionalArgsWithoutModulePaths`. Based on the v2 libclang API, we
should be using `getCommandLineWithoutModulePaths` now.
When looking for a Swift module on disk, we were scanning all module search paths if they contain the module we are searching for. In a setup where each module is contained in its own framework search path, this scaled quadratically with the number of modules being imported. E.g. a setup with 100 modules being imported form 100 module search paths could cause on the order of 10,000 checks of `FileSystem::exists`. While these checks are fairly fast (~10µs), they add up to ~100ms.
To improve this, perform a first scan of all module search paths and list the files they contain. From this, create a lookup map that maps filenames to the search paths they can be found in. E.g. for
```
searchPath1/
Module1.framework
searchPath2/
Module1.framework
Module2.swiftmodule
```
we create the following lookup table
```
Module1.framework -> [searchPath1, searchPath2]
Module2.swiftmodule -> [searchPath2]
```
The clang importer sets the clang resource directory to
`RuntimeResourcePath`/clang (usually "lib/swift/clang", a symbolic link
into "lib/clang/<version>"). This was inadvertently being removed in a
check to remove the clang *executable* path.
Modify that check to only the first argument so the resource directory
is properly passed through.
Doing so will allow clients to know which Swift-specific PCM arguments are already captured from the scan that first discovered this module.
SwiftDriver, in particular, will be able to use this information to avoid re-scanning a given Clang module if the initial scan was sufficient for all possible sets of PCM arguments on Swift modules that depend on said Clang module.
And only resolve cached dependencies that came from scanning actions with the same target triple.
This change means that the `GlobalModuleDependenciesCache` must be configured with a specific target triple for every scannig action, and it will only resolve previously-found dependencies from previous scannig actions using the exact same triple.
Furthermore, the `GlobalModuleDependenciesCache` separately tracks source-file-based module dependencies as those represent main Swift modules of previous scanning actions, and we must be able to resolve those regardless of the target triple.
Resolves rdar://83105455
The SingleCommandCompilationDatabase type was removed from LLVM and the
API's that took it were changed to just take the vector of arguments
instead. This patch updates the calls to `getFullDependencies` to pass
in the argument vector to reflect that change and get things building
again.
These kinds of modules differ from `SwiftTextual` modules in that they do not have an interface and have source-files.
It is cleaner to enforce this distinction with types, instead of checking for interface optionality everywhere.
The dependency scanner's cache persists across different queries and answering a subsequent query's module lookup with a module not in the query's search path is not correct.
For example, suppose we are looking for a Swift module `Foo` with a set of search paths `SP`.
And dependency scanner cache already contains a module `Foo`, for which we found an interface file at location `L`. If `L`∉`SP`, then we cannot re-use the cached entry because we’d be resolving the scanning query to a filesystem location that the current scanning context is not aware of.
Resolves rdar://81175942
https://github.com/apple/swift/pull/37774 (related to rdar://72480261) has made it so that the target of built clang modules (even downstream from Swift interface dependencies) can be consistent with that of the main module. This means that when building transitive Clang dependencies of the main module, they will always have a matching triple to the main module itself (ensured with `-clang-target`). This means we no longer have to report the target triple in the set of `extraPCMArgs` which encode an interface-specific requirement for building its dependencies.
Pass a wrapped VFS down into `clang::createInvocationFromCommandLine` so
that the working directory is set and then used in the underlying Clang
`CompilerInstance`.
Fixes the possibility of differing modules hashes when the same
arguments are used in Clang directly vs from the importer.
Resolves rdar://79376364
This value was replaced with a call to 'getNonPathCommandLine()' in
llvm-project commit 2b73565210ef0b3b29e0f067eef150a32adbb967. The API
was later renamed 'getAdditionalArgsWithoutModulePaths()' in
llvm-project commit 77fc4d1feef5f3041f614206447b4461992054bb.
