Teach scanner to respect the working directory set in the invocation
through scanner C API.
Also add test infrastructure to testing scanner from C API. Break up
DependencyScan lib into two so the swift-scan-test and remain small
without understanding swift AST.
rdar://127626011
This change modifies the dependency scanner to keep track of source locations of each encountered 'import' statement, in order to be able to emit diagnostics with source locations if an import failed to resolve.
- Keep track of each 'import' statement's source buffer, line number, and column number when adding it. The dependency scanner utilizes separate compilation instances, and therefore separate Source Managers for scanning `import` statements of user sources and textual interfaces of Swift dependencies. Since import resolution may happen in the main scanner compilation instance while the `import` itself was found by an interface-scanning sub-instance, we cannot simply hold on to the import's `SourceLoc`.
- Add libSwiftScan API for diagnostics to carry above source locations to clients.
Add a fast path to create swift CompilerInstance when it is only used to
replay output when there is a cache hit. The normal `setup` function is
very expensive to call, especially in cache mode to setup inputs, and it
needs to be called once per input file from libSwiftScan API due to the
current caching granularity.
The fast path will only construct the part that is needed for output
replay, including the CAS, the output backend and caching diagnostic
processor.
rdar://127062609
LLVM is gearing up to move to `std::endianness` and as part of that has
moved `llvm::support::endianness` to `llvm::endianness`
(bbdbcd83e6702f314d147a680247058a899ba261). Rename our uses.
`OptTable` was a source of consistent churn due to new arguments to the
`OPTION` macro. LLVM 3f092f37b7362447cbb13f5502dae4bdd5762afd extracted
the handling of the common option parts (eg. an ID and an info) out into
separate macros to reduce this - use those here (since unsurprisingly,
more arguments were added).
Otherwise they may have module dependencies of their own which will not be detected by the scanner and included in the list of explicit inputs for compilation.
The response file expansion function doesn't return all the arguments
allocated using StringSaver produced. It only use the StringSaver to
save the newly expanded arguments. Need to make sure all the original
arguments are saved so they can be used later during replay.
rdar://121808135
Build system can pass response file to libSwiftScan as the command-line
arguments. libSwiftScan needs to expand the response file before
performing tasks.
rdar://121291342
`llvm::cl::ParseCommandLineOptions()` is not thread-safe to be called
from libSwiftScan so replaying multiple commands in parallel can crash
the build system. Since the replay instance is not used for compilation
and only need information from command-line arguments to create outputs
and diagnostics, ignore LLVMArgs when replaying.
rdar://120423882
Avoid path encoding difference (for example, real_path vs. path from
symlink) by eliminating the path from cache key. Cache key is now
encoded with the index of the input file from all the input files from
the command-line, reguardless if those inputs will produce output or
not. This is to ensure stable ordering even the batching is different.
Add a new cache computation API that is preferred for using input index
directly. Old API for cache key is deprecated but still updated to
fallback to real_path comparsion if needed.
As a result of swift scan API change, rename the feature in JSON file to
avoid version confusion between swift-driver and libSwiftScan.
rdar://119387650
Add a new async download API from CASID. This helps build system better
schedule the job/remove duplicated downloads, than using an opaque
cached_output handle.
ASTGen always builds with the host Swift compiler, without requiring
bootstrapping, and is enabled in more places. Move the regex literal
parsing logic there so it is enabled in more host environments, and
makes use of CMake's Swift support. Enable all of the regex literal
tests when ASTGen is built, to ensure everything is working.
Remove the "AST" and "Parse" Swift modules from SwiftCompilerSources,
because they are no longer needed.
Add new APIs libSwiftScan that can be used for cache query and cache
replay. This enables swift-driver or build system to query the cache and
replay the compilation results without invocation swift-frontend for
better scheduling.
Conflicts:
- `lib/AST/TypeCheckRequests.cpp` renamed `isMoveOnly` which requires
a static_cast on rebranch because `Optional` is now a `std::optional`.
The code, previously, only properly handled such dependencies being a distinct category for Swift source and Swift textual dependency infos. Swift binary module dependencies must handle this similarly and this change adds the missing support for them. Recent refactor of the scanner also means that now Swift binary dependencies with Swift overlay dependencies may crash the scanner, and this change resolves this as well.
Resolves rdar://117088840
Update swift cache key computation mechanism from one cache key per
output, to one cache key per primary input file (for all outputs that
associated with that input).
The new schema allows fewer cache lookups while still preserving most of
the flexibility for batch mode and incremental mode.
Conflicts:
- `CMakeLists.txt` caused by the extra `-D` added in rebranch to
reduce the number of deprecation warnings.
- `lib/Frontend/PrintingDiagnosticConsumer.cpp` caused by the removal
of one of the `#if SWIFT_SWIFT_PARSER` on rebranch (probably should
have been done on main).
Use FetchContent to include swift-syntax directly in swift. This can be
thought of as an `add_subdirectory` for a directory outside the root.
The default build directory will be `_deps/swiftsyntax-subbuild/`, though
the modules and shared libraries will be built in `lib/swift/host` by
passing down `SWIFT_HOST_LIBRARIES_DEST_DIR` to avoid copying them as we
were doing previously.
libSwiftScan is built in 'lib' but installed in 'lib/swift/host' RUNPATH
should have correct '$ORIGIN/../{platform}' to load 'swiftCore' runtime
library.
For compiling codes required for macro support, we now need swiftc
compiler in the build machine.
Unlike Darwin OSes, where swiftCore runtime is guaranteed to be present
in /usr/lib, Linux doesn't have ABI stability and the stdlib of the
build machine is not at the specific location. So the built compiler
cannot relies on the shared object in the toolchain.
LLVM install_symlink takes a new argument on whether to create a symlink
or copy binaries when run. -- https://reviews.llvm.org/D145443
The variable that controls this in LLVM is `LLVM_USE_SYMLINKS`, which
defaults to `ON` on Unix-y hosts, but otherwise is false so that Windows
works. This is a configurable option, so Windows configs that can
support symlinks can take advantage of symlinks and save some space.
`LLVM_USE_SYMLINKS` is not exported from LLVM though, so we can't see
it to use. Instead, we have `SWIFT_USE_SYMLINKS`.
As of CMake 3.25, there are now global variables `LINUX=1`, `ANDROID=1`,
etc. These conflict with expressions that used these names as unquoted
strings in positions where CMake accepts 'variable|string', for example:
- `if(sdk STREQUAL LINUX)` would fail, because `LINUX` is now defined and
expands to 1, where it would previously coerce to a string.
- `if(${sdk} STREQUAL "LINUX")` would fail if `sdk=LINUX`, because the
left-hand side expands twice.
In this patch, I looked for a number of patterns to fix up, sometimes a
little defensively:
- Quoted right-hand side of `STREQUAL` where I was confident it was
intended to be a string literal.
- Removed manual variable expansion on left-hand side of `STREQUAL`,
`MATCHES` and `IN_LIST` where I was confident it was unintended.
Fixes#65028.
Teach swift dependency scanner to use CAS to capture the full dependencies for a build and construct build commands with immutable inputs from CAS.
This allows swift compilation caching using CAS.
Instead of being a part of 'directDependencies' on a module dependency info, make them a separate array of dependency IDs for Swift Source and Textual modules.
This will allow clients to still distinguish direct module dependencies imported from a given module, versus dependencies added because direct/transitive Clang module dependencies have Swift overlays.
This change does *not* remove overlay dependencies from 'directDependencies' yet, just adds them as a separate field on the module details info. A followup change will remove overlay and bridging header dependencies from 'directDependencies' once the clients have had a chance to adopt to this change.
This new version takes the path to the compiler executable as a parameter, in order for libSwiftScan to compute compiler-relative portions of runtimeLibraryPaths, runtimeResourcePath. V1, without knowing the path to the compiler executable, produced incomplete sets of these paths.
Previously no API returned a raw 'swiftscan_string_ref_t' directly so this was not needed.
However, 'swiftscan_compiler_target_info_query' returns a string directly, which the clients need to clean up.
