To materialize the object file correctly on a cache hit when MCCAS is
enabled, if lib_InternalSwiftScan.dylib is used to detect the cache hit,
we need to add MCCAS support to the replay code in SwiftCaching.cpp
Don't try to figure out the executable names during replay from
libSwiftScan.dylib. The actual executable path for the process actually
doesn't matter in this case to reconstruct the invocation and might
actually be misleading.
Just use `swift-frontend` as a placeheader executable name for
in-process cache replay.
rdar://132758308
Fix few issues from previous implementation from explicit module build
with macros and accurate macro dependency scanning in
https://github.com/swiftlang/swift/pull/73421.
First, there is a crash when propagating the macro dependencies. It
turns out that the current macro plugin implementation doesn't need the
downstream users to know about the plugin search path from the upstream
dependencies.
Secondly, fix a bug that the swiftinterface that has macro usage won't
build because the build command doesn't inherit the plugin search path
option.
Finally, add JSON output for macro dependencies so it is easier to
debug the macro dependencies.
rdar://131214106
Add support for serialized diagnostics, parseable output, and other
kinds of output from diagnostics engine to the libSwiftScan
replayCompilation API.
rdar://129015959
Separate swift-syntax libs for the compiler and for the library plugins.
Compiler communicates with library plugins using serialized messages
just like executable plugins.
* `lib/swift/host/compiler/lib_Compiler*.dylib`(`lib/CompilerSwiftSyntax`):
swift-syntax libraries for compiler. Library evolution is disabled.
* Compiler (`ASTGen` and `swiftIDEUtilsBridging`) only depends on
`lib/swift/host/compiler` libraries.
* `SwiftInProcPluginServer`: In-process plugin server shared library.
This has one `swift_inproc_plugins_handle_message` entry point that
receives a message and return the response.
* In the compiler
* Add `-in-process-plugin-server-path` front-end option, which specifies
the `SwiftInProcPluginServer` shared library path.
* Remove `LoadedLibraryPlugin`, because all library plugins are managed
by `SwiftInProcPluginServer`
* Introduce abstract `CompilerPlugin` class that has 2 subclasses:
* `LoadedExecutablePlugin` existing class that represents an
executable plugin
* `InProcessPlugins` wraps `dlopen`ed `SwiftInProcPluginServer`
* Unified the code path in `TypeCheckMacros.cpp` and `ASTGen`, the
difference between executable plugins and library plugins are now
abstracted by `CompilerPlugin`
Teach scanner to respect the working directory set in the invocation
through scanner C API.
Also add test infrastructure to testing scanner from C API. Break up
DependencyScan lib into two so the swift-scan-test and remain small
without understanding swift AST.
rdar://127626011
This change modifies the dependency scanner to keep track of source locations of each encountered 'import' statement, in order to be able to emit diagnostics with source locations if an import failed to resolve.
- Keep track of each 'import' statement's source buffer, line number, and column number when adding it. The dependency scanner utilizes separate compilation instances, and therefore separate Source Managers for scanning `import` statements of user sources and textual interfaces of Swift dependencies. Since import resolution may happen in the main scanner compilation instance while the `import` itself was found by an interface-scanning sub-instance, we cannot simply hold on to the import's `SourceLoc`.
- Add libSwiftScan API for diagnostics to carry above source locations to clients.
Add a fast path to create swift CompilerInstance when it is only used to
replay output when there is a cache hit. The normal `setup` function is
very expensive to call, especially in cache mode to setup inputs, and it
needs to be called once per input file from libSwiftScan API due to the
current caching granularity.
The fast path will only construct the part that is needed for output
replay, including the CAS, the output backend and caching diagnostic
processor.
rdar://127062609
Otherwise they may have module dependencies of their own which will not be detected by the scanner and included in the list of explicit inputs for compilation.
The response file expansion function doesn't return all the arguments
allocated using StringSaver produced. It only use the StringSaver to
save the newly expanded arguments. Need to make sure all the original
arguments are saved so they can be used later during replay.
rdar://121808135
Build system can pass response file to libSwiftScan as the command-line
arguments. libSwiftScan needs to expand the response file before
performing tasks.
rdar://121291342
`llvm::cl::ParseCommandLineOptions()` is not thread-safe to be called
from libSwiftScan so replaying multiple commands in parallel can crash
the build system. Since the replay instance is not used for compilation
and only need information from command-line arguments to create outputs
and diagnostics, ignore LLVMArgs when replaying.
rdar://120423882
Avoid path encoding difference (for example, real_path vs. path from
symlink) by eliminating the path from cache key. Cache key is now
encoded with the index of the input file from all the input files from
the command-line, reguardless if those inputs will produce output or
not. This is to ensure stable ordering even the batching is different.
Add a new cache computation API that is preferred for using input index
directly. Old API for cache key is deprecated but still updated to
fallback to real_path comparsion if needed.
As a result of swift scan API change, rename the feature in JSON file to
avoid version confusion between swift-driver and libSwiftScan.
rdar://119387650
Add a new async download API from CASID. This helps build system better
schedule the job/remove duplicated downloads, than using an opaque
cached_output handle.
ASTGen always builds with the host Swift compiler, without requiring
bootstrapping, and is enabled in more places. Move the regex literal
parsing logic there so it is enabled in more host environments, and
makes use of CMake's Swift support. Enable all of the regex literal
tests when ASTGen is built, to ensure everything is working.
Remove the "AST" and "Parse" Swift modules from SwiftCompilerSources,
because they are no longer needed.
Add new APIs libSwiftScan that can be used for cache query and cache
replay. This enables swift-driver or build system to query the cache and
replay the compilation results without invocation swift-frontend for
better scheduling.
Conflicts:
- `lib/AST/TypeCheckRequests.cpp` renamed `isMoveOnly` which requires
a static_cast on rebranch because `Optional` is now a `std::optional`.
The code, previously, only properly handled such dependencies being a distinct category for Swift source and Swift textual dependency infos. Swift binary module dependencies must handle this similarly and this change adds the missing support for them. Recent refactor of the scanner also means that now Swift binary dependencies with Swift overlay dependencies may crash the scanner, and this change resolves this as well.
Resolves rdar://117088840
Update swift cache key computation mechanism from one cache key per
output, to one cache key per primary input file (for all outputs that
associated with that input).
The new schema allows fewer cache lookups while still preserving most of
the flexibility for batch mode and incremental mode.
Conflicts:
- `CMakeLists.txt` caused by the extra `-D` added in rebranch to
reduce the number of deprecation warnings.
- `lib/Frontend/PrintingDiagnosticConsumer.cpp` caused by the removal
of one of the `#if SWIFT_SWIFT_PARSER` on rebranch (probably should
have been done on main).
Use FetchContent to include swift-syntax directly in swift. This can be
thought of as an `add_subdirectory` for a directory outside the root.
The default build directory will be `_deps/swiftsyntax-subbuild/`, though
the modules and shared libraries will be built in `lib/swift/host` by
passing down `SWIFT_HOST_LIBRARIES_DEST_DIR` to avoid copying them as we
were doing previously.
libSwiftScan is built in 'lib' but installed in 'lib/swift/host' RUNPATH
should have correct '$ORIGIN/../{platform}' to load 'swiftCore' runtime
library.
For compiling codes required for macro support, we now need swiftc
compiler in the build machine.
Unlike Darwin OSes, where swiftCore runtime is guaranteed to be present
in /usr/lib, Linux doesn't have ABI stability and the stdlib of the
build machine is not at the specific location. So the built compiler
cannot relies on the shared object in the toolchain.
LLVM install_symlink takes a new argument on whether to create a symlink
or copy binaries when run. -- https://reviews.llvm.org/D145443
The variable that controls this in LLVM is `LLVM_USE_SYMLINKS`, which
defaults to `ON` on Unix-y hosts, but otherwise is false so that Windows
works. This is a configurable option, so Windows configs that can
support symlinks can take advantage of symlinks and save some space.
`LLVM_USE_SYMLINKS` is not exported from LLVM though, so we can't see
it to use. Instead, we have `SWIFT_USE_SYMLINKS`.
As of CMake 3.25, there are now global variables `LINUX=1`, `ANDROID=1`,
etc. These conflict with expressions that used these names as unquoted
strings in positions where CMake accepts 'variable|string', for example:
- `if(sdk STREQUAL LINUX)` would fail, because `LINUX` is now defined and
expands to 1, where it would previously coerce to a string.
- `if(${sdk} STREQUAL "LINUX")` would fail if `sdk=LINUX`, because the
left-hand side expands twice.
In this patch, I looked for a number of patterns to fix up, sometimes a
little defensively:
- Quoted right-hand side of `STREQUAL` where I was confident it was
intended to be a string literal.
- Removed manual variable expansion on left-hand side of `STREQUAL`,
`MATCHES` and `IN_LIST` where I was confident it was unintended.
Fixes#65028.
