For clients, such as the debugger, who do not have access the full
output of the dependency scanner, it is a huger performance and
correctness improvement if each explicitly built Swift module not just
serialized all its Clang .pcm dependencies (via the serialized Clang
compiler invocation) but also its direct Swift module dependencies.
This patch changes the Swift module format to store the absolute path
or cas cache key for each dependency in the INPUT block, and makes
sure the deserialization makes these available to the ESML.
rdar://150969755
Previously this flag was only used to pass explicit dependencies to compilation tasks. This change adds support for the dependency scanner to also consider these inputs when resolving dependencies.
Resolves https://github.com/swiftlang/swift-driver/issues/1951
-nostdimport and -nostdlibimport only remove the toolchain and usr/lib/swift search paths, and they leave the framework search paths intact. That makes it impossible to get a fully custom SDK environment. Make their behavior match clang's -nostdinc/-nostdlibinc behavior: treat framework and non-framework paths the same. In other words, -nostdinc removes *all* compiler provided search paths, and -nostdlibinc removes *all* SDK search paths.
Rename SkipRuntimeLibraryImportPaths to SkipAllImportPaths, and ExcludeSDKPathsFromRuntimeLibraryImportPaths to SkipSDKImportPaths to reflect their updated behavior.
Move the DarwinImplicitFrameworkSearchPaths handling from SearchPathOptions to CompilerInvocation, where RuntimeLibraryImportPaths is managed. Rename it to just ImplicitFrameworkSearchPaths, and filter for Darwin when it's set up so that all of the clients don't have to do Darwin filtering themselves later.
rdar://150557632
This hash is also used for the dependency scanning hash. In both cases, PCH contents may differ based on whether a certain module they depend on is found in a system or non-system search path. In dependency scanning, systemness should cause a full change of scanning context requiring a from-scratch scan.
Resolves rdar://150334077
Currently, the macro plugin options are included as cache key and the
absolute path of the plugin executable and library will affect cache
hit, even the plugin itself is identical.
Using the new option `-resolved-plugin-validation` flag, the macro
plugin paths are remapped just like the other paths during dependency
scanning. `swift-frontend` will unmap to its original path during the
compilation, make sure the content hasn't changed, and load the plugin.
It also hands few other corner cases for macro plugins:
* Make sure the plugin options in the swift module is prefix mapped.
* Make sure the remarks of the macro loading is not cached, as the
mesasge includes the absolute path of the plugin, and is not
cacheable.
rdar://148465899
Add ability to automatically chaining the bridging headers discovered from all
dependencies module when doing swift caching build. This will eliminate all
implicit bridging header imports from the build and make the bridging header
importing behavior much more reliable, while keep the compatibility at maximum.
For example, if the current module A depends on module B and C, and both B and
C are binary modules that uses bridging header, when building module A,
dependency scanner will construct a new header that chains three bridging
headers together with the option to build a PCH from it. This will make all
importing errors more obvious while improving the performance.
Batch dependency scanning was added as a mechanism to support multiple compilation contexts within a single module dependency graph.
The Swift compiler and the Explicitly-built modules model has long since abandoned this approach and this code has long been stale. It is time to remove it and its associated C API.
Add a -nostdlibimport (analagous to clang's -nostdlibinc) to remove the SDK paths from the import search paths, but leave the toolchain paths.
rdar://139322299
When Swift passes search paths to clang, it does so directly into the HeaderSearch. That means that those paths get ordered inconsistently compared to the equivalent clang flag, and causes inconsistencies when building clang modules with clang and with Swift. Instead of touching the HeaderSearch directly, pass Swift search paths as driver flags, just do them after the -Xcc ones.
Swift doesn't have a way to pass a search path to clang as -isystem, only as -I which usually isn't the right flag. Add an -Isystem Swift flag so that those paths can be passed to clang as -isystem.
rdar://93951328
This change ensures that when loading some module dependency 'Bar' which has a package-only dependency on 'Foo', only the following clients attempt to resolve/load 'Foo':
- Source compilation with package-name equal to that of 'Bar'.
- Textual interface compilation of a *'package'* interface with package-name equal to that of 'Bar'.
Ensuring that the following kinds of clients do not attempt to resolve/load 'Foo':
- Source compilation with package-name different to that of 'Bar'
- Textual interface compilation of a public or private interface, regardless of package name.
This fixes the behavior where previously compilation of a Swift textual interface dependency 'X' from its public or private interface, with an interface-specified package-name, from a client without a matching package-name, resulted in a lookup of package-only dependencies of modules loaded into 'X'. This behavior is invalid if we are not building from the package textual interface, becuase the module dependency graph is defined by the package name of the source client, not individual module dependency package name. i.e. In-package module dependencies are resolved/loaded only if the parent source compile matches the package name.
Resolves rdar://139979180
Add flag `-load-resolved-plugin` to load macro plugin, which provides a
pre-resolved entry into PluginLoader so the plugins can be loaded based
on module name without searching the file system. The option is mainly
intended to be used by explicitly module build and the flag is supplied
by dependency scanner.
Modules defined within the SDK are considered
non-user modules, extend this to any module found
within the parent platform directory if there is
one. This ensures we include modules such as
XCTest and Testing.
rdar://131854240
As-is, this default interferes with the incremental build machinery which conservatively assumes that binary module dependencies must cause dependents to be re-built.
This introduces a secondary flag `-sysroot` for the non-Darwin targets,
primarily Unicies. The intention here is to support a split `-sdk`,
`-sysroot` model where the `-sdk` parameter provides the Swift "SDK"
which augments the native platform's C sysroot which is indicated as
`-sysroot`. For the case of Android, this would allow us to provide a
path to the NDK sysroot and the Swift SDK allowing us to cross-compile
Android binaries from Windows.
Separate swift-syntax libs for the compiler and for the library plugins.
Compiler communicates with library plugins using serialized messages
just like executable plugins.
* `lib/swift/host/compiler/lib_Compiler*.dylib`(`lib/CompilerSwiftSyntax`):
swift-syntax libraries for compiler. Library evolution is disabled.
* Compiler (`ASTGen` and `swiftIDEUtilsBridging`) only depends on
`lib/swift/host/compiler` libraries.
* `SwiftInProcPluginServer`: In-process plugin server shared library.
This has one `swift_inproc_plugins_handle_message` entry point that
receives a message and return the response.
* In the compiler
* Add `-in-process-plugin-server-path` front-end option, which specifies
the `SwiftInProcPluginServer` shared library path.
* Remove `LoadedLibraryPlugin`, because all library plugins are managed
by `SwiftInProcPluginServer`
* Introduce abstract `CompilerPlugin` class that has 2 subclasses:
* `LoadedExecutablePlugin` existing class that represents an
executable plugin
* `InProcessPlugins` wraps `dlopen`ed `SwiftInProcPluginServer`
* Unified the code path in `TypeCheckMacros.cpp` and `ASTGen`, the
difference between executable plugins and library plugins are now
abstracted by `CompilerPlugin`
Teach dependency scanner to report all the module canImport check result
to swift-frontend, so swift-frontend doesn't need to parse swiftmodule
or parse TBD file to determine the versions. This ensures dependency
scanner and swift-frontend will have the same resolution for all
canImport checks.
This also fixes two related issues:
* Previously, in order to get consistant results between scanner and
frontend, scanner will request building the module in canImport check
even it is not imported later. This slightly alters the definition of
the canImport to only succeed when the module can be found AND be
built. This also can affect the auto-link in such cases.
* For caching build, the location of the clang module is abstracted away
so swift-frontend cannot locate the TBD file to resolve
underlyingVersion.
rdar://128067152
Teach dependency scanner to pass cross import overlay file to
swift-frontend for main module compilation. This allows swift-frontend
not to repeat the file system search for overlay files when loading
modules.
This also fixes the issue when caching is enabled, the cross import
doesn't work when the first module is a clang module because the module
built with caching using clang include tree does not preserve
DefinitionLoc which is used to inferred the modulemap location for cross
import overlay search.
rdar://127844120
This change introduces a new compilation target platform to the Swift compiler - visionOS.
- Changes to the compiler build infrastrucuture to support building compiler-adjacent artifacts and test suites for the new target.
- Addition of the new platform kind definition.
- Support for the new platform in language constructs such as compile-time availability annotations or runtime OS version queries.
- Utilities to read out Darwin platform SDK info containing platform mapping data.
- Utilities to support re-mapping availability annotations from iOS to visionOS (e.g. 'updateIntroducedPlatformForFallback', 'updateDeprecatedPlatformForFallback', 'updateObsoletedPlatformForFallback').
- Additional tests exercising platform-specific availability handling and availability re-mapping fallback code-path.
- Changes to existing test suite to accomodate the new platform.
Improve swift dependency scanner by validating and selecting dependency
module into scanner. This provides benefits that:
* Build system does not need to schedule interface compilation task if
the candidate module is picked, it can just use the candidate module
directly.
* There is no need for forwarding module in the explicit module build.
Since the build system is coordinating the build, there is no need for
the forwarding module in the module cache to avoid duplicated work,
* This also correctly supports all the module loading modes in the
dependency scanner.
This is achieved by only adding validate and up-to-date binary module as
the candidate module for swift interface module dependency. This allows
caching build to construct the correct dependency in the CAS. If there
is a candidate module for the interface module, dependency scanner will
return a binary module dependency in the dependency graph.
The legacy behavior is mostly preserved with a hidden frontend flag
`-no-scanner-module-validation`, while the scanner output is mostly
interchangeable with new scanner behavior with `prefer-interface` module
loading mode except the candidate module will not be returned.
rdar://123711823
Relying on the corresponding field in the '-explicit-swift-module-map-file' provided by the driver.
Only bridging headers require a module map because that's what aids header include resolution. With lazy module loading today, '.modulemap' parsing which happens when instantiating Clang is responsible for associating headers with modules. Then upon encountering a header include inside the bridging header the compiler knows which module corresponds to said header and is then able to load explicitly-provided PCM for that module. For all other module dependencies, they are only ever queried by-name from Swift, so '.modulemap' parsing is not necessary.
Conflict in CAS options when
`std::vector<std::string> CacheReplayPrefixMap;` was added.
Conflicts:
include/swift/Frontend/FrontendOptions.h
Resolution: Take both
Allow DependencyScanner to canonicalize path using a prefix map. When
option `-scanner-prefix-map` option is used, dependency scanner will
remap all the input paths in following:
* all the paths in the CAS file system or clang include tree
* all the paths related to input on the command-line returned by scanner
This allows all the input paths to be canonicalized so cache key can be
computed reguardless of the exact on disk path.
The sourceFile field is not remapped so build system can track the exact
file as on the local file system.
'load-plugin-library', 'load-plugin-executable', '-plugin-path' and
'-external-plugin-path' should be searched in the order they are
specified in the arguments.
Previously, for example '-plugin-path' used to precede
'-external-plugin-path' regardless of the position in the arguments.
* Argument to '-load-plugin-library' now must have a filename that's
'{libprefix}{modulename}.{sharedlibraryextension}'
* Load '-load-plugin-library' plugins are now lazily loaded in
'CompilerPluginLoadRequest'
* Remove ASTContext.LoadedSymbols cache because they are cached by
'ExternalMacroDefinitionRequest' anyway
* `-load-plugin-executable` format validation is now in
'ParseSearchPathArgs'
This executable is intended to be installed in the toolchain and act as
an executable compiler plugin just like other 'macro' plugins.
This plugin server has an optional method 'loadPluginLibrary' that
dynamically loads dylib plugins.
The compiler has a newly added option '-external-plugin-path'. This
option receives a pair of the plugin library search path (just like
'-plugin-path') and the corresponding "plugin server" path, separated
by '#'. i.e.
-external-plugin-path
<plugin library search path>#<plugin server executable path>
For exmaple, when there's a macro decl:
@freestanding(expression)
macro stringify<T>(T) -> (T, String) =
#externalMacro(module: "BasicMacro", type: "StringifyMacro")
The compiler look for 'libBasicMacro.dylib' in '-plugin-path' paths,
if not found, it falls back to '-external-plugin-path' and tries to find
'libBasicMacro.dylib' in them. If it's found, the "plugin server" path
is launched just like an executable plugin, then 'loadPluginLibrary'
method is invoked via IPC, which 'dlopen' the library path in the plugin
server. At the actual macro expansion, the mangled name for
'BasicMacro.StringifyMacro' is used to resolve the macro just like
dylib plugins in the compiler.
This is useful for
* Isolating the plugin process, so the plugin crashes doesn't result
the compiler crash
* Being able to use library plugins linked with other `swift-syntax`
versions
rdar://105104850
This adds the following four new options:
- `-windows-sdk-root`
- `-windows-sdk-version`
- `-visualc-tools-root`
- `-visualc-tools-version`
Together these options make one the master of Windows SDK selection for
the Swift compilation. This is important as now that the injection is
no longer done by the user, we need to ensure that we have enough
control over the paths so that the synthesized overlay is going to map
the files to the proper location.
Add a compiler option `-load-plugin-executable <path>#<module names>`.
Where '<path>' is a path to a plugin executable, '<module-name>' is a
comma-separated module names the plugin provides.
Nothing is using it at this point. Actual plugin infratructure are
introduced in follow-up commits
Introduce `-plugin-path <path>` to add a search path where we will look
for compiler plugins. When resolving an external macro definition, look
for libraries in these search paths whose names match the module name
of the macro.
Implements rdar://105095761.
Introduces a concept of a dependency scanning action context hash, which is used to select an instance of a global dependency scanning cache which gets re-used across dependency scanning actions.
Allow user-defined macros to be loaded from dynamic libraries and evaluated.
- Introduce a _CompilerPluginSupport module installed into the toolchain. Its `_CompilerPlugin` protocol acts as a stable interface between the compiler and user-defined macros.
- Introduce a `-load-plugin-library <path>` attribute which allows users to specify dynamic libraries to be loaded into the compiler.
A macro library must declare a public top-level computed property `public var allMacros: [Any.Type]` and be compiled to a dynamic library. The compiler will call the getter of this property to obtain and register all macros.
Known issues:
- We current do not have a way to strip out unnecessary symbols from the plugin dylib, i.e. produce a plugin library that does not contain SwiftSyntax symbols that will collide with the compiler itself.
- `MacroExpansionExpr`'s type is hard-coded as `(Int, String)`. It should instead be specified by the macro via protocol requirements such as `signature` and `genericSignature`. We need more protocol requirements in `_CompilerPlugin` to handle this.
- `dlopen` is not secure and is only for prototyping use here.
Friend PR: apple/swift-syntax#1022
Two paths missed setting up overlays:
- `CompletionInstance` when checking files from dependencies
- `SwiftASTManager` when reading in files that it would later replace
all inputs with
(1) would cause the AST context not to be re-used, even though nothing
had changed. (2) caused all non-completion functionality to fail for any
symbols within files only specified by the overlay.
Resolves rdar://85508213.
The idea behind storing a StringRef was to reduce the memory footprint because we made the assumption that a `ModuleSearchPath` always outlives the `SearchPathOptions` it was created from. What I did not consider was that the path was referencing into a `std::vector`, which could get resized, thus invaliding the memory the `ModuleSearchPath`’s `StringRef` was pointing to, causing memory corruption.
To fix this, store the path string inisde the `ModuleSearchPath` itself. Since we store a `ModuleSearchPath` for every file inside that module search path in the `LookupTable`, by itself this would cause a new copy of the path to be stored for every file inside a module search path. To avoid this, make `ModuleSearchPath` ref counted and only store a reference to one shared `ModuleSearchPath` entry in the lookup table.
rdar://88888679
When looking for a Swift module on disk, we were scanning all module search paths if they contain the module we are searching for. In a setup where each module is contained in its own framework search path, this scaled quadratically with the number of modules being imported. E.g. a setup with 100 modules being imported form 100 module search paths could cause on the order of 10,000 checks of `FileSystem::exists`. While these checks are fairly fast (~10µs), they add up to ~100ms.
To improve this, perform a first scan of all module search paths and list the files they contain. From this, create a lookup map that maps filenames to the search paths they can be found in. E.g. for
```
searchPath1/
Module1.framework
searchPath2/
Module1.framework
Module2.swiftmodule
```
we create the following lookup table
```
Module1.framework -> [searchPath1, searchPath2]
Module2.swiftmodule -> [searchPath2]
```
