We're going to play a dirty, dirty trick - but it'll make our users'
lives better in the end so stick with me here.
In order to build up an incremental compilation, we need two sources of
dependency information:
1) "Priors" - Swiftdeps with dependency information from the past
build(s)
2) "Posteriors" - Swiftdeps with dependencies from after we rebuild the
file or module or whatever
With normal swift files built in incremental mode, the priors are given by the
swiftdeps files which are generated parallel to a swift file and usually
placed in the build directory alongside the object files. Because we
have entries in the output file map, we can always know where these
swiftdeps files are. The priors are integrated by the driver and then
the build is scheduled. As the build runs and jobs complete, their
swiftdeps are reloaded and re-integrated. The resulting changes are then
traversed and more jobs are scheduled if necessary. These give us the
posteriors we desire.
A module flips this on its head. The swiftdeps information serialized
in a module functions as the *posterior* since the driver consuming the
module has no way of knowing how to rebuild the module, and because its
dependencies are, for all intents and purposes, fixed in time. The
missing piece of the puzzle is the priors. That is, we need some way of
knowing what the "past" interface of the module looked like so we can
compare it to the "present" interface. Moreover, we need to always know
where to look for these priors.
We solve this problem by serializing a file alongside the build record:
the "external" build record. This is given by a... creative encoding
of multiple source file dependency graphs into a single source file
dependency graph. The rough structure of this is:
SourceFile => interface <BUILD_RECORD>.external
| - Incremental External Dependency => interface <MODULE_1>.swiftmodule
| | - <dependency> ...
| | - <dependency> ...
| | - <dependency> ...
| - Incremental External Dependency => interface <MODULE_2>.swiftmodule
| | - <dependency> ...
| | - <dependency> ...
| - Incremental External Dependency => interface <MODULE_3>.swiftmodule
| - ...
Sorta, `cat`'ing a bunch of source file dependency graphs together but
with incremental external dependency nodes acting as glue.
Now for the trick:
We have to unpack this structure and integrate it to get our priors.
This is easy. The tricky bit comes in integrate itself. Because the
top-level source file node points directly at the external build record,
not the original swift modules that defined these dependency nodes, we
swap the key it wants to use (the external build record) for the
incremental external dependency acting as the "parent" of the dependency
node. We do this by following the arc we carefully laid down in the
structure above.
For rdar://69595010
Goes a long way towards rdar://48955139, rdar://64238133
Plumb the logic necessary to schedule merge-modules incrementally. This means that if any inputs jobs to merge-modules run, merge-modules is run. But, if they are all skipped, merge-modules will be skipped as well.
This requires some light special-casing of the legacy driver's incremental job handling because it assumes in a few places it can always extract a swiftdeps file. This invariant will be further broken when the precompile step for bridging headers is skipped as well.
rdar://65893400
With this option enabled, the dependency scanner gathers all import statements in source files of the main module (non-transitive) and outputs a list of imported modules.
This will be used by build systems and the swift-driver as a way to avoid redundant re-scanning in incremental contexts.
The driver can now schedule jobs which typecheck just-emitted module interfaces to ensure that they can be consumed later. This can be enabled manually by passing `-verify-emitted-module-interface` to the driver.
This commit adds LTO support for handling linker options and LLVM BC
emission. Even for ELF, swift-autolink-extract is unnecessary because
linker options are embeded in LLVM BC content when LTO.
- deduplicate the logic to compute the resource folder
- install headers and module files in shared and static resource folders
- forward -static flag when calling swiftc with -print-target-info
Previously the path to covered files in the __LLVM_COV / __llvm_covmap
section were absolute. This made remote builds with coverage information
difficult because all machines would have to have the same build root.
This change uses the values for `-coverage-prefix-map` to remap files in
the coverage info to relative paths. These paths work correctly with
llvm-cov when it is run from the same source directory as the
compilation, or from a different directory using the `-path-equivalence`
argument.
This is analogous to this change in clang https://reviews.llvm.org/D81122
This commit adds -lto flag for driver to enable LTO at LLVM level.
When -lto=llvm given, compiler emits LLVM bitcode file instead of object
file and perform thin LTO using libLTO.dylib plugin.
When -lto=llvm-full given, perform full LTO instead of thin LTO.
Implement a new "fast" dependency scanning option,
`-scan-dependencies`, in the Swift frontend that determines all
of the source file and module dependencies for a given set of
Swift sources. It covers four forms of modules:
1) Swift (serialized) module files, by reading the module header
2) Swift interface files, by parsing the source code to find imports
3) Swift source modules, by parsing the source code to find imports
4) Clang modules, using Clang's fast dependency scanning tool
A single `-scan-dependencies` operation maps out the full
dependency graph for the given Swift source files, including all
of the Swift and Clang modules that may need to be built, such
that all of the work can be scheduled up front by the Swift
driver or any other build system that understands this
option. The dependency graph is emitted as JSON, which can be
consumed by these other tools.
Some code paths that see target triples go through the frontend
without seeing the driver. Therefore, perform the same "simulator"
inference for x86 iOS/tvOS/watchOS triples also in the frontend,
to ensure that we remain compatible. Also make sure that
-print-target-info performs the appropriate adjustment.
Localize the hack to infer simulator-ness of the target in the driver
itself, when it first processes the target. Emit a warning about the
missing "-simulator" and correct the triple immediately.
This gives a longer grace period for tools that might still not pass
through the simulator environment, while narrowing the hack.
`-no-warnings-as-errors`
This functionality is required for build systems to be able to overload/disable a given Swift project's preference of treating warnings as errors.
Resolves rdar://problem/35699776
This simplifies fixing the master-next build. Upstream LLVM already
has a copy of this function, so on master-next we only need to delete
the Swift copy, reducing the potential for merge conflicts.
Some code paths that see target triples go through the frontend
without seeing the driver. Therefore, perform the same "simulator"
inference for x86 iOS/tvOS/watchOS triples also in the frontend,
to ensure that we remain compatible. Also make sure that
-print-target-info performs the appropriate adjustment.
Localize the hack to infer simulator-ness of the target in the driver
itself, when it first processes the target. Emit a warning about the
missing "-simulator" and correct the triple immediately.
This gives a longer grace period for tools that might still not pass
through the simulator environment, while narrowing the hack.
Recent-ish SDKs for Darwin platforms include an SDKSettings.json
file with version information and Catalyst SDK version mappings. Read
these (when available) and use them to pass the appropriate SDK
version down to the Darwin linker via `-platform_version`.
Finishes rdar://problem/55972144.
Prior to performing linking on ELF platforms (except the PS4) we extract
autolink information from the object files that will be used during linking.
The current implementation passes all the linker inputs that are marked as an
TY_Object to swift-autolink-extract. There are a two cases where this is
not necessary or problematic.
In the first case, we are looking at an ELF shared object. Although
harmless, this is wasted work. Specifically, the `.swift1_autolink_entries`
entry in the object files are marked as `SHF_EXCLUDE`, meaning they will not be
merged into the final product during linking.
In the second case, we are linking against a linker script that looks like an
ELF shared object (ends with `.so`). In the previous case, the autolink-extract
step will succeed even if it does unnecessary work. In this case, the
autolink-extract step will fail because it cannot recognize the linker script as
an object file. You will observe an error something like this:
```
<unknown>:0: error: error opening input file '/path/to/libLinkerScript.so'
(The file was not recognized as a valid object file
```
Although your linker will know what to do with it, autolink-extract will halt
before you get to that point.
This allows the usage of the whole remark infrastructure developed in
LLVM, which includes a new binary format, metadata in object files, etc.
This gets rid of the YAMLTraits-based remark serialization and does the
plumbing for hooking to LLVM's main remark streamer.
For more about the idea behind LLVM's main remark streamer, see the
docs/Remarks.rst changes in https://reviews.llvm.org/D73676.
The flags are now:
* -save-optimization-record: enable remarks, defaults to YAML
* -save-optimization-record=<format>: enable remarks, use <format> for
serialization
* -save-optimization-record-passes <regex>: only serialize passes that
match <regex>.
The YAMLTraits in swift had a different `flow` setting for the debug
location, resulting in some test changes.
