We're going to play a dirty, dirty trick - but it'll make our users'
lives better in the end so stick with me here.
In order to build up an incremental compilation, we need two sources of
dependency information:
1) "Priors" - Swiftdeps with dependency information from the past
build(s)
2) "Posteriors" - Swiftdeps with dependencies from after we rebuild the
file or module or whatever
With normal swift files built in incremental mode, the priors are given by the
swiftdeps files which are generated parallel to a swift file and usually
placed in the build directory alongside the object files. Because we
have entries in the output file map, we can always know where these
swiftdeps files are. The priors are integrated by the driver and then
the build is scheduled. As the build runs and jobs complete, their
swiftdeps are reloaded and re-integrated. The resulting changes are then
traversed and more jobs are scheduled if necessary. These give us the
posteriors we desire.
A module flips this on its head. The swiftdeps information serialized
in a module functions as the *posterior* since the driver consuming the
module has no way of knowing how to rebuild the module, and because its
dependencies are, for all intents and purposes, fixed in time. The
missing piece of the puzzle is the priors. That is, we need some way of
knowing what the "past" interface of the module looked like so we can
compare it to the "present" interface. Moreover, we need to always know
where to look for these priors.
We solve this problem by serializing a file alongside the build record:
the "external" build record. This is given by a... creative encoding
of multiple source file dependency graphs into a single source file
dependency graph. The rough structure of this is:
SourceFile => interface <BUILD_RECORD>.external
| - Incremental External Dependency => interface <MODULE_1>.swiftmodule
| | - <dependency> ...
| | - <dependency> ...
| | - <dependency> ...
| - Incremental External Dependency => interface <MODULE_2>.swiftmodule
| | - <dependency> ...
| | - <dependency> ...
| - Incremental External Dependency => interface <MODULE_3>.swiftmodule
| - ...
Sorta, `cat`'ing a bunch of source file dependency graphs together but
with incremental external dependency nodes acting as glue.
Now for the trick:
We have to unpack this structure and integrate it to get our priors.
This is easy. The tricky bit comes in integrate itself. Because the
top-level source file node points directly at the external build record,
not the original swift modules that defined these dependency nodes, we
swap the key it wants to use (the external build record) for the
incremental external dependency acting as the "parent" of the dependency
node. We do this by following the arc we carefully laid down in the
structure above.
For rdar://69595010
Goes a long way towards rdar://48955139, rdar://64238133
In order to extract the module dependency graph from the compilation the driver just ran, define a separate semantic type to hold a result code and the graph itself.
The "wave" of a compilation job describes the number of indirections through other compile jobs the driver required to reach the decision to schedule a job. In incremental mode, it should always be the case that it takes no more than two complete waves to arrive at a fixpoint in the build. This is a natural consequence of the structure of the dependencies emitted by the Swift frontend - namely we rely on transitivity in dependency arcs.
A quick proof sketch: Suppose an arbitrary perturbation of the inputs to an incremental compilation session are made. In the first wave, dependency edges from the prior build's state (the "zeroeth wave") are loaded and the files corresponding to invalidated edges are scheduled into the first wave. Supposing the second wave is not the null set - the trivial case - there are additional arcs that were invalidated. Now suppose that there were a third wave. Take an arbitrary arc invalidated by this third wave. It must be the case that the file containing the use is not new - else it would be scheduled. Further it must be the case that its def was not invalidated by the zeroeth or first waves of compilation otherwise we would have scheduled it into the first or second waves. Finally, it must have a use that was discovered in the second wave. But in order for that use to have been included in the second wave, there must have been an invalidated arc created by the first wave. By transitivity of dependency arcs, there must therefore be a dependency arc from a definition invalidated in the first wave to our third wave job, which implies that the file would be scheduled into the second wave!
[Insert contradiction pig image here]
The Optional parameter here was being copied instead of being taken by
const reference. The expectation is that we call this function and
extract the data from a ModuleDepGraphNode node, but instead we were
extracting a copy which would promptly blow up.
Thanks ASAN!
A more durable form of #34218. Keep a side cache of externally-dependent
jobs for now. This ensures our pseudo-Jobs don't get prematurely
deallocated before the tracing machinery has had a chance to report the
structure of the Job graph.
rdar://70053563
Treat any incremental external depends like normal external depends. This will eventually become the fallback behavior for cross-module incremental builds.
LLVM, as of 77e0e9e17daf0865620abcd41f692ab0642367c4, now builds with
-Wsuggest-override. Let's clean up the swift sources rather than disable
the warning locally.
The driver can now schedule jobs which typecheck just-emitted module interfaces to ensure that they can be consumed later. This can be enabled manually by passing `-verify-emitted-module-interface` to the driver.
This commit adds LTO support for handling linker options and LLVM BC
emission. Even for ELF, swift-autolink-extract is unnecessary because
linker options are embeded in LLVM BC content when LTO.
This doesn't yet allow including C++ headers on platforms where libc++
isn't the default; see comments in UnixToolChains.cpp for details.
However, it does, for example, allow throwing and catching exceptions in C++
code used through interop, unblocking
https://github.com/apple/swift/pull/30674/files.
The flags (-enable-experimental-cxx-interop and -experimental-cxx-stdlib) carry
"experimental" in the name to emphasize that C++ interop is still an
experimental feature.
Co-authored-by: Michael Forster <forster@google.com>
This commit adds -lto flag for driver to enable LTO at LLVM level.
When -lto=llvm given, compiler emits LLVM bitcode file instead of object
file and perform thin LTO using libLTO.dylib plugin.
When -lto=llvm-full given, perform full LTO instead of thin LTO.
When producing frontend arguments for sourcekitd, force the output mode
to -typecheck so that we do not create any temporary output files in the
driver. Previously, any sourcekitd operation that created a compiler
invocation would create 0-sized .o file inside $TMPDIR that would never
be cleaned up.
The new swift-driver project handles temporaries much better as
VirtualPath, and should not need this approach.
rdar://62366123
Teach the driver to pass the SDK version it computes (from the SDK
settings JSON in a Darwin-based platform's SDK) down into the frontend.
The frontend then sets that SDK version in the LLVM module, which
eventually makes its way into the Mach-O file.
Last part of rdar://problem/60332732.
Recent-ish SDKs for Darwin platforms include an SDKSettings.json
file with version information and Catalyst SDK version mappings. Read
these (when available) and use them to pass the appropriate SDK
version down to the Darwin linker via `-platform_version`.
Finishes rdar://problem/55972144.
This allows the usage of the whole remark infrastructure developed in
LLVM, which includes a new binary format, metadata in object files, etc.
This gets rid of the YAMLTraits-based remark serialization and does the
plumbing for hooking to LLVM's main remark streamer.
For more about the idea behind LLVM's main remark streamer, see the
docs/Remarks.rst changes in https://reviews.llvm.org/D73676.
The flags are now:
* -save-optimization-record: enable remarks, defaults to YAML
* -save-optimization-record=<format>: enable remarks, use <format> for
serialization
* -save-optimization-record-passes <regex>: only serialize passes that
match <regex>.
The YAMLTraits in swift had a different `flow` setting for the debug
location, resulting in some test changes.
