It is a maintenance burden and having the legacy driver exist in a simplified state reduces the possibility of things going wrong and hitting old bugs.
Although I don't plan to bring over new assertions wholesale
into the current qualification branch, it's entirely possible
that various minor changes in main will use the new assertions;
having this basic support in the release branch will simplify that.
(This is why I'm adding the includes as a separate pass from
rewriting the individual assertions)
Reformatting everything now that we have `llvm` namespaces. I've
separated this from the main commit to help manage merge-conflicts and
for making it a bit easier to read the mega-patch.
This is phase-1 of switching from llvm::Optional to std::optional in the
next rebranch. llvm::Optional was removed from upstream LLVM, so we need
to migrate off rather soon. On Darwin, std::optional, and llvm::Optional
have the same layout, so we don't need to be as concerned about ABI
beyond the name mangling. `llvm::Optional` is only returned from one
function in
```
getStandardTypeSubst(StringRef TypeName,
bool allowConcurrencyManglings);
```
It's the return value, so it should not impact the mangling of the
function, and the layout is the same as `std::optional`, so it should be
mostly okay. This function doesn't appear to have users, and the ABI was
already broken 2 years ago for concurrency and no one seemed to notice
so this should be "okay".
I'm doing the migration incrementally so that folks working on main can
cherry-pick back to the release/5.9 branch. Once 5.9 is done and locked
away, then we can go through and finish the replacement. Since `None`
and `Optional` show up in contexts where they are not `llvm::None` and
`llvm::Optional`, I'm preparing the work now by going through and
removing the namespace unwrapping and making the `llvm` namespace
explicit. This should make it fairly mechanical to go through and
replace llvm::Optional with std::optional, and llvm::None with
std::nullopt. It's also a change that can be brought onto the
release/5.9 with minimal impact. This should be an NFC change.
The swift driver emits extra newlines because it applies the LF ->
CRLF conversions again on the outpt from the child processes. Fix it
by using the binary mode when outputting the output from the child
processes.
Fixes: #64413
`getValue` -> `value`
`getValueOr` -> `value_or`
`hasValue` -> `has_value`
`map` -> `transform`
The old API will be deprecated in the rebranch.
To avoid merge conflicts, use the new API already in the main branch.
rdar://102362022
This cleans up 90 instances of this warning and reduces the build spew
when building on Linux. This helps identify actual issues when
building which can get lost in the stream of warning messages. It also
helps restore the ability to build the compiler with gcc.
The frontend supports this via new options -index-unit-output-path and
-index-unit-output-path-filelist that mirror -o and -output-filelist. These are
intended to allow sharing index data across builds in separate directories (so
different -o values) that are otherwise equivalent as far as the index data is
concerned (e.g. an ASAN build and a non-ASAN build) by supplying the same
-index-unit-output-path for both.
This change updates the driver to add these new options to the frontend
invocation 1) when a new "index-unit-output-path" entry is specified for one
or more input files in the -output-file-map json or 2) if -index-file is
specified, when a new -index-unit-output-path driver option is passed.
Resolves rdar://problem/74816412
Remove this distinction without a difference. Originally, the thought
was to
1) Isolate the cross-module build infrastructure
2) Provide a signal to the driver that a dependency had swiftdeps info
in it
But the driver need only notice swiftmodule files as external
dependencies and try to extract that information if it can to divine the
signal it needs. Additionally, we can give it fingerprints as priors to
let it know there might be incremental info to be had.
Introducing new entry-points that can be used from both Driver and Frontend clients, using an intermediary new type: `DetailedMessagePayload`, when needed.
VS2017 seems to have problems with ternaries where the two branches are
given as std::move() (see https://stackoverflow.com/questions/53374182)
and the compiler was given a warning C4172 "returning address of local
or temporary".
Switching the ternary operator into a `if/else` works correctly for
VS2017 and should work for other compilers too, without changing the
meaning (as far as I see).
This was causing the Windows VS2017 CI to fail compiling the stdlib.
When bisecting, the problem was reduced to the changes in #35034. In a
debugger I managed to see the problem being a heap corruption. With all
the moves I was suspicious that something was being used while
destroyed, but the code was correct. While recompiling the compiler
warning gave me the clue.
Starting at a crude -1000, each invocation primary input will get its own unique quasi-Pid.
Invocations with only one primary (non-batch) will get a real OS Pid.
The selection of the constant starting point matches what the driver does when outputting its parseable output.
We're going to play a dirty, dirty trick - but it'll make our users'
lives better in the end so stick with me here.
In order to build up an incremental compilation, we need two sources of
dependency information:
1) "Priors" - Swiftdeps with dependency information from the past
build(s)
2) "Posteriors" - Swiftdeps with dependencies from after we rebuild the
file or module or whatever
With normal swift files built in incremental mode, the priors are given by the
swiftdeps files which are generated parallel to a swift file and usually
placed in the build directory alongside the object files. Because we
have entries in the output file map, we can always know where these
swiftdeps files are. The priors are integrated by the driver and then
the build is scheduled. As the build runs and jobs complete, their
swiftdeps are reloaded and re-integrated. The resulting changes are then
traversed and more jobs are scheduled if necessary. These give us the
posteriors we desire.
A module flips this on its head. The swiftdeps information serialized
in a module functions as the *posterior* since the driver consuming the
module has no way of knowing how to rebuild the module, and because its
dependencies are, for all intents and purposes, fixed in time. The
missing piece of the puzzle is the priors. That is, we need some way of
knowing what the "past" interface of the module looked like so we can
compare it to the "present" interface. Moreover, we need to always know
where to look for these priors.
We solve this problem by serializing a file alongside the build record:
the "external" build record. This is given by a... creative encoding
of multiple source file dependency graphs into a single source file
dependency graph. The rough structure of this is:
SourceFile => interface <BUILD_RECORD>.external
| - Incremental External Dependency => interface <MODULE_1>.swiftmodule
| | - <dependency> ...
| | - <dependency> ...
| | - <dependency> ...
| - Incremental External Dependency => interface <MODULE_2>.swiftmodule
| | - <dependency> ...
| | - <dependency> ...
| - Incremental External Dependency => interface <MODULE_3>.swiftmodule
| - ...
Sorta, `cat`'ing a bunch of source file dependency graphs together but
with incremental external dependency nodes acting as glue.
Now for the trick:
We have to unpack this structure and integrate it to get our priors.
This is easy. The tricky bit comes in integrate itself. Because the
top-level source file node points directly at the external build record,
not the original swift modules that defined these dependency nodes, we
swap the key it wants to use (the external build record) for the
incremental external dependency acting as the "parent" of the dependency
node. We do this by following the arc we carefully laid down in the
structure above.
For rdar://69595010
Goes a long way towards rdar://48955139, rdar://64238133
In order to extract the module dependency graph from the compilation the driver just ran, define a separate semantic type to hold a result code and the graph itself.
The "wave" of a compilation job describes the number of indirections through other compile jobs the driver required to reach the decision to schedule a job. In incremental mode, it should always be the case that it takes no more than two complete waves to arrive at a fixpoint in the build. This is a natural consequence of the structure of the dependencies emitted by the Swift frontend - namely we rely on transitivity in dependency arcs.
A quick proof sketch: Suppose an arbitrary perturbation of the inputs to an incremental compilation session are made. In the first wave, dependency edges from the prior build's state (the "zeroeth wave") are loaded and the files corresponding to invalidated edges are scheduled into the first wave. Supposing the second wave is not the null set - the trivial case - there are additional arcs that were invalidated. Now suppose that there were a third wave. Take an arbitrary arc invalidated by this third wave. It must be the case that the file containing the use is not new - else it would be scheduled. Further it must be the case that its def was not invalidated by the zeroeth or first waves of compilation otherwise we would have scheduled it into the first or second waves. Finally, it must have a use that was discovered in the second wave. But in order for that use to have been included in the second wave, there must have been an invalidated arc created by the first wave. By transitivity of dependency arcs, there must therefore be a dependency arc from a definition invalidated in the first wave to our third wave job, which implies that the file would be scheduled into the second wave!
[Insert contradiction pig image here]
In order to unblock the SwiftWASM project, which relies on an
incremental build of the Swift driver that relies on the merge-modules
job always being run. The situation appears to be something like this:
1) An incremental build is run
2) Temporary swiftmodule outputs are laid down
3) merge-modules is skipped
4) modulewrap is run anyways and reads the empty temp file
We should fix this by skipping modulewrap if we can skip merge-modules.
But for now, be conservative and fall back to the status quo behavior of
always running merge-modules whenever we encounter a modulewrap job.
Plumb the logic necessary to schedule merge-modules incrementally. This means that if any inputs jobs to merge-modules run, merge-modules is run. But, if they are all skipped, merge-modules will be skipped as well.
This requires some light special-casing of the legacy driver's incremental job handling because it assumes in a few places it can always extract a swiftdeps file. This invariant will be further broken when the precompile step for bridging headers is skipped as well.
rdar://65893400
A more durable form of #34218. Keep a side cache of externally-dependent
jobs for now. This ensures our pseudo-Jobs don't get prematurely
deallocated before the tracing machinery has had a chance to report the
structure of the Job graph.
rdar://70053563
An incremental build involving incremental external dependencies behaves as a hybrid between an external dependency and a normal swiftdeps-laden Swift file.
In the simplest case, we will fall back to the behavior of a plain external dependency today. That is, we will check its timestamp, then schedule all jobs that involve these external dependencies if it is out of date.
Where things get interesting is when cross-module incremental builds are enabled. In such a case, we know that a previous compiler has already emitted serialized swiftdeps information inside of a swiftmodule file. Moreover, we know that that swiftmodule file was loaded by the build of the current swift module. Finally, thanks to the previous stack of commits, we now know exactly how to extract this information from the swiftmodule file. To bring this all home, we unpack incremental dependency information from external dependencies, then integrate them into the current dependency graph - as though they were any other swiftdeps file. This neatly extends the single-module incremental logic to the multi-module case.
Treat any incremental external depends like normal external depends. This will eventually become the fallback behavior for cross-module incremental builds.
