Unlike the frontend, the driver doesn't account for any VFS overlays set up
by the -vfsoverlay option when processing its input files. It also errors
if any of those input files don't exist on the file system. This makes it
impossible to use a file that only exists in the VFS as input to the driver,
even though the same file would be handled without issue by the frontend.
If the file doesn't exist even accounting for the VFS the frontend emits
a missing file diagnostic, so this change just suppresses the existence
check for inputs when a -vfsoverlay option is present.
Resolves rdar://problem/72485443
We've seen bad file descriptor errors in certain build environments (rdar://73157185) and retrying
opening those files seems to be a walkaround. In this change, we allow the
compiler to retry expanding reponse files in argument lists.
rdar://73892564
Remove this distinction without a difference. Originally, the thought
was to
1) Isolate the cross-module build infrastructure
2) Provide a signal to the driver that a dependency had swiftdeps info
in it
But the driver need only notice swiftmodule files as external
dependencies and try to extract that information if it can to divine the
signal it needs. Additionally, we can give it fingerprints as priors to
let it know there might be incremental info to be had.
Previously Swift enabled the "UseOdrIndicator" ASan instrumentation mode
and gave no option to disable this. This probably wasn't intentional but
happened due to the fact the
`createModuleAddressSanitizerLegacyPassPass()` function has a default
value for the `UseOdrIndicator` parameter of `true` and in Swift we
never specified this parameter explicitly.
Clang disables the "UseOdrIndicator" mode by default but allows it to be
enabled using the `-fsanitize-address-use-odr-indicator` flag.
Having "UseOdrIndicator" off by default is probably the right
default choice because it bloats the binary. So this patch changes the
Swift compiler to match Clang's behavior.
This patch disables the "UseOdrIndicator" mode by default but adds a
hidden driver and frontend flag (`-sanitize-address-use-odr-indicator`)
to enable it. The flag is hidden so that we can remove it in the future
if needed.
A side effect of disabling "UseOdrIndicator" is that by we will no
longer use private aliases for poisoning globals. Private aliases were
introduced to avoid crashes
(https://github.com/google/sanitizers/issues/398) due to ODR violations
with non-instrumented binaries. On Apple platforms the use of two-level
namespaces probably means that using private aliases wasn't ever really
necessary to avoid crashes. On platforms with a flat linking namespace
(e.g. Linux) using private aliases might matter more but should users
actually run into problems they can either:
* Fix their environment to remove the ODR, thus avoiding the crash.
* Instrument the previously non-instrumented code to avoid the crash.
* Use the new `-sanitize-address-use-odr-indicator` flag
rdar://problem/69335186
Introducing new entry-points that can be used from both Driver and Frontend clients, using an intermediary new type: `DetailedMessagePayload`, when needed.
VS2017 seems to have problems with ternaries where the two branches are
given as std::move() (see https://stackoverflow.com/questions/53374182)
and the compiler was given a warning C4172 "returning address of local
or temporary".
Switching the ternary operator into a `if/else` works correctly for
VS2017 and should work for other compilers too, without changing the
meaning (as far as I see).
This was causing the Windows VS2017 CI to fail compiling the stdlib.
When bisecting, the problem was reduced to the changes in #35034. In a
debugger I managed to see the problem being a heap corruption. With all
the moves I was suspicious that something was being used while
destroyed, but the code was correct. While recompiling the compiler
warning gave me the clue.
Starting at a crude -1000, each invocation primary input will get its own unique quasi-Pid.
Invocations with only one primary (non-batch) will get a real OS Pid.
The selection of the constant starting point matches what the driver does when outputting its parseable output.
We're going to play a dirty, dirty trick - but it'll make our users'
lives better in the end so stick with me here.
In order to build up an incremental compilation, we need two sources of
dependency information:
1) "Priors" - Swiftdeps with dependency information from the past
build(s)
2) "Posteriors" - Swiftdeps with dependencies from after we rebuild the
file or module or whatever
With normal swift files built in incremental mode, the priors are given by the
swiftdeps files which are generated parallel to a swift file and usually
placed in the build directory alongside the object files. Because we
have entries in the output file map, we can always know where these
swiftdeps files are. The priors are integrated by the driver and then
the build is scheduled. As the build runs and jobs complete, their
swiftdeps are reloaded and re-integrated. The resulting changes are then
traversed and more jobs are scheduled if necessary. These give us the
posteriors we desire.
A module flips this on its head. The swiftdeps information serialized
in a module functions as the *posterior* since the driver consuming the
module has no way of knowing how to rebuild the module, and because its
dependencies are, for all intents and purposes, fixed in time. The
missing piece of the puzzle is the priors. That is, we need some way of
knowing what the "past" interface of the module looked like so we can
compare it to the "present" interface. Moreover, we need to always know
where to look for these priors.
We solve this problem by serializing a file alongside the build record:
the "external" build record. This is given by a... creative encoding
of multiple source file dependency graphs into a single source file
dependency graph. The rough structure of this is:
SourceFile => interface <BUILD_RECORD>.external
| - Incremental External Dependency => interface <MODULE_1>.swiftmodule
| | - <dependency> ...
| | - <dependency> ...
| | - <dependency> ...
| - Incremental External Dependency => interface <MODULE_2>.swiftmodule
| | - <dependency> ...
| | - <dependency> ...
| - Incremental External Dependency => interface <MODULE_3>.swiftmodule
| - ...
Sorta, `cat`'ing a bunch of source file dependency graphs together but
with incremental external dependency nodes acting as glue.
Now for the trick:
We have to unpack this structure and integrate it to get our priors.
This is easy. The tricky bit comes in integrate itself. Because the
top-level source file node points directly at the external build record,
not the original swift modules that defined these dependency nodes, we
swap the key it wants to use (the external build record) for the
incremental external dependency acting as the "parent" of the dependency
node. We do this by following the arc we carefully laid down in the
structure above.
For rdar://69595010
Goes a long way towards rdar://48955139, rdar://64238133
In order to extract the module dependency graph from the compilation the driver just ran, define a separate semantic type to hold a result code and the graph itself.
The target was added for Unix toolchains in #901, but a later pull #1891 added
it again. Since clang only uses the last target flag that's passed in, all
customization done for the first one was unused these last 4+ years, so remove
it and change tests that look for custom strings passed by the first one.
The "wave" of a compilation job describes the number of indirections through other compile jobs the driver required to reach the decision to schedule a job. In incremental mode, it should always be the case that it takes no more than two complete waves to arrive at a fixpoint in the build. This is a natural consequence of the structure of the dependencies emitted by the Swift frontend - namely we rely on transitivity in dependency arcs.
A quick proof sketch: Suppose an arbitrary perturbation of the inputs to an incremental compilation session are made. In the first wave, dependency edges from the prior build's state (the "zeroeth wave") are loaded and the files corresponding to invalidated edges are scheduled into the first wave. Supposing the second wave is not the null set - the trivial case - there are additional arcs that were invalidated. Now suppose that there were a third wave. Take an arbitrary arc invalidated by this third wave. It must be the case that the file containing the use is not new - else it would be scheduled. Further it must be the case that its def was not invalidated by the zeroeth or first waves of compilation otherwise we would have scheduled it into the first or second waves. Finally, it must have a use that was discovered in the second wave. But in order for that use to have been included in the second wave, there must have been an invalidated arc created by the first wave. By transitivity of dependency arcs, there must therefore be a dependency arc from a definition invalidated in the first wave to our third wave job, which implies that the file would be scheduled into the second wave!
[Insert contradiction pig image here]
In order to unblock the SwiftWASM project, which relies on an
incremental build of the Swift driver that relies on the merge-modules
job always being run. The situation appears to be something like this:
1) An incremental build is run
2) Temporary swiftmodule outputs are laid down
3) merge-modules is skipped
4) modulewrap is run anyways and reads the empty temp file
We should fix this by skipping modulewrap if we can skip merge-modules.
But for now, be conservative and fall back to the status quo behavior of
always running merge-modules whenever we encounter a modulewrap job.
Plumb the logic necessary to schedule merge-modules incrementally. This means that if any inputs jobs to merge-modules run, merge-modules is run. But, if they are all skipped, merge-modules will be skipped as well.
This requires some light special-casing of the legacy driver's incremental job handling because it assumes in a few places it can always extract a swiftdeps file. This invariant will be further broken when the precompile step for bridging headers is skipped as well.
rdar://65893400
The Optional parameter here was being copied instead of being taken by
const reference. The expectation is that we call this function and
extract the data from a ModuleDepGraphNode node, but instead we were
extracting a copy which would promptly blow up.
Thanks ASAN!
