In order to unblock the SwiftWASM project, which relies on an
incremental build of the Swift driver that relies on the merge-modules
job always being run. The situation appears to be something like this:
1) An incremental build is run
2) Temporary swiftmodule outputs are laid down
3) merge-modules is skipped
4) modulewrap is run anyways and reads the empty temp file
We should fix this by skipping modulewrap if we can skip merge-modules.
But for now, be conservative and fall back to the status quo behavior of
always running merge-modules whenever we encounter a modulewrap job.
Plumb the logic necessary to schedule merge-modules incrementally. This means that if any inputs jobs to merge-modules run, merge-modules is run. But, if they are all skipped, merge-modules will be skipped as well.
This requires some light special-casing of the legacy driver's incremental job handling because it assumes in a few places it can always extract a swiftdeps file. This invariant will be further broken when the precompile step for bridging headers is skipped as well.
rdar://65893400
A more durable form of #34218. Keep a side cache of externally-dependent
jobs for now. This ensures our pseudo-Jobs don't get prematurely
deallocated before the tracing machinery has had a chance to report the
structure of the Job graph.
rdar://70053563
An incremental build involving incremental external dependencies behaves as a hybrid between an external dependency and a normal swiftdeps-laden Swift file.
In the simplest case, we will fall back to the behavior of a plain external dependency today. That is, we will check its timestamp, then schedule all jobs that involve these external dependencies if it is out of date.
Where things get interesting is when cross-module incremental builds are enabled. In such a case, we know that a previous compiler has already emitted serialized swiftdeps information inside of a swiftmodule file. Moreover, we know that that swiftmodule file was loaded by the build of the current swift module. Finally, thanks to the previous stack of commits, we now know exactly how to extract this information from the swiftmodule file. To bring this all home, we unpack incremental dependency information from external dependencies, then integrate them into the current dependency graph - as though they were any other swiftdeps file. This neatly extends the single-module incremental logic to the multi-module case.
Treat any incremental external depends like normal external depends. This will eventually become the fallback behavior for cross-module incremental builds.
Annotate the covered switches with `llvm_unreachable` to avoid the MSVC
warning which does not recognise the covered switches. This allows us
to avoid a spew of warnings.
Restructure fine-grained-dependencies to enable unit testing
Get frontend to emit correct swiftdeps file (fine-grained when needed) and only emit dot file for -emit-fine-grained-dependency-sourcefile-dot-files
Use deterministic order for more information outputs.
Set EnableFineGrainedDependencies consistently in frontend.
Tolerate errors that result in null getExtendedNominal()
Fix memory issue by removing node everywhere.
Break up print routine
Be more verbose so it will compile on Linux.
Sort batchable jobs, too.
Restructure fine-grained-dependencies to enable unit testing
Get frontend to emit correct swiftdeps file (fine-grained when needed) and only emit dot file for -emit-fine-grained-dependency-sourcefile-dot-files
Use deterministic order for more information outputs.
Set EnableFineGrainedDependencies consistently in frontend.
Tolerate errors that result in null getExtendedNominal()
Fix memory issue by removing node everywhere.
Break up print routine
Be more verbose so it will compile on Linux.
Sort batchable jobs, too.
Repair the VS2017 build after #28543. It would fail to convert the
dependency set's keys into a vector of StringRefs:
```
error C2678: binary '*': no operator found which takes a left-hand operand of type 'const _Iter' (or there is no acceptable conversion)
with [
_Iter=llvm::StringMapKeyIterator<char>
]
C:\agent\_work\2\s\toolchain\llvm\include\llvm/ADT/StringMap.h(549): note: could be 'llvm::StringRef &llvm::StringMapKeyIterator<ValueTy>::operator *(void)'
with
[
ValueTy=char
]
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\\include\xmemory(217): note: while trying to match the argument list '(const _Iter)'
with
[
_Iter=llvm::StringMapKeyIterator<char>
]
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\\include\vector(1823): note: see reference to function template instantiation '_FwdIt *std::_Uninitialized_copy<_Iter,llvm::StringRef*,std::allocator<_Ty>>(const _InIt,const _InIt,_FwdIt,_Alloc &)' being compiled
with
[
_FwdIt=llvm::StringRef *,
_Iter=llvm::StringMapKeyIterator<char>,
_Ty=llvm::StringRef,
_InIt=llvm::StringMapKeyIterator<char>,
_Alloc=std::allocator<llvm::StringRef>
]
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\\include\vector(708): note: see reference to function template instantiation 'llvm::StringRef *std::vector<llvm::StringRef,std::allocator<_Ty>>::_Ucopy<_Iter>(_Iter,_Iter,llvm::StringRef *)' being compiled
with
[
_Ty=llvm::StringRef,
_Iter=llvm::StringMapKeyIterator<char>
]
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\\include\vector(708): note: see reference to function template instantiation 'llvm::StringRef *std::vector<llvm::StringRef,std::allocator<_Ty>>::_Ucopy<_Iter>(_Iter,_Iter,llvm::StringRef *)' being compiled
with
[
_Ty=llvm::StringRef,
_Iter=llvm::StringMapKeyIterator<char>
]
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\\include\vector(723): note: see reference to function template instantiation 'void std::vector<llvm::StringRef,std::allocator<_Ty>>::_Range_construct_or_tidy<_Iter>(_Iter,_Iter,std::forward_iterator_tag)' being compiled
with
[
_Ty=llvm::StringRef,
_Iter=llvm::StringMapKeyIterator<char>
]
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\\include\vector(723): note: see reference to function template instantiation 'void std::vector<llvm::StringRef,std::allocator<_Ty>>::_Range_construct_or_tidy<_Iter>(_Iter,_Iter,std::forward_iterator_tag)' being compiled
with
[
_Ty=llvm::StringRef,
_Iter=llvm::StringMapKeyIterator<char>
]
C:\agent\_work\2\s\toolchain\swift\lib\Driver\Compilation.cpp(1571): note: see reference to function template instantiation 'std::vector<llvm::StringRef,std::allocator<_Ty>>::vector<IteratorT,void>(_Iter,_Iter,const _Alloc &)' being compiled
with
[
_Ty=llvm::StringRef,
IteratorT=llvm::StringMapKeyIterator<char>,
_Iter=llvm::StringMapKeyIterator<char>,
_Alloc=std::allocator<llvm::StringRef>
]
C:\agent\_work\2\s\toolchain\swift\lib\Driver\Compilation.cpp(1571): note: see reference to function template instantiation 'std::vector<llvm::StringRef,std::allocator<_Ty>>::vector<IteratorT,void>(_Iter,_Iter,const _Alloc &)' being compiled
with
[
_Ty=llvm::StringRef,
IteratorT=llvm::StringMapKeyIterator<char>,
_Iter=llvm::StringMapKeyIterator<char>,
_Alloc=std::allocator<llvm::StringRef>
]
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\\include\xmemory(217): error C2100: illegal indirection
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\\include\xmemory(217): error C2062: type 'unknown-type' unexpected
```
Attempt to make it more explicit with a `std::copy`.
