Improve swift dependency scanner by validating and selecting dependency
module into scanner. This provides benefits that:
* Build system does not need to schedule interface compilation task if
the candidate module is picked, it can just use the candidate module
directly.
* There is no need for forwarding module in the explicit module build.
Since the build system is coordinating the build, there is no need for
the forwarding module in the module cache to avoid duplicated work,
* This also correctly supports all the module loading modes in the
dependency scanner.
This is achieved by only adding validate and up-to-date binary module as
the candidate module for swift interface module dependency. This allows
caching build to construct the correct dependency in the CAS. If there
is a candidate module for the interface module, dependency scanner will
return a binary module dependency in the dependency graph.
The legacy behavior is mostly preserved with a hidden frontend flag
`-no-scanner-module-validation`, while the scanner output is mostly
interchangeable with new scanner behavior with `prefer-interface` module
loading mode except the candidate module will not be returned.
rdar://123711823
It is possible that import resolution failed because we are attempting to resolve a module which can only be brought in via a modulemap of a different Clang module dependency which is not otherwise on the current search paths. For example, suppose we are scanning a '.swiftinterface' for module 'Foo', which contains:
'''
@_exported import Foo
import Bar
...
Where 'Foo' is the underlying Framework clang module whose '.modulemap' defines an auxiliary module 'Bar'. Because 'Foo' is a framework, its modulemap is under '<some_framework_search_path>/Foo.framework/Modules/module.modulemap'. Which means that lookup of `Bar` alone from Swift will not be able to locate the module in it. However, the lookup of Foo will itself bring in the auxiliary module becuase the Clang scanner instance scanning for clang module Foo will be able to find it in the corresponding framework module's modulemap and register it as a dependency which means it will be registered with the scanner's cache in the step above. To handle such cases, we first add all successfully-resolved modules and (for Clang modules) their transitive dependencies to the cache, and then attempt to re-query imports for which resolution originally failed from the cache. If this fails, then the scanner genuinely failed to resolve this dependency.
In case import resolution order somehow sometimes matters, it's prudent to process/resolve/locate implicitly-imported modules first.
Resolves rdar://113917657
The code of `ScanDependencies.cpp` was creating invalid JSON since #66031
because in the case of having `extraPcmArgs` and `swiftOverlayDependencies`,
but not `bridgingHeader`, a comma will not be added at the end of
`extraPcmArgs`, creating an invalid JSON file. Additionally that same PR
added a trailing comma at the end of the `swiftOverlayDependencies`, which
valid JSON does not allow, but that bug was removed in #66366.
Both problems are, however, present in the 5.9 branch, because #66936
included #66031, but not #66366.
Besides fixing the problem in `ScanDependencies.cpp` I modified every test
that uses `--scan-dependencies` to pass the produced JSON through
Python's `json.tool` in order to validate proper JSON is produced. In
most cases I was able to pipe the output of the tool into `FileCheck`,
but in some cases the validation is done by itself because the checks
depend on the exact format generated by `--scan-dependencies`. In
a couple of tests I added a call to `FileCheck` that seemed to be
missing.
Without these changes, two tests seems to be generating invalid JSON in
my machine:
- `ScanDependencies/local_cache_consistency.swift` (which outputs `Expecting ',' delimiter: line 525 column 11 (char 22799)`)
- `ScanDependencies/placholder_overlay_deps.swift`
This ensures that when the dependency scanner is invoked with additional clang (`-Xcc`) options, the Clang scanner is correctly configured using these options.
In the fast dependency scanner, depending on whether a module intrface was found via the import search path or framework search path, encode into the dependency graph Swift module details, whether a given module is a framework.
