Rather than exposing an `addFile` member on
ModuleDecl, have the `create` members take a
lambda that populates the files for the module.
Once module construction has finished, the files
are immutable.
Use IncludeTreeFileList instead of full feature CASFS for swift
dependency filesystem. This allows smaller CAS based VFS that is smaller
and faster. This is enabled by the CAS enabled compilation does not
need to iterate file system.
rdar://136787368
ModuleDecl kept track of all of the source files in the module so that it
could find the source file containing a given location, which relied on
a sorted array all of these source files. SourceManager has its own
similar data structure for a similar query mapping the locations to
buffer IDs.
Replace ModuleDecl's dats structure with a use of the SourceManager's version
with the mapping from buffer IDs to source files.
When '.package.swiftinterface' loading ('-experimental-package-interface-load') is disabled and when '-scanner-module-validation' is disabled, the scanner defaults to locating the non-package textual interface and may specify its adjacent binary module as a valid candidate binary module to use. If said candidate is up-to-date and ends up getting used, and belongs to the same package as the loading Swift source, then the source compilation may attempt to load its package-only dependencies. Since the scanner only parsed the non-package textual interface, those dependencies are not located and specified as inputs to compilation. This change causes the scanner, in such cases, to also lookup package-only dependencies in adjacent binary Swift modules of textual Swift module dependencies, if such dependency belongs to the same package as the source target being scanned.
Resolves rdar://135215789
As-is, this default interferes with the incremental build machinery which conservatively assumes that binary module dependencies must cause dependents to be re-built.
The decision to exclude `-Xcc -D` options from swift module hash
actually doesn't help to solve the problem. It wouldn't reduce the
module variants (or the number of swiftmodule build commands) because
the command-line also encodes all the clang PCM dependencies that do get
affected by `-Xcc` flags.
To avoid the false sharing and the nondeterministic build products,
include most of the `-Xcc` flags, except include search path, into swift
module hash.
rdar://132046247
When the swiftmodule is built with different clang importer arguments,
they can have the same module hash, causing them to be wrongly re-used even
they contains different interfaces. Add ReducedExtraArgs to the module hash to
disambiguate them.
However, some Xcc arguments, most commonly `-D` options do not affect the
swiftmodule being generated. Do not pass `-Xcc -DARGS` to swift
interface compilation to reduce the amount of module variants in the
build.
rdar://131408266
Fix the problem that when the only module can be found is an
invalid/out-of-date swift binary module, canImport and import statement
can have different view for if the module can be imported or not.
Now canImport will evaluate to false if the only module can be found for
name is an invalid swiftmodule, with a warning with the path to the
module so users will not be surprised by such behavior.
rdar://128876895
Although I don't plan to bring over new assertions wholesale
into the current qualification branch, it's entirely possible
that various minor changes in main will use the new assertions;
having this basic support in the release branch will simplify that.
(This is why I'm adding the includes as a separate pass from
rewriting the individual assertions)
This change modifies the dependency scanner to keep track of source locations of each encountered 'import' statement, in order to be able to emit diagnostics with source locations if an import failed to resolve.
- Keep track of each 'import' statement's source buffer, line number, and column number when adding it. The dependency scanner utilizes separate compilation instances, and therefore separate Source Managers for scanning `import` statements of user sources and textual interfaces of Swift dependencies. Since import resolution may happen in the main scanner compilation instance while the `import` itself was found by an interface-scanning sub-instance, we cannot simply hold on to the import's `SourceLoc`.
- Add libSwiftScan API for diagnostics to carry above source locations to clients.
Cleanup testable module lookup implementation by explicitly lookup
binary module only when requested, rather than changing the scanner
module load mode.
Follow-up adjustment for binary module selection in dependency scanning
time. If a testable binary module doesn't have an interface file, it
should be used even it might pull in more dependencies.
Teach scanner to pick and choose binary modules correctly based on if it
is testable import or not. Some situations that scanner need to be
careful when testable is involved:
* When it is a regular import, it should not import binary modules that
are built with -enable-testing, it should prefer interfaces if that is
available.
* When testable import, it should only load binary module and it should
make sure the internal imports from binary modules are actually
required for testable import to work.
If a testable import only find a regular binary module, dependency
scanner currently will just preceed with such module and leave the
diagnostics to swift-frontend, because the alternative (failed to find
module) can be confusing to users.
rdar://125914165
Improve swift dependency scanner by validating and selecting dependency
module into scanner. This provides benefits that:
* Build system does not need to schedule interface compilation task if
the candidate module is picked, it can just use the candidate module
directly.
* There is no need for forwarding module in the explicit module build.
Since the build system is coordinating the build, there is no need for
the forwarding module in the module cache to avoid duplicated work,
* This also correctly supports all the module loading modes in the
dependency scanner.
This is achieved by only adding validate and up-to-date binary module as
the candidate module for swift interface module dependency. This allows
caching build to construct the correct dependency in the CAS. If there
is a candidate module for the interface module, dependency scanner will
return a binary module dependency in the dependency graph.
The legacy behavior is mostly preserved with a hidden frontend flag
`-no-scanner-module-validation`, while the scanner output is mostly
interchangeable with new scanner behavior with `prefer-interface` module
loading mode except the candidate module will not be returned.
rdar://123711823
Add support for cross import modules by ingesting swiftoverlay files for
the cross import into CAS file system.
The long-term better fix will be just passing the cross import
information from scanner to swift-frontend so frontend doesn't need to
read overlay files again to figure out the cross import module.
rdar://123839248
Add an experimental option to tell dependency scanner to report clang
cc1 args should be used to construct clang importer in all constructed
swift-frontend tasks.
When caching build is enabled, teach dependency scanner to report
command-lines with `-direct-clang-cc1-module-build` so the later
compilation can instantiate clang importer with cc1 args directly. This
avoids running clang driver code, which might involve file system
lookups, which are the file deps that are not captured and might result
in different compilation mode.
rdar://119275464
When scanning finds a dependency in the same package, do not load
public/private swiftinterface since they do not have the package level
decl to compile the current module. Always prefer package module (if
enabled), or use binary module, unless it is building a public/private
swiftinterface file in which case the interface file is preferred.
This also does some clean up to sync up the code path between implicit
and explicit module finding path.
rdar://122356964
Always prefer binary module when using @testable imports because the
swiftmodule rebuilt from interface cannot be imported as testable.
rdar://123120159
The SDK build version is a decent heuristic for expected changes in the
SDK. Any change in SDK, to clang headers in particular, can break
references from cached swiftmodules.
Track the SDK build version as part of the swiftmodule cache hash. This
will ensure we rebuild from swiftinterfaces on SDK updates.
rdar://122655978
A swiftmodule can only be correctly ingested by a compiler
that has a matching state of using or not-using
NoncopyableGenerics.
The reason for this is fundamental: the absence of a Copyable
conformance in the swiftmodule indicates that a type is
noncopyable. Thus, if a compiler with NoncopyableGenerics
reads a swiftmodule that was not compiled with that feature,
it will think every type in that module is noncopyable.
Similarly, if a compiler with NoncopyableGenerics produces a
swiftmodule, there will be Copyable requirements on each
generic parameter that the compiler without the feature will
become confused about.
The solution here is to trigger a module mismatch, so that
the compiler re-generates the swiftmodule file using the
swiftinterface, which has been kept compatible with the compiler
regardless of whether the feature is enabled.
Add a new flag to enable package interface loading.
Use the last value of package-name in case of dupes.
Rename PrintInterfaceContentMode as InterfaceMode.
Update diagnostics.
Test package interface loading with various scenarios.
Test duplicate package-name.
It has an extension .package.swiftinterface and contains package decls
as well as SPIs and public/inlinable decls. When a module is loaded
from interface, it now looks up the package-name in the interface
and checks if the importer is in the same package. If so, it uses
that package interface found to load the module. If not, uses the existing
logic to load modules.
Resolves rdar://104617854
Allow DependencyScanner to canonicalize path using a prefix map. When
option `-scanner-prefix-map` option is used, dependency scanner will
remap all the input paths in following:
* all the paths in the CAS file system or clang include tree
* all the paths related to input on the command-line returned by scanner
This allows all the input paths to be canonicalized so cache key can be
computed reguardless of the exact on disk path.
The sourceFile field is not remapped so build system can track the exact
file as on the local file system.
'ModuleDependencyScanner' maintains a Thread Pool along with a pool of workers
which are capable of executing a filesystem lookup of a named module dependency.
When resolving imports of a given Swift module, each import's resolution
operation can be issued asunchronously.
