Replace the one-off compiler flag for Library Evolution with an
optional language feature. This makes the
`hasFeature(LibraryEvolution)` check work in an `#if`, and is
otherwise just cleanup.
Tracked by rdar://161125572.
This will mainly be useful once extension binding is fully
requestified, but even now it's a good idea to ensure module loading
isn't kicking name lookup.
- 'SwiftModuleScanner' will now be owned directly by the 'ModuleDependencyScanningWorker' and will contain all the necessary custom logic, instead of being instantiated by the module interface loader for each query
- Moves ownership over module output path and sdk module output path directly into the scanning worker, instead of the cache
With '-sdk-module-cache-path', Swift textual interfaces found in the SDK will be built into a separate SDK-specific module cache.
Clang modules are not yet affected by this change, pending addition of the required API.
Use IncludeTreeFileList instead of full feature CASFS for swift
dependency filesystem. This allows smaller CAS based VFS that is smaller
and faster. This is enabled by the CAS enabled compilation does not
need to iterate file system.
rdar://136787368
Fix the problem that when the only module can be found is an
invalid/out-of-date swift binary module, canImport and import statement
can have different view for if the module can be imported or not.
Now canImport will evaluate to false if the only module can be found for
name is an invalid swiftmodule, with a warning with the path to the
module so users will not be surprised by such behavior.
rdar://128876895
Allow DependencyScanner to canonicalize path using a prefix map. When
option `-scanner-prefix-map` option is used, dependency scanner will
remap all the input paths in following:
* all the paths in the CAS file system or clang include tree
* all the paths related to input on the command-line returned by scanner
This allows all the input paths to be canonicalized so cache key can be
computed reguardless of the exact on disk path.
The sourceFile field is not remapped so build system can track the exact
file as on the local file system.
From being a scattered collection of 'static' methods in ScanDependencies.cpp
and member methods of ASTContext. This makes 'ScanDependencies.cpp' much easier
to read, and abstracts the actual scanning logic away to a place with common
state which will make it easier to reason about in the future.
Reformatting everything now that we have `llvm` namespaces. I've
separated this from the main commit to help manage merge-conflicts and
for making it a bit easier to read the mega-patch.
This is phase-1 of switching from llvm::Optional to std::optional in the
next rebranch. llvm::Optional was removed from upstream LLVM, so we need
to migrate off rather soon. On Darwin, std::optional, and llvm::Optional
have the same layout, so we don't need to be as concerned about ABI
beyond the name mangling. `llvm::Optional` is only returned from one
function in
```
getStandardTypeSubst(StringRef TypeName,
bool allowConcurrencyManglings);
```
It's the return value, so it should not impact the mangling of the
function, and the layout is the same as `std::optional`, so it should be
mostly okay. This function doesn't appear to have users, and the ABI was
already broken 2 years ago for concurrency and no one seemed to notice
so this should be "okay".
I'm doing the migration incrementally so that folks working on main can
cherry-pick back to the release/5.9 branch. Once 5.9 is done and locked
away, then we can go through and finish the replacement. Since `None`
and `Optional` show up in contexts where they are not `llvm::None` and
`llvm::Optional`, I'm preparing the work now by going through and
removing the namespace unwrapping and making the `llvm` namespace
explicit. This should make it fairly mechanical to go through and
replace llvm::Optional with std::optional, and llvm::None with
std::nullopt. It's also a change that can be brought onto the
release/5.9 with minimal impact. This should be an NFC change.
For a `@Testable` import in program source, if a Swift interface dependency is discovered, and has an adjacent binary `.swiftmodule`, open up the module, and pull in its optional dependencies. If an optional dependency cannot be resolved on the filesystem, fail silently without raising a diagnostic.
This changes the scanner's behavior to "resolve" a discovered module's dependencies to a set of Module IDs: module name + module kind (swift textual, swift binary, clang, etc.).
The 'ModuleDependencyInfo' objects that are stored in the dependency scanner's cache now carry a set of kind-qualified ModuleIDs for their dependencies, in addition to unqualified imported module names of their dependencies.
Previously, the scanner's internal state would cache a module dependnecy as having its own set of dependencies which were stored as names of imported modules. This led to a design where any time we needed to process the dependency downstream from its discovery (e.g. cycle detection, graph construction), we had to query the ASTContext to resolve this dependency's imports, which shouldn't be necessary. Now, upon discovery, we "resolve" a discovered dependency by executing a lookup for each of its imported module names (this operation happens regardless of this patch) and store a fully-resolved set of dependencies in the dependency module info.
Moreover, looking up a given module dependency by name (via `ASTContext`'s `getModuleDependencies`) would result in iterating over the scanner's module "loaders" and querying each for the module name. The corresponding modules would then check the scanner's cache for a respective discovered module, and if no such module is found the "loader" would search the filesystem.
This meant that in practice, we searched the filesystem on many occasions where we actually had cached the required dependency, as follows:
Suppose we had previously discovered a Clang module "foo" and cached its dependency info.
-> ASTContext.getModuleDependencies("foo")
--> (1) Swift Module "Loader" checks caches for a Swift module "foo" and doesn't find one, so it searches the filesystem for "foo" and fails to find one.
--> (2) Clang Module "Loader" checks caches for a Clang module "foo", finds one and returns it to the client.
This means that we were always searching the filesystem in (1) even if we knew that to be futile.
With this change, queries to `ASTContext`'s `getModuleDependencies` will always check all the caches first, and only delegate to the scanner "loaders" if no cached dependency is found. The loaders are then no longer in the business of checking the cached contents.
To handle cases in the scanner where we must only lookup either a Swift-only module or a Clang-only module, this patch splits 'getModuleDependencies' into an alrady-existing 'getSwiftModuleDependencies' and a newly-added 'getClangModuleDependencies'.
Previously, when evaluating a `#if canImport(Module, _version: 42)` directive the compiler could diagnose and ignore the directive under the following conditions:
- The associated binary module is corrupt/bogus.
- The .tbd for an underlying Clang module is missing a current-version field.
This behavior is surprising when there is a valid `.swiftinterface` available and it only becomes apparent when building against an SDK with an old enough version of the module that the version in the `.swiftinterface` is too low, making this failure easy to miss. Some modules have different versioning systems for their Swift and Clang modules and it can also be intentional for a distributed binary `.swiftmodule` to contain bogus data (to force the compiler to recompile the `.swiftinterface`) so we need to handle both of these cases gracefully and predictably.
Now the compiler will enumerate all module loaders, ask each of them to attempt to parse the module version and then consistently use the parsed version from a single source. The `.swiftinterface` is preferred if present, then the binary module if present, and then finally the `.tbd`. The `.tbd` is still always used exclusively for the `_underlyingVersion` variant of `canImport()`.
Resolves rdar://88723492
This change tweaks the 'GlobalModuleDependenciesCache', which persists across scanner invocations with the same 'DependencyScanningTool' to no longer cache discovered Clang modules.
Doing so felt like a premature optimization, and we should instead attempt to share as much state as possible by keeping around the actual Clang scanner's state, which performs its own caching. Caching discovered dependencies both in the Clang scanner instance, and in our own cache is much more error-prone - the Clang scanner has a richer context for what is okay and not okay to cache/re-use.
Instead, we still cache discovered Clang dependencies *within* a given scan, since those are discovered using a common Clang scanner instance and should be safe to keep for the duration of the scan.
This change should make it simpler to pin down the core functionality and correctness of the scanner.
Once we turn our attention to the scanner's performance, we can revisit this strategy and optimize the caching behaviour.
The ObjCMethodLookupTable for protocols was not being serialized and rebuilt on load, so NominalTypeDecl::lookupDirect() on selectors was not working correctly for deserialized types. Correct this oversight.
canImport should be able to take an additional parameter labeled by either version or
underlyingVersion. We need underlyingVersion for clang modules with Swift overlays because they
have separate version numbers. The library users are usually interested in checking the importability
of the underlying clang module instead of its Swift overlay.
Part of rdar://73992299
Module interface builder used to maintain a separate compiler instance for
building Swift modules. The configuration of this compiler instance is also
useful for dependencies scanner because it needs to emit front-end compiler invocation
for building Swift modules explicitly.
This patch refactor the configuration out to a delegate class, and the
delegate class is also used by the dependency scanner.
Additional flags in interface files may change parsing behavior like #if
statements. We should use a fresh ASTContext with these additional
flags when parsing interface files to collect imports.
rdar://62612027
Implement a new "fast" dependency scanning option,
`-scan-dependencies`, in the Swift frontend that determines all
of the source file and module dependencies for a given set of
Swift sources. It covers four forms of modules:
1) Swift (serialized) module files, by reading the module header
2) Swift interface files, by parsing the source code to find imports
3) Swift source modules, by parsing the source code to find imports
4) Clang modules, using Clang's fast dependency scanning tool
A single `-scan-dependencies` operation maps out the full
dependency graph for the given Swift source files, including all
of the Swift and Clang modules that may need to be built, such
that all of the work can be scheduled up front by the Swift
driver or any other build system that understands this
option. The dependency graph is emitted as JSON, which can be
consumed by these other tools.
Use the delayed parsing of function bodies for source imports, and
switch source imports over to lazy type checking. There's no point in
doing a full type check for them.
We've been running doxygen with the autobrief option for a couple of
years now. This makes the \brief markers into our comments
redundant. Since they are a visual distraction and we don't want to
encourage more \brief markers in new code either, this patch removes
them all.
Patch produced by
for i in $(git grep -l '\\brief'); do perl -pi -e 's/\\brief //g' $i & done
* Pack the bits for IfConfigDecls into Decl
* Don't open symbols into a module when evaluating canImport statements
The module loaders now have API to check whether a given module can be
imported without importing the referenced module. This provides a
significant speed boost to condition resolution and no longer
introduces symbols from the referenced module into the current context
without the user explicitly requesting it.
The definition of ‘canImport’ does not necessarily mean that a full
import without error is possible, merely that the path to the import is
visible to the compiler and the module is loadable in some form or
another.
Note that this means this check is insufficient to guarantee that you
are on one platform or another. For those kinds of checks, use
‘os(OSNAME)’.
Since resilience is a property of the module being compiled,
not decls being accessed, we need to record which types are
resilient as part of the module.
Previously we would only ever look at the @_fixed_layout
attribute on a type. If the flag was not specified, Sema
would slap this attribute on every type that gets validated.
This is wasteful for non-resilient builds, because there
all types get the attribute. It was also apparently wrong,
and I don't fully understand when Sema decides to validate
which decls.
It is much cleaner conceptually to just serialize this flag
with the module, and check for its presence if the
attribute was not found on a type.
Modules occupy a weird space in the AST now: they can be treated like
types (Swift.Int), which is captured by ModuleType. They can be
treated like values for disambiguation (Swift.print), which is
captured by ModuleExpr. And we jump through hoops in various places to
store "either a module or a decl".
Start cleaning this up by transforming Module into ModuleDecl, a
TypeDecl that's implicitly created to describe a module. Subsequent
changes will start folding away the special cases (ModuleExpr ->
DeclRefExpr, name lookup results stop having a separate Module case,
etc.).
Note that the Module -> ModuleDecl typedef is there to limit the
changes needed. Much of this patch is actually dealing with the fact
that Module used to have Ctx and Name public members that now need to
be accessed via getASTContext() and getName(), respectively.
Swift SVN r28284
This functionality doesn’t really change what we accept right now, because we eagerly import all of the methods of a class when we do *any* kind of lookup into the class. However, when we manage to stop doing that, this operation will become more important.
Swift SVN r23289
This performs very conservative dependency generation for each compile task
within a full compilation. Any source file, swiftmodule, or Objective-C
header file that is /touched/ gets added to the dependencies list, which
is written out on a per-input basis at the end of compilation.
This does /not/ handle dependencies for the aggregated swiftmodule, swiftdoc,
generated header, or linked binary. This is just the minimum needed to get
Xcode to recognize what needs to be rebuilt when a header or Swift source
file changes. We can revisit this later.
This finishes <rdar://problem/14899639> for now.
Swift SVN r18045
Also, create the Clang module loader directly rather than indirecting through
a "get constructor" function. It's no longer a valid configuration to not
have a Clang importer.
Swift SVN r16862
...unless the functions are declared [transparent], or if we're in an
immediate mode (in which case we won't get a separate chance to link
against the imported TUs).
This is an optimization that will matter more when we start dealing with
Xcode projects with many cross-file dependencies, especially if we have
some kind of implicit import of the other source files in the project.
In the future, we may want to parse more function bodies for the purpose
of inlining, not just the transparent ones, but we weren't taking
advantage of that now, so it's not a regression. (We're still not taking
advantage of it even for [transparent] functions.)
Swift SVN r7698
Now that we have true serialized modules, the standard library can import
the Builtin module without any special direction (beyond -parse-stdlib),
and anyone can include those modules without special direction.
Swift SVN r6752
This causes the SourceLoader to recursively parse the imported module in standard
library mode, giving it access to the Builtin module.
This is all a terrible hack and should be ripped out with great victory someday, but
until we have binary modules that persist the build setting used to produce the
module, this is the best we can do.
Swift SVN r5847
This replaces the obscure, inefficient lookup into extensions with
something more straightforward: walk all of the known extensions
(available as a simple list), then eliminate any declarations that
have been shadowed by other declarations. The shadowing rules still
need to consider the module re-export DAG, but we'll leave that for
later.
As part of this, keep track of the last time we loaded extensions for
a given nominal type. If the list of extensions is out-of-date with
respect to the global generation count (which tracks resolved module
imports), ask the modules to load any additional extensions. Only the
Clang module importer can currently load extensions in this manner.
Swift SVN r5223
This paves the way for having a Swift module importer. The eventual goal
here is to eliminate all explicit uses of the Clang module loader, but
I'm not going to push too hard on that for now.
Swift SVN r5092
