[ParseableInterface] Add ‘forwarding modules’

A ‘forwarding module’ is a YAML file that’s meant to stand in for a .swiftmodule file and provide an up-to-date description of its dependencies, always using modification times.

When a ‘prebuilt module’ is first loaded, we verify that it’s up-to-date by hashing all of its dependencies. Since this is orders of magnitude slower than reading mtimes, we’ll install a `forwarding module` containing the mtimes of the now-validated dependencies.

include/swift/Frontend/FrontendInputsAndOutputs.h

@@ -173,7 +173,7 @@ public:
 
 private:
   friend class ArgsToFrontendOptionsConverter;
-  friend class ParseableInterfaceModuleLoader;
+  friend class ParseableInterfaceBuilder;
   void setMainAndSupplementaryOutputs(
       ArrayRef<std::string> outputFiles,
       ArrayRef<SupplementaryOutputPaths> supplementaryOutputs);

include/swift/Frontend/ParseableInterfaceModuleLoader.h

@@ -0,0 +1,164 @@
+//===--- ParseableInterfaceModuleLoader.h - Loads .swiftinterface files ---===//
+//
+// This source file is part of the Swift.org open source project
+//
+// Copyright (c) 2019 Apple Inc. and the Swift project authors
+// Licensed under Apache License v2.0 with Runtime Library Exception
+//
+// See https://swift.org/LICENSE.txt for license information
+// See https://swift.org/CONTRIBUTORS.txt for the list of Swift project authors
+//
+//===----------------------------------------------------------------------===//
+/// \file This implements the logic for loading and building parseable module
+/// interfaces.
+///
+/// === Loading Parseable Modules ===
+///
+/// If there is a .swiftinterface file corresponding to a given module name
+/// present in the frontend's search paths, then this module loader will look in
+/// the following places for a module:
+///
+/// - First, look in the module cache (specified by -module-cache-path)
+///   - We check here first because an existing .swiftmodule might be
+///     out-of-date, necessitating a rebuild. If a cached module is out-of-date,
+///     it's simply rebuilt.
+/// - Next, look adjacent to the .swiftinterface. If we find a module that's
+///   either loadable by this compiler, valid, and up-to-date, or totally
+///   unreadable, then delegate to the serialized module loader to either load
+///   or diagnose it.
+/// - Finally, look in the prebuilt module cache (specified
+///   by -prebuilt-module-cache-path)
+///
+/// If we can't find an appropriate module to load, we can always fall back and
+/// recompile the .swiftinterface file.
+///
+/// === Dependency Checking ===
+///
+/// Cached modules keep track of their dependencies' last modification time and
+/// file size. This means that checking if a module is up-to-date requires
+/// `stat`ing the dependencies and comparing the results from the filesystem
+/// with the results in the module.
+///
+/// Prebuilt modules, on the other hand, won't have a reliable modification
+/// time, as their dependencies live in the SDK. Prebuilt modules will instead
+/// keep track of the size and content hash of their dependencies.
+/// In order to avoid constantly re-hashing the dependencies, however, we will
+/// install a "forwarding module" in the regular cache.
+/// This "forwarding module"
+///  - Points to the prebuilt module on disk, and
+///  - Lists modification times from the last time we verified the content
+///
+/// So, to recap, there are 4 kinds of modules:
+/// ┌───────────────────────────────┐
+/// │     ┌───┐           ┌───┐     │
+/// │     │ I │           │ M │     │
+/// │     └───┘           └───┘     │
+/// │ .swiftinterface  .swiftmodule │
+/// │     ┌───┐           ┌───┐     │
+/// │     │ P │           │ F │     │
+/// │     └───┘           └───┘     │
+/// │   Prebuilt        Forwarding  │
+/// │ .swiftmodule     .swiftmodule │
+/// └───────────────────────────────┘
+///
+/// - Prebuilt modules have hash-based dependencies, cached modules have
+///   mod-time-based dependencies
+/// - Forwarding modules point to prebuilt modules and augment them with
+///   modification times
+///
+/// === Example Cache ===
+///
+/// Here's an example of what's in a prebuilt cache or module cache.
+///
+/// Say there are 4 frameworks, each exposing a .swiftinterface file.
+/// Then, we pre-build 2 of those frameworks and put them in the prebuilt cache.
+/// Finally, we import all 4 of those frameworks while building a project.
+///
+/// For the 2 frameworks with modules in the prebuilt cache, we'll have
+/// installed 2 forwarding modules. For the other 2 frameworks, we'll have
+/// compiled the interfaces and put them in the module cache.
+///
+///                                ┌─────┐
+///  ┌─────────────────────────────┤ SDK ├─────────────────────────┐
+///  │          ┌────────────────┐ └─────┘     ┌────────────────┐  │
+///  │  ┌───────┤ Framework Dirs ├────────┐   ┌┤ Prebuilt Cache ├┐ │
+///  │  │       └────────────────┘        │   │└────────────────┘│ │
+///  │  │  ┌───┐   ┌───┐   ┌───┐   ┌───┐  │   │   ┌───┐  ┌───┐   │ │
+///  │  │  │ I │   │ I │   │ I │   │ I │◀─┼───┼───│ P │  │ P │◀═╗│ │
+///  │  │  └───┘   └───┘   └───┘   └───┘  │   │   └───┘  └───┘  ║│ │
+///  │  │    ▲       ▲       ▲            │   │     ▲      │    ║│ │
+///  │  └────┼───────┼───────┼────────────┘   └─────╫──────┼────╫┘ │
+///  │       │       │       └──────────────────────╫──────┘    ║  │
+///  └───────┼───────┼──────────────────────────────╫───────────╫──┘
+///          │       │   ┌───────────────┐          ║           ║
+///          │  ┌────┼───┤ Module Cache  ├────────┐ ║           ║
+///          │  │    │   └───────────────┘        │ ║           ║
+///          │  │  ┌───┐   ┌───┐   ┌───┐   ┌───┐  │ ║           ║
+///          │  │  │ M │   │ M │   │ F │   │ F │  │ ║           ║
+///          │  │  └───┘   └───┘   └───┘   └───┘  │ ║           ║
+///          │  │            │       ║       ╚════╪═╝           ║
+///          │  └────────────┼───────╫────────────┘             ║
+///          └───────────────┘       ╚══════════════════════════╝
+///
+//===----------------------------------------------------------------------===//
+
+#include "swift/Basic/LLVM.h"
+#include "swift/Frontend/ParseableInterfaceSupport.h"
+#include "swift/Serialization/SerializedModuleLoader.h"
+
+namespace clang {
+  class CompilerInstance;
+}
+
+namespace swift {
+
+/// A ModuleLoader that runs a subordinate \c CompilerInvocation and
+/// \c CompilerInstance to convert .swiftinterface files to .swiftmodule
+/// files on the fly, caching the resulting .swiftmodules in the module cache
+/// directory, and loading the serialized .swiftmodules from there.
+class ParseableInterfaceModuleLoader : public SerializedModuleLoaderBase {
+  explicit ParseableInterfaceModuleLoader(ASTContext &ctx, StringRef cacheDir,
+                                          StringRef prebuiltCacheDir,
+                                          DependencyTracker *tracker,
+                                          ModuleLoadingMode loadMode)
+  : SerializedModuleLoaderBase(ctx, tracker, loadMode),
+  CacheDir(cacheDir), PrebuiltCacheDir(prebuiltCacheDir)
+  {}
+
+  std::string CacheDir;
+  std::string PrebuiltCacheDir;
+
+  std::error_code findModuleFilesInDirectory(
+    AccessPathElem ModuleID, StringRef DirPath, StringRef ModuleFilename,
+    StringRef ModuleDocFilename,
+    std::unique_ptr<llvm::MemoryBuffer> *ModuleBuffer,
+    std::unique_ptr<llvm::MemoryBuffer> *ModuleDocBuffer) override;
+
+
+public:
+  static std::unique_ptr<ParseableInterfaceModuleLoader>
+  create(ASTContext &ctx, StringRef cacheDir, StringRef prebuiltCacheDir,
+         DependencyTracker *tracker, ModuleLoadingMode loadMode) {
+    return std::unique_ptr<ParseableInterfaceModuleLoader>(
+      new ParseableInterfaceModuleLoader(ctx, cacheDir, prebuiltCacheDir,
+                                         tracker, loadMode));
+  }
+
+  /// Unconditionally build \p InPath (a swiftinterface file) to \p OutPath (as
+  /// a swiftmodule file).
+  ///
+  /// A simplified version of the core logic in #openModuleFiles, mostly for
+  /// testing purposes.
+  static bool buildSwiftModuleFromSwiftInterface(
+    ASTContext &Ctx, StringRef CacheDir, StringRef PrebuiltCacheDir,
+    StringRef ModuleName, StringRef InPath, StringRef OutPath,
+    bool SerializeDependencyHashes);
+};
+
+/// Extract the specified-or-defaulted -module-cache-path that winds up in
+/// the clang importer, for reuse as the .swiftmodule cache path when
+/// building a ParseableInterfaceModuleLoader.
+std::string
+getModuleCachePathFromClang(const clang::CompilerInstance &Instance);
+
+}

include/swift/Frontend/ParseableInterfaceSupport.h

@@ -2,7 +2,7 @@
 //
 // This source file is part of the Swift.org open source project
 //
-// Copyright (c) 2018 Apple Inc. and the Swift project authors
+// Copyright (c) 2019 Apple Inc. and the Swift project authors
 // Licensed under Apache License v2.0 with Runtime Library Exception
 //
 // See https://swift.org/LICENSE.txt for license information
@@ -14,9 +14,13 @@
 #define SWIFT_FRONTEND_PARSEABLEINTERFACESUPPORT_H
 
 #include "swift/Basic/LLVM.h"
-#include "swift/Serialization/SerializedModuleLoader.h"
+#include "swift/Basic/Version.h"
 #include "llvm/Support/Regex.h"
 
+#define SWIFT_INTERFACE_FORMAT_VERSION_KEY "swift-interface-format-version"
+#define SWIFT_TOOLS_VERSION_KEY "swift-tools-version"
+#define SWIFT_MODULE_FLAGS_KEY "swift-module-flags"
+
 namespace swift {
 
 class ModuleDecl;
@@ -29,6 +33,8 @@ struct ParseableInterfaceOptions {
   std::string ParseableInterfaceFlags;
 };
 
+extern version::Version InterfaceFormatVersion;
+
 llvm::Regex getSwiftInterfaceFormatVersionRegex();
 llvm::Regex getSwiftInterfaceModuleFlagsRegex();
 
@@ -49,71 +55,6 @@ bool emitParseableInterface(raw_ostream &out,
                             ParseableInterfaceOptions const &Opts,
                             ModuleDecl *M);
 
-/// Extract the specified-or-defaulted -module-cache-path that winds up in
-/// the clang importer, for reuse as the .swiftmodule cache path when
-/// building a ParseableInterfaceModuleLoader.
-std::string
-getModuleCachePathFromClang(const clang::CompilerInstance &Instance);
-
-/// A ModuleLoader that runs a subordinate \c CompilerInvocation and \c
-/// CompilerInstance to convert .swiftinterface files to .swiftmodule
-/// files on the fly, caching the resulting .swiftmodules in the module cache
-/// directory, and loading the serialized .swiftmodules from there.
-class ParseableInterfaceModuleLoader : public SerializedModuleLoaderBase {
-  explicit ParseableInterfaceModuleLoader(ASTContext &ctx, StringRef cacheDir,
-                                          StringRef prebuiltCacheDir,
-                                          DependencyTracker *tracker,
-                                          ModuleLoadingMode loadMode)
-    : SerializedModuleLoaderBase(ctx, tracker, loadMode),
-      CacheDir(cacheDir), PrebuiltCacheDir(prebuiltCacheDir)
-  {}
-
-  std::string CacheDir;
-  std::string PrebuiltCacheDir;
-
-  /// Wire up the SubInvocation's InputsAndOutputs to contain both input and
-  /// output filenames.
-  ///
-  /// This is a method rather than a helper function in the implementation file
-  /// because it accesses non-public bits of FrontendInputsAndOutputs.
-  static void configureSubInvocationInputsAndOutputs(
-    CompilerInvocation &SubInvocation, StringRef InPath, StringRef OutPath);
-
-  static bool buildSwiftModuleFromSwiftInterface(
-    llvm::vfs::FileSystem &FS, DiagnosticEngine &Diags, SourceLoc DiagLoc,
-    CompilerInvocation &SubInvocation, StringRef InPath, StringRef OutPath,
-    StringRef ModuleCachePath, DependencyTracker *OuterTracker,
-    bool ShouldSerializeDeps);
-
-  std::error_code findModuleFilesInDirectory(
-      AccessPathElem ModuleID, StringRef DirPath, StringRef ModuleFilename,
-      StringRef ModuleDocFilename,
-      std::unique_ptr<llvm::MemoryBuffer> *ModuleBuffer,
-      std::unique_ptr<llvm::MemoryBuffer> *ModuleDocBuffer) override;
-
-public:
-  static std::unique_ptr<ParseableInterfaceModuleLoader>
-  create(ASTContext &ctx, StringRef cacheDir, StringRef prebuiltCacheDir,
-         DependencyTracker *tracker, ModuleLoadingMode loadMode) {
-    return std::unique_ptr<ParseableInterfaceModuleLoader>(
-        new ParseableInterfaceModuleLoader(ctx, cacheDir, prebuiltCacheDir,
-                                           tracker, loadMode));
-  }
-
-  /// Unconditionally build \p InPath (a swiftinterface file) to \p OutPath (as
-  /// a swiftmodule file).
-  ///
-  /// A simplified version of the core logic in #openModuleFiles, mostly for
-  /// testing purposes.
-  static bool buildSwiftModuleFromSwiftInterface(ASTContext &Ctx,
-                                                 StringRef CacheDir,
-                                                 StringRef PrebuiltCacheDir,
-                                                 StringRef ModuleName,
-                                                 StringRef InPath,
-                                                 StringRef OutPath);
-};
-
-
 } // end namespace swift
 
 #endif

lib/Frontend/CMakeLists.txt

@@ -7,6 +7,7 @@ add_swift_host_library(swiftFrontend STATIC
   Frontend.cpp
   FrontendInputsAndOutputs.cpp
   FrontendOptions.cpp
+  ParseableInterfaceModuleLoader.cpp
   ParseableInterfaceSupport.cpp
   PrintingDiagnosticConsumer.cpp
   SerializedDiagnosticConsumer.cpp

lib/Frontend/Frontend.cpp

@@ -24,6 +24,7 @@
 #include "swift/Basic/SourceManager.h"
 #include "swift/Basic/Statistic.h"
 #include "swift/DWARFImporter/DWARFImporter.h"
+#include "swift/Frontend/ParseableInterfaceModuleLoader.h"
 #include "swift/Parse/DelayedParsingCallbacks.h"
 #include "swift/Parse/Lexer.h"
 #include "swift/SIL/SILModule.h"

lib/Frontend/ParseableInterfaceSupport.cpp

@@ -2,7 +2,7 @@
 //
 // This source file is part of the Swift.org open source project
 //
-// Copyright (c) 2018 Apple Inc. and the Swift project authors
+// Copyright (c) 2019 Apple Inc. and the Swift project authors
 // Licensed under Apache License v2.0 with Runtime Library Exception
 //
 // See https://swift.org/LICENSE.txt for license information
@@ -10,7 +10,6 @@
 //
 //===----------------------------------------------------------------------===//
 
-#define DEBUG_TYPE "textual-module-interface"
 #include "swift/AST/ASTContext.h"
 #include "swift/AST/Decl.h"
 #include "swift/AST/DiagnosticsFrontend.h"
@@ -24,627 +23,19 @@
 #include "swift/Frontend/ParseableInterfaceSupport.h"
 #include "swift/Frontend/PrintingDiagnosticConsumer.h"
 #include "swift/SILOptimizer/PassManager/Passes.h"
+#include "swift/Serialization/ModuleFormat.h"
 #include "swift/Serialization/SerializationOptions.h"
+#include "swift/Serialization/Validation.h"
 #include "clang/Basic/Module.h"
-#include "clang/Frontend/CompilerInstance.h"
-#include "clang/Lex/Preprocessor.h"
-#include "clang/Lex/PreprocessorOptions.h"
-#include "clang/Lex/HeaderSearch.h"
 #include "llvm/ADT/Hashing.h"
 #include "llvm/ADT/StringSet.h"
-#include "llvm/Support/xxhash.h"
-#include "llvm/Support/Debug.h"
-#include "llvm/Support/CommandLine.h"
-#include "llvm/Support/CrashRecoveryContext.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/Regex.h"
 #include "llvm/Support/StringSaver.h"
 
 using namespace swift;
-using FileDependency = SerializationOptions::FileDependency;
 
-#define SWIFT_INTERFACE_FORMAT_VERSION_KEY "swift-interface-format-version"
+version::Version swift::InterfaceFormatVersion({1, 0});
-#define SWIFT_TOOLS_VERSION_KEY "swift-tools-version"
-#define SWIFT_MODULE_FLAGS_KEY "swift-module-flags"
-
-static swift::version::Version InterfaceFormatVersion({1, 0});
-
-static bool
-extractSwiftInterfaceVersionAndArgs(DiagnosticEngine &Diags, SourceLoc DiagLoc,
-                                    llvm::vfs::FileSystem &FS,
-                                    StringRef SwiftInterfacePathIn,
-                                    swift::version::Version &Vers,
-                                    llvm::StringSaver &SubArgSaver,
-                                    SmallVectorImpl<const char *> &SubArgs) {
-  auto FileOrError = swift::vfs::getFileOrSTDIN(FS, SwiftInterfacePathIn);
-  if (!FileOrError) {
-    Diags.diagnose(DiagLoc, diag::error_open_input_file,
-                   SwiftInterfacePathIn, FileOrError.getError().message());
-    return true;
-  }
-  auto SB = FileOrError.get()->getBuffer();
-  auto VersRe = getSwiftInterfaceFormatVersionRegex();
-  auto FlagRe = getSwiftInterfaceModuleFlagsRegex();
-  SmallVector<StringRef, 1> VersMatches, FlagMatches;
-  if (!VersRe.match(SB, &VersMatches)) {
-    Diags.diagnose(DiagLoc,
-                   diag::error_extracting_version_from_parseable_interface);
-    return true;
-  }
-  if (!FlagRe.match(SB, &FlagMatches)) {
-    Diags.diagnose(DiagLoc,
-                   diag::error_extracting_flags_from_parseable_interface);
-    return true;
-  }
-  assert(VersMatches.size() == 2);
-  assert(FlagMatches.size() == 2);
-  Vers = swift::version::Version(VersMatches[1], SourceLoc(), &Diags);
-  llvm::cl::TokenizeGNUCommandLine(FlagMatches[1], SubArgSaver, SubArgs);
-  return false;
-}
-
-static std::unique_ptr<llvm::MemoryBuffer>
-getBufferOfDependency(llvm::vfs::FileSystem &FS,
-                      StringRef ModulePath, StringRef DepPath,
-                      DiagnosticEngine &Diags, SourceLoc DiagLoc) {
-  auto DepBuf = FS.getBufferForFile(DepPath, /*FileSize=*/-1,
-                                    /*RequiresNullTerminator=*/false);
-  if (!DepBuf) {
-    Diags.diagnose(DiagLoc,
-                   diag::missing_dependency_of_parseable_module_interface,
-                   DepPath, ModulePath, DepBuf.getError().message());
-    return nullptr;
-  }
-  return std::move(DepBuf.get());
-}
-
-static Optional<llvm::vfs::Status>
-getStatusOfDependency(llvm::vfs::FileSystem &FS,
-                      StringRef ModulePath, StringRef DepPath,
-                      DiagnosticEngine &Diags, SourceLoc DiagLoc) {
-  auto Status = FS.status(DepPath);
-  if (!Status) {
-    Diags.diagnose(DiagLoc,
-                   diag::missing_dependency_of_parseable_module_interface,
-                   DepPath, ModulePath, Status.getError().message());
-    return None;
-  }
-  return Status.get();
-}
-
-/// Construct a cache key for the .swiftmodule being generated. There is a
-/// balance to be struck here between things that go in the cache key and
-/// things that go in the "up to date" check of the cache entry. We want to
-/// avoid fighting over a single cache entry too much when (say) running
-/// different compiler versions on the same machine or different inputs
-/// that happen to have the same short module name, so we will disambiguate
-/// those in the key. But we want to invalidate and rebuild a cache entry
-/// -- rather than making a new one and potentially filling up the cache
-/// with dead entries -- when other factors change, such as the contents of
-/// the .swiftinterface input or its dependencies.
-static std::string getCacheHash(ASTContext &Ctx,
-                                const CompilerInvocation &SubInvocation,
-                                StringRef InPath) {
-  // Start with the compiler version (which will be either tag names or revs).
-  // Explicitly don't pass in the "effective" language version -- this would
-  // mean modules built in different -swift-version modes would rebuild their
-  // dependencies.
-  llvm::hash_code H = hash_value(swift::version::getSwiftFullVersion());
-
-  // Simplest representation of input "identity" (not content) is just a
-  // pathname, and probably all we can get from the VFS in this regard anyways.
-  H = hash_combine(H, InPath);
-
-  // Include the target CPU architecture. In practice, .swiftinterface files
-  // will be in architecture-specific subdirectories and would have
-  // architecture-specific pieces #if'd out. However, it doesn't hurt to
-  // include it, and it guards against mistakenly reusing cached modules across
-  // architectures.
-  H = hash_combine(H, SubInvocation.getLangOptions().Target.getArchName());
-
-  // The SDK path is going to affect how this module is imported, so include it.
-  H = hash_combine(H, SubInvocation.getSDKPath());
-
-  return llvm::APInt(64, H).toString(36, /*Signed=*/false);
-}
-
-static CompilerInvocation
-createInvocationForBuildingFromInterface(ASTContext &Ctx, StringRef ModuleName,
-                                         StringRef CacheDir,
-                                         StringRef PrebuiltCacheDir) {
-  auto &SearchPathOpts = Ctx.SearchPathOpts;
-  auto &LangOpts = Ctx.LangOpts;
-
-  CompilerInvocation SubInvocation;
-
-  // Start with a SubInvocation that copies various state from our
-  // invoking ASTContext.
-  SubInvocation.setImportSearchPaths(SearchPathOpts.ImportSearchPaths);
-  SubInvocation.setFrameworkSearchPaths(SearchPathOpts.FrameworkSearchPaths);
-  SubInvocation.setSDKPath(SearchPathOpts.SDKPath);
-  SubInvocation.setInputKind(InputFileKind::SwiftModuleInterface);
-  SubInvocation.setRuntimeResourcePath(SearchPathOpts.RuntimeResourcePath);
-  SubInvocation.setTargetTriple(LangOpts.Target);
-
-  SubInvocation.setModuleName(ModuleName);
-  SubInvocation.setClangModuleCachePath(CacheDir);
-  SubInvocation.getFrontendOptions().PrebuiltModuleCachePath = PrebuiltCacheDir;
-
-  // Respect the detailed-record preprocessor setting of the parent context.
-  // This, and the "raw" clang module format it implicitly enables, are required
-  // by sourcekitd.
-  if (auto *ClangLoader = Ctx.getClangModuleLoader()) {
-    auto &Opts = ClangLoader->getClangInstance().getPreprocessorOpts();
-    if (Opts.DetailedRecord) {
-      SubInvocation.getClangImporterOptions().DetailedPreprocessingRecord = true;
-    }
-  }
-
-  // Inhibit warnings from the SubInvocation since we are assuming the user
-  // is not in a position to fix them.
-  SubInvocation.getDiagnosticOptions().SuppressWarnings = true;
-
-  // Inherit this setting down so that it can affect error diagnostics (mostly
-  // by making them non-fatal).
-  SubInvocation.getLangOptions().DebuggerSupport = LangOpts.DebuggerSupport;
-
-  // Disable this; deinitializers always get printed with `@objc` even in
-  // modules that don't import Foundation.
-  SubInvocation.getLangOptions().EnableObjCAttrRequiresFoundation = false;
-
-  return SubInvocation;
-}
-
-/// Calculate an output filename in \p SubInvocation's cache path that includes
-/// a hash of relevant key data.
-static void computeCachedOutputPath(ASTContext &Ctx,
-                                    const CompilerInvocation &SubInvocation,
-                                    StringRef InPath,
-                                    llvm::SmallString<256> &OutPath) {
-  OutPath = SubInvocation.getClangModuleCachePath();
-  llvm::sys::path::append(OutPath, SubInvocation.getModuleName());
-  OutPath.append("-");
-  OutPath.append(getCacheHash(Ctx, SubInvocation, InPath));
-  OutPath.append(".");
-  auto OutExt = file_types::getExtension(file_types::TY_SwiftModuleFile);
-  OutPath.append(OutExt);
-}
-
-void ParseableInterfaceModuleLoader::configureSubInvocationInputsAndOutputs(
-    CompilerInvocation &SubInvocation, StringRef InPath, StringRef OutPath) {
-  auto &SubFEOpts = SubInvocation.getFrontendOptions();
-  SubFEOpts.RequestedAction = FrontendOptions::ActionType::EmitModuleOnly;
-  SubFEOpts.EnableParseableModuleInterface = true;
-  SubFEOpts.InputsAndOutputs.addPrimaryInputFile(InPath);
-  SupplementaryOutputPaths SOPs;
-  SOPs.ModuleOutputPath = OutPath.str();
-
-  // Pick a primary output path that will cause problems to use.
-  StringRef MainOut = "/<unused>";
-  SubFEOpts.InputsAndOutputs.setMainAndSupplementaryOutputs({MainOut}, {SOPs});
-}
-
-// Checks that a dependency read from the cached module is up to date compared
-// to the interface file it represents. If it's up-to-date, return its status,
-// so we can switch hash-based dependencies to mtimes after we've validated
-// the hashes.
-static Optional<llvm::vfs::Status>
-dependencyIsUpToDate(llvm::vfs::FileSystem &FS, FileDependency In,
-                     StringRef ModulePath, DiagnosticEngine &Diags,
-                     SourceLoc DiagLoc) {
-  auto Status = getStatusOfDependency(FS, ModulePath, In.getPath(),
-                                      Diags, DiagLoc);
-  if (!Status) return None;
-
-  // If the sizes differ, then we know the file has changed.
-  if (Status->getSize() != In.getSize()) return None;
-
-  // Otherwise, if this dependency is verified by modification time, check
-  // it vs. the modification time of the file.
-  uint64_t mtime =
-    Status->getLastModificationTime().time_since_epoch().count();
-
-  if (In.isModificationTimeBased())
-    return mtime == In.getModificationTime() ? Status : None;
-
-  // Slow path: if the dependency is verified by content hash, check it vs. the
-  // hash of the file.
-  auto buf = getBufferOfDependency(FS, ModulePath, In.getPath(),
-                                   Diags, DiagLoc);
-  if (!buf) return None;
-
-  if (xxHash64(buf->getBuffer()) == In.getContentHash())
-    return Status;
-
-  return None;
-}
-
-}
-
-// Check that the output .swiftmodule file is at least as new as all the
-// dependencies it read when it was built last time.
-static bool
-swiftModuleIsUpToDate(llvm::vfs::FileSystem &FS,
-                      std::pair<Identifier, SourceLoc> ModuleID,
-                      StringRef OutPath,
-                      DiagnosticEngine &Diags,
-                      DependencyTracker *OuterTracker) {
-
-  auto OutBuf = FS.getBufferForFile(OutPath);
-  if (!OutBuf)
-    return false;
-
-  LLVM_DEBUG(llvm::dbgs() << "Validating deps of " << OutPath << "\n");
-  SmallVector<FileDependency, 16> AllDeps;
-  auto VI = serialization::validateSerializedAST(
-      OutBuf.get()->getBuffer(),
-      /*ExtendedValidationInfo=*/nullptr, &AllDeps);
-
-  if (VI.status != serialization::Status::Valid)
-    return false;
-
-  assert(VI.name == ModuleID.first.str() &&
-         "we built a module at this path with a different name?");
-
-  for (auto In : AllDeps) {
-    if (OuterTracker)
-      OuterTracker->addDependency(In.Path, /*IsSystem=*/false);
-    if (!dependencyIsUpToDate(FS, In, OutPath, Diags, ModuleID.second)) {
-      LLVM_DEBUG(llvm::dbgs() << "Dep " << In.Path
-                 << " is directly out of date\n");
-      return false;
-    }
-    LLVM_DEBUG(llvm::dbgs() << "Dep " << In.Path << " is up to date\n");
-  }
-  return true;
-}
-
-/// Populate the provided \p Deps with \c FileDependency entries including:
-///
-///    - \p InPath - The .swiftinterface input file
-///
-///    - All the dependencies mentioned by \p SubInstance's DependencyTracker,
-///      that were read while compiling the module.
-///
-///    - For any file in the latter set that is itself a .swiftmodule
-///      living in \p ModuleCachePath, all of _its_ dependencies, copied
-///      out to avoid having to do recursive scanning when rechecking this
-///      dependency in the future.
-static bool
-collectDepsForSerialization(llvm::vfs::FileSystem &FS,
-                            CompilerInstance &SubInstance,
-                            StringRef InPath, StringRef ModuleCachePath,
-                            SmallVectorImpl<FileDependency> &Deps,
-                            bool IsHashBased,
-                            DiagnosticEngine &Diags, SourceLoc DiagLoc,
-                            DependencyTracker *OuterTracker) {
-  auto DTDeps = SubInstance.getDependencyTracker()->getDependencies();
-  SmallVector<StringRef, 16> InitialDepNames(DTDeps.begin(), DTDeps.end());
-  InitialDepNames.push_back(InPath);
-  llvm::StringSet<> AllDepNames;
-  for (auto const &DepName : InitialDepNames) {
-    if (AllDepNames.insert(DepName).second && OuterTracker) {
-      OuterTracker->addDependency(DepName, /*IsSystem=*/false);
-    }
-    auto DepBuf = getBufferOfDependency(FS, InPath, DepName, Diags, DiagLoc);
-    if (!DepBuf)
-      return true;
-    auto Status = getStatusOfDependency(FS, InPath, DepName, Diags, DiagLoc);
-    if (!Status)
-      return true;
-
-    if (IsHashBased) {
-      uint64_t hash = xxHash64(DepBuf->getBuffer());
-      Deps.push_back(
-        FileDependency::hashBased(DepName, Status->getSize(), hash));
-    } else {
-      uint64_t mtime =
-        Status->getLastModificationTime().time_since_epoch().count();
-      Deps.push_back(
-        FileDependency::modTimeBased(DepName, Status->getSize(), mtime));
-    }
-
-    if (ModuleCachePath.empty())
-      continue;
-
-    // If Dep is itself a .swiftmodule in the cache dir, pull out its deps
-    // and include them in our own, so we have a single-file view of
-    // transitive deps: removes redundancies, and avoids opening and reading
-    // multiple swiftmodules during future loads.
-    auto Ext = llvm::sys::path::extension(DepName);
-    auto Ty = file_types::lookupTypeForExtension(Ext);
-    if (Ty == file_types::TY_SwiftModuleFile &&
-        DepName.startswith(ModuleCachePath)) {
-      SmallVector<FileDependency, 16> SubDeps;
-      auto VI = serialization::validateSerializedAST(
-          DepBuf->getBuffer(),
-          /*ExtendedValidationInfo=*/nullptr, &SubDeps);
-      if (VI.status != serialization::Status::Valid) {
-        Diags.diagnose(DiagLoc,
-                       diag::error_extracting_dependencies_from_cached_module,
-                       DepName);
-        return true;
-      }
-      for (auto const &SubDep : SubDeps) {
-        if (AllDepNames.insert(SubDep.getPath()).second) {
-          Deps.push_back(SubDep);
-          if (OuterTracker)
-            OuterTracker->addDependency(SubDep.getPath(), /*IsSystem=*/false);
-        }
-      }
-    }
-  }
-  return false;
-}
-
-bool ParseableInterfaceModuleLoader::buildSwiftModuleFromSwiftInterface(
-    llvm::vfs::FileSystem &FS, DiagnosticEngine &Diags, SourceLoc DiagLoc,
-    CompilerInvocation &SubInvocation, StringRef InPath, StringRef OutPath,
-    StringRef ModuleCachePath, DependencyTracker *OuterTracker,
-    bool ShouldSerializeDeps) {
-  bool SubError = false;
-  bool RunSuccess = llvm::CrashRecoveryContext().RunSafelyOnThread([&] {
-    // Note that we don't assume ModuleCachePath is the same as the Clang
-    // module cache path at this point.
-    if (!ModuleCachePath.empty())
-      (void)llvm::sys::fs::create_directory(ModuleCachePath);
-
-    configureSubInvocationInputsAndOutputs(SubInvocation, InPath, OutPath);
-
-    FrontendOptions &FEOpts = SubInvocation.getFrontendOptions();
-    const auto &InputInfo = FEOpts.InputsAndOutputs.firstInput();
-    StringRef InPath = InputInfo.file();
-    const auto &OutputInfo =
-        InputInfo.getPrimarySpecificPaths().SupplementaryOutputs;
-    StringRef OutPath = OutputInfo.ModuleOutputPath;
-
-    llvm::BumpPtrAllocator SubArgsAlloc;
-    llvm::StringSaver SubArgSaver(SubArgsAlloc);
-    SmallVector<const char *, 16> SubArgs;
-    swift::version::Version Vers;
-    if (extractSwiftInterfaceVersionAndArgs(Diags, DiagLoc, FS, InPath, Vers,
-                                            SubArgSaver, SubArgs)) {
-      SubError = true;
-      return;
-    }
-
-    // For now: we support anything with the same "major version" and assume
-    // minor versions might be interesting for debugging, or special-casing a
-    // compatible field variant.
-    if (Vers.asMajorVersion() != InterfaceFormatVersion.asMajorVersion()) {
-      Diags.diagnose(DiagLoc,
-                     diag::unsupported_version_of_parseable_interface,
-                     InPath, Vers);
-      SubError = true;
-      return;
-    }
-
-    SmallString<32> ExpectedModuleName = SubInvocation.getModuleName();
-    if (SubInvocation.parseArgs(SubArgs, Diags)) {
-      SubError = true;
-      return;
-    }
-
-    if (SubInvocation.getModuleName() != ExpectedModuleName) {
-      auto DiagKind = diag::serialization_name_mismatch;
-      if (SubInvocation.getLangOptions().DebuggerSupport)
-        DiagKind = diag::serialization_name_mismatch_repl;
-      Diags.diagnose(DiagLoc, DiagKind, SubInvocation.getModuleName(),
-                     ExpectedModuleName);
-      SubError = true;
-      return;
-    }
-
-    // Optimize emitted modules. This has to happen after we parse arguments,
-    // because parseSILOpts would override the current optimization mode.
-    SubInvocation.getSILOptions().OptMode = OptimizationMode::ForSpeed;
-
-    // Build the .swiftmodule; this is a _very_ abridged version of the logic in
-    // performCompile in libFrontendTool, specialized, to just the one
-    // module-serialization task we're trying to do here.
-    LLVM_DEBUG(llvm::dbgs() << "Setting up instance to compile "
-               << InPath << " to " << OutPath << "\n");
-    CompilerInstance SubInstance;
-    SubInstance.getSourceMgr().setFileSystem(&FS);
-
-    ForwardingDiagnosticConsumer FDC(Diags);
-    SubInstance.addDiagnosticConsumer(&FDC);
-
-    SubInstance.createDependencyTracker(/*TrackSystemDeps=*/false);
-    if (SubInstance.setup(SubInvocation)) {
-      SubError = true;
-      return;
-    }
-
-    LLVM_DEBUG(llvm::dbgs() << "Performing sema\n");
-    SubInstance.performSema();
-    if (SubInstance.getASTContext().hadError()) {
-      LLVM_DEBUG(llvm::dbgs() << "encountered errors\n");
-      SubError = true;
-      return;
-    }
-
-    SILOptions &SILOpts = SubInvocation.getSILOptions();
-    auto Mod = SubInstance.getMainModule();
-    auto SILMod = performSILGeneration(Mod, SILOpts);
-    if (!SILMod) {
-      LLVM_DEBUG(llvm::dbgs() << "SILGen did not produce a module\n");
-      SubError = true;
-      return;
-    }
-
-    // Setup the callbacks for serialization, which can occur during the
-    // optimization pipeline.
-    SerializationOptions SerializationOpts;
-    std::string OutPathStr = OutPath;
-    SerializationOpts.OutputPath = OutPathStr.c_str();
-    SerializationOpts.ModuleLinkName = FEOpts.ModuleLinkName;
-    SmallVector<FileDependency, 16> Deps;
-    if (collectDepsForSerialization(
-          FS, SubInstance, InPath, ModuleCachePath, Deps,
-          FEOpts.SerializeParseableModuleInterfaceDependencyHashes,
-          Diags, DiagLoc, OuterTracker)) {
-      SubError = true;
-      return;
-    }
-    if (ShouldSerializeDeps)
-      SerializationOpts.Dependencies = Deps;
-    SILMod->setSerializeSILAction([&]() {
-      serialize(Mod, SerializationOpts, SILMod.get());
-    });
-
-    LLVM_DEBUG(llvm::dbgs() << "Running SIL processing passes\n");
-    if (SubInstance.performSILProcessing(SILMod.get())) {
-      LLVM_DEBUG(llvm::dbgs() << "encountered errors\n");
-      SubError = true;
-      return;
-    }
-
-    SubError = SubInstance.getDiags().hadAnyError();
-  });
-  return !RunSuccess || SubError;
-}
-
-static bool serializedASTLooksValidOrCannotBeRead(llvm::vfs::FileSystem &FS,
-                                                  StringRef ModPath) {
-  auto ModBuf = FS.getBufferForFile(ModPath, /*FileSize=*/-1,
-                                    /*RequiresNullTerminator=*/false);
-  if (!ModBuf)
-    return ModBuf.getError() != std::errc::no_such_file_or_directory;
-
-  auto VI = serialization::validateSerializedAST(ModBuf.get()->getBuffer());
-  return VI.status == serialization::Status::Valid;
-}
-
-/// Load a .swiftmodule associated with a .swiftinterface either from a
-/// cache or by converting it in a subordinate \c CompilerInstance, caching
-/// the results.
-std::error_code ParseableInterfaceModuleLoader::findModuleFilesInDirectory(
-    AccessPathElem ModuleID, StringRef DirPath, StringRef ModuleFilename,
-    StringRef ModuleDocFilename,
-    std::unique_ptr<llvm::MemoryBuffer> *ModuleBuffer,
-    std::unique_ptr<llvm::MemoryBuffer> *ModuleDocBuffer) {
-
-  namespace path = llvm::sys::path;
-
-  // If running in OnlySerialized mode, ParseableInterfaceModuleLoader
-  // should not have been constructed at all.
-  assert(LoadMode != ModuleLoadingMode::OnlySerialized);
-
-  auto &FS = *Ctx.SourceMgr.getFileSystem();
-  auto &Diags = Ctx.Diags;
-  llvm::SmallString<256> ModPath, InPath, OutPath;
-
-  // First check to see if the .swiftinterface exists at all. Bail if not.
-  ModPath = DirPath;
-  path::append(ModPath, ModuleFilename);
-
-  auto Ext = file_types::getExtension(file_types::TY_SwiftParseableInterfaceFile);
-  InPath = ModPath;
-  path::replace_extension(InPath, Ext);
-  if (!FS.exists(InPath))
-    return std::make_error_code(std::errc::no_such_file_or_directory);
-
-  // Next, if we're in the load mode that prefers .swiftmodules, see if there's
-  // one here we can _likely_ load (validates OK). If so, bail early with
-  // errc::not_supported, so the next (serialized) loader in the chain will load
-  // it. Alternately, if there's a .swiftmodule present but we can't even read
-  // it (for whatever reason), we should let the other module loader diagnose
-  // it.
-  if (LoadMode == ModuleLoadingMode::PreferSerialized &&
-      serializedASTLooksValidOrCannotBeRead(FS, ModPath)) {
-    return std::make_error_code(std::errc::not_supported);
-  }
-
-  // If we have a prebuilt cache path, check that too if the interface comes
-  // from the SDK.
-  if (!PrebuiltCacheDir.empty()) {
-    StringRef SDKPath = Ctx.SearchPathOpts.SDKPath;
-    if (!SDKPath.empty() && hasPrefix(path::begin(InPath),
-                                      path::end(InPath),
-                                      path::begin(SDKPath),
-                                      path::end(SDKPath))) {
-      // Assemble the expected path: $PREBUILT_CACHE/Foo.swiftmodule or
-      // $PREBUILT_CACHE/Foo.swiftmodule/arch.swiftmodule. Note that there's no
-      // cache key here.
-      OutPath = PrebuiltCacheDir;
-
-      // FIXME: Would it be possible to only have architecture-specific names
-      // here? Then we could skip this check.
-      StringRef InParentDirName = path::filename(path::parent_path(InPath));
-      if (path::extension(InParentDirName) == ".swiftmodule") {
-        assert(path::stem(InParentDirName) == ModuleID.first.str());
-        path::append(OutPath, InParentDirName);
-      }
-      path::append(OutPath, ModuleFilename);
-
-      if (!swiftModuleIsUpToDate(FS, ModuleID, OutPath, Diags,
-                                 dependencyTracker)) {
-        OutPath.clear();
-      }
-    }
-  }
-
-  if (OutPath.empty()) {
-    // At this point we're either in PreferParseable mode or there's no credible
-    // adjacent .swiftmodule so we'll go ahead and start trying to convert the
-    // .swiftinterface.
-
-    // Set up a _potential_ sub-invocation to consume the .swiftinterface and
-    // emit the .swiftmodule.
-    CompilerInvocation SubInvocation =
-        createInvocationForBuildingFromInterface(Ctx, ModuleID.first.str(),
-                                                 CacheDir, PrebuiltCacheDir);
-    computeCachedOutputPath(Ctx, SubInvocation, InPath, OutPath);
-
-    // Evaluate if we need to run this sub-invocation, and if so run it.
-    if (!swiftModuleIsUpToDate(FS, ModuleID, OutPath, Diags,
-                               dependencyTracker)) {
-      if (buildSwiftModuleFromSwiftInterface(FS, Diags, ModuleID.second,
-                                             SubInvocation, InPath, OutPath,
-                                             CacheDir, dependencyTracker,
-                                             /*ShouldSerializeDeps*/true))
-        return std::make_error_code(std::errc::invalid_argument);
-    }
-  }
-
-  // Finish off by delegating back up to the SerializedModuleLoaderBase
-  // routine that can load the recently-manufactured serialized module.
-  LLVM_DEBUG(llvm::dbgs() << "Loading " << OutPath
-             << " via normal module loader\n");
-  llvm::SmallString<256> DocPath{DirPath};
-  path::append(DocPath, ModuleDocFilename);
-  auto ErrorCode = SerializedModuleLoaderBase::openModuleFiles(
-      ModuleID, OutPath, DocPath, ModuleBuffer, ModuleDocBuffer);
-  LLVM_DEBUG(llvm::dbgs() << "Loaded " << OutPath
-             << " via normal module loader");
-  if (ErrorCode) {
-    LLVM_DEBUG(llvm::dbgs() << " with error: " << ErrorCode.message());
-  }
-  LLVM_DEBUG(llvm::dbgs() << "\n");
-  return ErrorCode;
-}
-
-bool
-ParseableInterfaceModuleLoader::buildSwiftModuleFromSwiftInterface(
-    ASTContext &Ctx, StringRef CacheDir, StringRef PrebuiltCacheDir,
-    StringRef ModuleName, StringRef InPath, StringRef OutPath) {
-  CompilerInvocation SubInvocation =
-      createInvocationForBuildingFromInterface(Ctx, ModuleName, CacheDir,
-                                               PrebuiltCacheDir);
-
-  auto &FS = *Ctx.SourceMgr.getFileSystem();
-  auto &Diags = Ctx.Diags;
-  // FIXME: We don't really want to ignore dependencies here, but we have to
-  // identify which ones are important, and make them relocatable
-  // (SDK-relative) if we want to ship the built swiftmodules to another
-  // machine. Just leave them out for now.
-  return buildSwiftModuleFromSwiftInterface(FS, Diags, /*DiagLoc*/SourceLoc(),
-                                            SubInvocation, InPath, OutPath,
-                                            /*CachePath*/"",
-                                            /*OuterTracker*/nullptr,
-                                            /*ShouldSerializeDeps*/false);
-}
 
 /// Diagnose any scoped imports in \p imports, i.e. those with a non-empty
 /// access path. These are not yet supported by parseable interfaces, since the
@@ -690,23 +81,6 @@ llvm::Regex swift::getSwiftInterfaceModuleFlagsRegex() {
                      llvm::Regex::Newline);
 }
 
-/// Extract the specified-or-defaulted -module-cache-path that winds up in
-/// the clang importer, for reuse as the .swiftmodule cache path when
-/// building a ParseableInterfaceModuleLoader.
-std::string
-swift::getModuleCachePathFromClang(const clang::CompilerInstance &Clang) {
-  if (!Clang.hasPreprocessor())
-    return "";
-  std::string SpecificModuleCachePath = Clang.getPreprocessor()
-    .getHeaderSearchInfo()
-    .getModuleCachePath();
-
-  // The returned-from-clang module cache path includes a suffix directory
-  // that is specific to the clang version and invocation; we want the
-  // directory above that.
-  return llvm::sys::path::parent_path(SpecificModuleCachePath);
-}
-
 /// Prints the imported modules in \p M to \p out in the form of \c import
 /// source declarations.
 static void printImports(raw_ostream &out, ModuleDecl *M) {

lib/FrontendTool/FrontendTool.cpp

@@ -51,6 +51,7 @@
 #include "swift/Frontend/Frontend.h"
 #include "swift/Frontend/PrintingDiagnosticConsumer.h"
 #include "swift/Frontend/SerializedDiagnosticConsumer.h"
+#include "swift/Frontend/ParseableInterfaceModuleLoader.h"
 #include "swift/Frontend/ParseableInterfaceSupport.h"
 #include "swift/Immediate/Immediate.h"
 #include "swift/Index/IndexRecord.h"
@@ -585,7 +586,8 @@ static bool buildModuleFromParseableInterface(CompilerInvocation &Invocation,
   return ParseableInterfaceModuleLoader::buildSwiftModuleFromSwiftInterface(
       Instance.getASTContext(), Invocation.getClangModuleCachePath(),
       PrebuiltCachePath, Invocation.getModuleName(), InputPath,
-      Invocation.getOutputFilename());
+      Invocation.getOutputFilename(),
+      FEOpts.SerializeParseableModuleInterfaceDependencyHashes);
 }
 
 static bool compileLLVMIR(CompilerInvocation &Invocation,

test/ParseableInterface/ModuleCache/Inputs/check-is-forwarding-module.py

@@ -0,0 +1,7 @@
+import sys
+
+with open(sys.argv[1], 'r') as yaml_file:
+    # Forwarding files are YAML files that start with '---'
+    if yaml_file.read(3) != '---':
+        print("swiftmodule '%s' is not a forwarding module!")
+        sys.exit(1)

test/ParseableInterface/ModuleCache/force-module-loading-mode-archs.swift

@@ -22,20 +22,29 @@
 // RUN: %empty-directory(%t/Lib.swiftmodule)
 // RUN: sed -e 's/FromInterface/FromSerialized/g' %S/Inputs/force-module-loading-mode/Lib.swiftinterface | %target-swift-frontend -parse-stdlib -module-cache-path %t/MCP -emit-module-path %t/Lib.swiftmodule/%target-swiftmodule-name - -module-name Lib
 // RUN: not env SWIFT_FORCE_MODULE_LOADING=prefer-parseable %target-swift-frontend -typecheck -parse-stdlib -module-cache-path %t/MCP %s -I %t 2>&1 | %FileCheck -check-prefix=FROM-SERIALIZED %s
+// RUN: %empty-directory(%t/MCP)
 // RUN: not env SWIFT_FORCE_MODULE_LOADING=prefer-serialized %target-swift-frontend -typecheck -parse-stdlib -module-cache-path %t/MCP %s -I %t 2>&1 | %FileCheck -check-prefix=FROM-SERIALIZED %s
+// RUN: %empty-directory(%t/MCP)
 // RUN: not env SWIFT_FORCE_MODULE_LOADING=only-parseable %target-swift-frontend -typecheck -parse-stdlib -module-cache-path %t/MCP %s -I %t 2>&1 | %FileCheck -check-prefix=NO-SUCH-MODULE %s
+// RUN: %empty-directory(%t/MCP)
 // RUN: not env SWIFT_FORCE_MODULE_LOADING=only-serialized %target-swift-frontend -typecheck -parse-stdlib -module-cache-path %t/MCP %s -I %t 2>&1 | %FileCheck -check-prefix=FROM-SERIALIZED %s
+// RUN: %empty-directory(%t/MCP)
 // (default)
 // RUN: not %target-swift-frontend -typecheck -parse-stdlib -module-cache-path %t/MCP -I %t %s 2>&1 | %FileCheck -check-prefix=FROM-SERIALIZED %s
+// RUN: %empty-directory(%t/MCP)
 
 // 4. Both are present.
 // RUN: cp %S/Inputs/force-module-loading-mode/Lib.swiftinterface %t/Lib.swiftmodule/%target-cpu.swiftinterface
 // RUN: not env SWIFT_FORCE_MODULE_LOADING=prefer-parseable %target-swift-frontend -typecheck -parse-stdlib -module-cache-path %t/MCP %s -I %t 2>&1 | %FileCheck -check-prefix=FROM-INTERFACE %s
+// RUN: %empty-directory(%t/MCP)
 // RUN: not env SWIFT_FORCE_MODULE_LOADING=prefer-serialized %target-swift-frontend -typecheck -parse-stdlib -module-cache-path %t/MCP %s -I %t 2>&1 | %FileCheck -check-prefix=FROM-SERIALIZED %s
+// RUN: %empty-directory(%t/MCP)
 // RUN: not env SWIFT_FORCE_MODULE_LOADING=only-parseable %target-swift-frontend -typecheck -parse-stdlib -module-cache-path %t/MCP %s -I %t 2>&1 | %FileCheck -check-prefix=FROM-INTERFACE %s
+// RUN: %empty-directory(%t/MCP)
 // RUN: not env SWIFT_FORCE_MODULE_LOADING=only-serialized %target-swift-frontend -typecheck -parse-stdlib -module-cache-path %t/MCP %s -I %t 2>&1 | %FileCheck -check-prefix=FROM-SERIALIZED %s
 // (default)
 // RUN: not %target-swift-frontend -typecheck -parse-stdlib -module-cache-path %t/MCP -I %t %s 2>&1 | %FileCheck -check-prefix=FROM-SERIALIZED %s
+// RUN: %empty-directory(%t/MCP)
 
 // 5. Both are present but the module is invalid.
 // RUN: rm %t/Lib.swiftmodule/%target-swiftmodule-name && touch %t/Lib.swiftmodule/%target-swiftmodule-name
@@ -49,11 +58,16 @@
 // 6. Both are present but the module can't be opened.
 // RUN: chmod a-r %t/Lib.swiftmodule/%target-swiftmodule-name
 // RUN: not env SWIFT_FORCE_MODULE_LOADING=prefer-parseable %target-swift-frontend -typecheck -parse-stdlib -module-cache-path %t/MCP %s -I %t 2>&1 | %FileCheck -check-prefix=FROM-INTERFACE %s
+// RUN: %empty-directory(%t/MCP)
 // RUN: not env SWIFT_FORCE_MODULE_LOADING=prefer-serialized %target-swift-frontend -typecheck -parse-stdlib -module-cache-path %t/MCP %s -I %t 2>&1 | %FileCheck -check-prefix=NO-SUCH-MODULE %s
+// RUN: %empty-directory(%t/MCP)
 // RUN: not env SWIFT_FORCE_MODULE_LOADING=only-parseable %target-swift-frontend -typecheck -parse-stdlib -module-cache-path %t/MCP %s -I %t 2>&1 | %FileCheck -check-prefix=FROM-INTERFACE %s
+// RUN: %empty-directory(%t/MCP)
 // RUN: not env SWIFT_FORCE_MODULE_LOADING=only-serialized %target-swift-frontend -typecheck -parse-stdlib -module-cache-path %t/MCP %s -I %t 2>&1 | %FileCheck -check-prefix=NO-SUCH-MODULE %s
+// RUN: %empty-directory(%t/MCP)
 // (default)
 // RUN: not %target-swift-frontend -typecheck -parse-stdlib -module-cache-path %t/MCP -I %t %s 2>&1 | %FileCheck -check-prefix=NO-SUCH-MODULE %s
+// RUN: %empty-directory(%t/MCP)
 
 // 7. Both are present but for the wrong architecture.
 // RUN: %empty-directory(%t/Lib.swiftmodule)

test/ParseableInterface/ModuleCache/force-module-loading-mode-framework.swift

@@ -31,29 +31,44 @@
 // 4. Both are present.
 // RUN: cp %S/Inputs/force-module-loading-mode/Lib.swiftinterface %t/Lib.framework/Modules/Lib.swiftmodule/%target-cpu.swiftinterface
 // RUN: not env SWIFT_FORCE_MODULE_LOADING=prefer-parseable %target-swift-frontend -typecheck -parse-stdlib -module-cache-path %t/MCP %s -F %t 2>&1 | %FileCheck -check-prefix=FROM-INTERFACE %s
+// RUN: %empty-directory(%t/MCP)
 // RUN: not env SWIFT_FORCE_MODULE_LOADING=prefer-serialized %target-swift-frontend -typecheck -parse-stdlib -module-cache-path %t/MCP %s -F %t 2>&1 | %FileCheck -check-prefix=FROM-SERIALIZED %s
+// RUN: %empty-directory(%t/MCP)
 // RUN: not env SWIFT_FORCE_MODULE_LOADING=only-parseable %target-swift-frontend -typecheck -parse-stdlib -module-cache-path %t/MCP %s -F %t 2>&1 | %FileCheck -check-prefix=FROM-INTERFACE %s
+// RUN: %empty-directory(%t/MCP)
 // RUN: not env SWIFT_FORCE_MODULE_LOADING=only-serialized %target-swift-frontend -typecheck -parse-stdlib -module-cache-path %t/MCP %s -F %t 2>&1 | %FileCheck -check-prefix=FROM-SERIALIZED %s
+// RUN: %empty-directory(%t/MCP)
 // (default)
 // RUN: not %target-swift-frontend -typecheck -parse-stdlib -module-cache-path %t/MCP -F %t %s 2>&1 | %FileCheck -check-prefix=FROM-SERIALIZED %s
+// RUN: %empty-directory(%t/MCP)
 
 // 5. Both are present but the module is invalid.
 // RUN: rm %t/Lib.framework/Modules/Lib.swiftmodule/%target-swiftmodule-name && touch %t/Lib.framework/Modules/Lib.swiftmodule/%target-swiftmodule-name
 // RUN: not env SWIFT_FORCE_MODULE_LOADING=prefer-parseable %target-swift-frontend -typecheck -parse-stdlib -module-cache-path %t/MCP %s -F %t 2>&1 | %FileCheck -check-prefix=FROM-INTERFACE %s
+// RUN: %empty-directory(%t/MCP)
 // RUN: not env SWIFT_FORCE_MODULE_LOADING=prefer-serialized %target-swift-frontend -typecheck -parse-stdlib -module-cache-path %t/MCP %s -F %t 2>&1 | %FileCheck -check-prefix=FROM-INTERFACE %s
+// RUN: %empty-directory(%t/MCP)
 // RUN: not env SWIFT_FORCE_MODULE_LOADING=only-parseable %target-swift-frontend -typecheck -parse-stdlib -module-cache-path %t/MCP %s -F %t 2>&1 | %FileCheck -check-prefix=FROM-INTERFACE %s
+// RUN: %empty-directory(%t/MCP)
 // RUN: not env SWIFT_FORCE_MODULE_LOADING=only-serialized %target-swift-frontend -typecheck -parse-stdlib -module-cache-path %t/MCP %s -F %t 2>&1 | %FileCheck -check-prefix=BAD-MODULE %s
+// RUN: %empty-directory(%t/MCP)
 // (default)
 // RUN: not %target-swift-frontend -typecheck -parse-stdlib -module-cache-path %t/MCP -F %t %s 2>&1 | %FileCheck -check-prefix=BAD-MODULE %s
+// RUN: %empty-directory(%t/MCP)
 
 // 6. Both are present but the module can't be opened.
 // RUN: chmod a-r %t/Lib.framework/Modules/Lib.swiftmodule/%target-swiftmodule-name
 // RUN: not env SWIFT_FORCE_MODULE_LOADING=prefer-parseable %target-swift-frontend -typecheck -parse-stdlib -module-cache-path %t/MCP %s -F %t 2>&1 | %FileCheck -check-prefix=FROM-INTERFACE %s
+// RUN: %empty-directory(%t/MCP)
 // RUN: not env SWIFT_FORCE_MODULE_LOADING=prefer-serialized %target-swift-frontend -typecheck -parse-stdlib -module-cache-path %t/MCP %s -F %t 2>&1 | %FileCheck -check-prefix=NO-SUCH-MODULE %s
+// RUN: %empty-directory(%t/MCP)
 // RUN: not env SWIFT_FORCE_MODULE_LOADING=only-parseable %target-swift-frontend -typecheck -parse-stdlib -module-cache-path %t/MCP %s -F %t 2>&1 | %FileCheck -check-prefix=FROM-INTERFACE %s
+// RUN: %empty-directory(%t/MCP)
 // RUN: not env SWIFT_FORCE_MODULE_LOADING=only-serialized %target-swift-frontend -typecheck -parse-stdlib -module-cache-path %t/MCP %s -F %t 2>&1 | %FileCheck -check-prefix=NO-SUCH-MODULE %s
+// RUN: %empty-directory(%t/MCP)
 // (default)
 // RUN: not %target-swift-frontend -typecheck -parse-stdlib -module-cache-path %t/MCP -F %t %s 2>&1 | %FileCheck -check-prefix=NO-SUCH-MODULE %s
+// RUN: %empty-directory(%t/MCP)
 
 // 7. Both are present but for the wrong architecture.
 // RUN: %empty-directory(%t/Lib.framework/Modules/Lib.swiftmodule)

test/ParseableInterface/ModuleCache/force-module-loading-mode.swift

@@ -28,29 +28,44 @@
 // 4. Both are present.
 // RUN: cp %S/Inputs/force-module-loading-mode/Lib.swiftinterface %t
 // RUN: not env SWIFT_FORCE_MODULE_LOADING=prefer-parseable %target-swift-frontend -typecheck -parse-stdlib -module-cache-path %t/MCP %s -I %t 2>&1 | %FileCheck -check-prefix=FROM-INTERFACE %s
+// RUN: %empty-directory(%t/MCP)
 // RUN: not env SWIFT_FORCE_MODULE_LOADING=prefer-serialized %target-swift-frontend -typecheck -parse-stdlib -module-cache-path %t/MCP %s -I %t 2>&1 | %FileCheck -check-prefix=FROM-SERIALIZED %s
+// RUN: %empty-directory(%t/MCP)
 // RUN: not env SWIFT_FORCE_MODULE_LOADING=only-parseable %target-swift-frontend -typecheck -parse-stdlib -module-cache-path %t/MCP %s -I %t 2>&1 | %FileCheck -check-prefix=FROM-INTERFACE %s
+// RUN: %empty-directory(%t/MCP)
 // RUN: not env SWIFT_FORCE_MODULE_LOADING=only-serialized %target-swift-frontend -typecheck -parse-stdlib -module-cache-path %t/MCP %s -I %t 2>&1 | %FileCheck -check-prefix=FROM-SERIALIZED %s
+// RUN: %empty-directory(%t/MCP)
 // (default)
 // RUN: not %target-swift-frontend -typecheck -parse-stdlib -module-cache-path %t/MCP -I %t %s 2>&1 | %FileCheck -check-prefix=FROM-SERIALIZED %s
+// RUN: %empty-directory(%t/MCP)
 
 // 5. Both are present but the module is invalid.
 // RUN: rm %t/Lib.swiftmodule && touch %t/Lib.swiftmodule
 // RUN: not env SWIFT_FORCE_MODULE_LOADING=prefer-parseable %target-swift-frontend -typecheck -parse-stdlib -module-cache-path %t/MCP %s -I %t 2>&1 | %FileCheck -check-prefix=FROM-INTERFACE %s
+// RUN: %empty-directory(%t/MCP)
 // RUN: not env SWIFT_FORCE_MODULE_LOADING=prefer-serialized %target-swift-frontend -typecheck -parse-stdlib -module-cache-path %t/MCP %s -I %t 2>&1 | %FileCheck -check-prefix=FROM-INTERFACE %s
+// RUN: %empty-directory(%t/MCP)
 // RUN: not env SWIFT_FORCE_MODULE_LOADING=only-parseable %target-swift-frontend -typecheck -parse-stdlib -module-cache-path %t/MCP %s -I %t 2>&1 | %FileCheck -check-prefix=FROM-INTERFACE %s
+// RUN: %empty-directory(%t/MCP)
 // RUN: not env SWIFT_FORCE_MODULE_LOADING=only-serialized %target-swift-frontend -typecheck -parse-stdlib -module-cache-path %t/MCP %s -I %t 2>&1 | %FileCheck -check-prefix=BAD-MODULE %s
+// RUN: %empty-directory(%t/MCP)
 // (default)
 // RUN: not %target-swift-frontend -typecheck -parse-stdlib -module-cache-path %t/MCP -I %t %s 2>&1 | %FileCheck -check-prefix=BAD-MODULE %s
+// RUN: %empty-directory(%t/MCP)
 
 // 6. Both are present but the module can't be opened.
 // RUN: chmod a-r %t/Lib.swiftmodule
 // RUN: not env SWIFT_FORCE_MODULE_LOADING=prefer-parseable %target-swift-frontend -typecheck -parse-stdlib -module-cache-path %t/MCP %s -I %t 2>&1 | %FileCheck -check-prefix=FROM-INTERFACE %s
+// RUN: %empty-directory(%t/MCP)
 // RUN: not env SWIFT_FORCE_MODULE_LOADING=prefer-serialized %target-swift-frontend -typecheck -parse-stdlib -module-cache-path %t/MCP %s -I %t 2>&1 | %FileCheck -check-prefix=NO-SUCH-MODULE %s
+// RUN: %empty-directory(%t/MCP)
 // RUN: not env SWIFT_FORCE_MODULE_LOADING=only-parseable %target-swift-frontend -typecheck -parse-stdlib -module-cache-path %t/MCP %s -I %t 2>&1 | %FileCheck -check-prefix=FROM-INTERFACE %s
+// RUN: %empty-directory(%t/MCP)
 // RUN: not env SWIFT_FORCE_MODULE_LOADING=only-serialized %target-swift-frontend -typecheck -parse-stdlib -module-cache-path %t/MCP %s -I %t 2>&1 | %FileCheck -check-prefix=NO-SUCH-MODULE %s
+// RUN: %empty-directory(%t/MCP)
 // (default)
 // RUN: not %target-swift-frontend -typecheck -parse-stdlib -module-cache-path %t/MCP -I %t %s 2>&1 | %FileCheck -check-prefix=NO-SUCH-MODULE %s
+// RUN: %empty-directory(%t/MCP)
 
 import Lib
 // NO-SUCH-MODULE: [[@LINE-1]]:8: error: no such module 'Lib'

test/ParseableInterface/ModuleCache/prebuilt-module-cache-archs.swift

@@ -16,7 +16,9 @@
 // RUN: %empty-directory(%t/prebuilt-cache/Lib.swiftmodule)
 // RUN: sed -e 's/FromInterface/FromPrebuilt/g' %S/Inputs/prebuilt-module-cache/Lib.swiftinterface | %target-swift-frontend -parse-stdlib -module-cache-path %t/MCP -emit-module-path %t/prebuilt-cache/Lib.swiftmodule/%target-swiftmodule-name - -module-name Lib
 // RUN: not %target-swift-frontend -typecheck -enable-parseable-module-interface -parse-stdlib -module-cache-path %t/MCP -sdk %t/include -I %t/include/ -prebuilt-module-cache-path %t/prebuilt-cache %s 2>&1 | %FileCheck -check-prefix=FROM-PREBUILT %s
-// RUN: ls %t/MCP | not grep swiftmodule
+
+// Make sure we installed a forwarding module.
+// RUN: %{python} %S/Inputs/check-is-forwarding-module.py %t/MCP/Lib-*.swiftmodule
 
 // What if the module is invalid?
 // RUN: rm %t/prebuilt-cache/Lib.swiftmodule/%target-swiftmodule-name && touch %t/prebuilt-cache/Lib.swiftmodule/%target-swiftmodule-name

test/ParseableInterface/ModuleCache/prebuilt-module-cache-forwarding.swift

@@ -0,0 +1,46 @@
+// RUN: %empty-directory(%t)
+// RUN: %empty-directory(%t/MCP)
+// RUN: %empty-directory(%t/prebuilt-cache)
+
+// First, prebuild a module and put it in the prebuilt cache.
+// RUN: sed -e 's/FromInterface/FromPrebuilt/g' %S/Inputs/prebuilt-module-cache/Lib.swiftinterface > %t/Lib.swiftinterface
+// RUN: %target-swift-frontend -build-module-from-parseable-interface -module-cache-path %t/MCP -serialize-parseable-module-interface-dependency-hashes -o %t/prebuilt-cache/Lib.swiftmodule %t/Lib.swiftinterface
+
+// Next, use the module and check if the forwarding module is in place.
+// RUN: not %target-swift-frontend -typecheck -enable-parseable-module-interface -parse-stdlib -module-cache-path %t/MCP -sdk %S/Inputs -I %S/Inputs/prebuilt-module-cache/ -prebuilt-module-cache-path %t/prebuilt-cache %s 2>&1 | %FileCheck -check-prefix=FROM-PREBUILT %s
+
+// Make sure we installed a forwarding module.
+// RUN: %{python} %S/Inputs/check-is-forwarding-module.py %t/MCP/Lib-*.swiftmodule
+
+// Now invalidate a dependency of the prebuilt module, and make sure the forwarding file is replaced with a real module.
+// RUN: touch %t/Lib.swiftinterface
+// RUN: not %target-swift-frontend -typecheck -enable-parseable-module-interface -parse-stdlib -module-cache-path %t/MCP -sdk %S/Inputs -I %S/Inputs/prebuilt-module-cache/ -prebuilt-module-cache-path %t/prebuilt-cache %s 2>&1 | %FileCheck -check-prefix=FROM-INTERFACE %s
+
+// Delete the cached module we just created, and create the forwarding module again
+// RUN: %empty-directory(%t/MCP)
+// RUN: not %target-swift-frontend -typecheck -enable-parseable-module-interface -parse-stdlib -module-cache-path %t/MCP -sdk %S/Inputs -I %S/Inputs/prebuilt-module-cache/ -prebuilt-module-cache-path %t/prebuilt-cache %s 2>&1 | %FileCheck -check-prefix=FROM-PREBUILT %s
+
+// Move the prebuilt module out of the way, so the forwarding module points to nothing.
+// RUN: mv %t/prebuilt-cache/Lib.swiftmodule %t/prebuilt-cache/NotLib.swiftmodule
+
+// Make sure we delete the existing forwarding module and rebuild from an interface.
+// RUN: not %target-swift-frontend -typecheck -enable-parseable-module-interface -parse-stdlib -module-cache-path %t/MCP -sdk %S/Inputs -I %S/Inputs/prebuilt-module-cache/ -prebuilt-module-cache-path %t/prebuilt-cache %s 2>&1 | %FileCheck -check-prefix=FROM-INTERFACE %s
+
+// Move the prebuilt module back to its original path
+// RUN: mv %t/prebuilt-cache/NotLib.swiftmodule %t/prebuilt-cache/Lib.swiftmodule
+
+// If the forwarding module is corrupted, we shouldn't rebuild the module in the cache,
+// we should delete it and generate a new forwarding module.
+// RUN: %empty-directory(%t/MCP)
+// RUN: not %target-swift-frontend -typecheck -enable-parseable-module-interface -parse-stdlib -module-cache-path %t/MCP -sdk %S/Inputs -I %S/Inputs/prebuilt-module-cache/ -prebuilt-module-cache-path %t/prebuilt-cache %s 2>&1
+// RUN: %{python} %S/Inputs/check-is-forwarding-module.py %t/MCP/Lib-*.swiftmodule
+// RUN: chmod a-r %t/MCP/Lib-*.swiftmodule
+// RUN: not %target-swift-frontend -typecheck -enable-parseable-module-interface -parse-stdlib -module-cache-path %t/MCP -sdk %S/Inputs -I %S/Inputs/prebuilt-module-cache/ -prebuilt-module-cache-path %t/prebuilt-cache %s 2>&1 | %FileCheck -check-prefix=FROM-PREBUILT %s
+// RUN: %{python} %S/Inputs/check-is-forwarding-module.py %t/MCP/Lib-*.swiftmodule
+
+import Lib
+
+struct X {}
+let _: X = Lib.testValue
+// FROM-INTERFACE: [[@LINE-1]]:16: error: cannot convert value of type 'FromInterface' to specified type 'X'
+// FROM-PREBUILT: [[@LINE-2]]:16: error: cannot convert value of type 'FromPrebuilt' to specified type 'X'

test/ParseableInterface/ModuleCache/prebuilt-module-cache-recursive.swift

@@ -9,7 +9,9 @@
 // RUN: %empty-directory(%t/MCP)
 // RUN: sed -e 's/FromInterface/FromPrebuilt/g' %S/Inputs/prebuilt-module-cache/Lib.swiftinterface | %target-swift-frontend -parse-stdlib -module-cache-path %t/MCP -emit-module-path %t/prebuilt-cache/Lib.swiftmodule - -module-name Lib
 // RUN: not %target-swift-frontend -typecheck -enable-parseable-module-interface -parse-stdlib -module-cache-path %t/MCP -sdk %S/Inputs -I %S/Inputs/prebuilt-module-cache/ -prebuilt-module-cache-path %t/prebuilt-cache %s 2>&1 | %FileCheck -check-prefix=FROM-PREBUILT %s
-// RUN: ls %t/MCP | grep -v LibExporter | not grep swiftmodule
+
+// Make sure we installed a forwarding module.
+// RUN: %{python} %S/Inputs/check-is-forwarding-module.py %t/MCP/Lib-*.swiftmodule
 
 // What if the module is invalid?
 // RUN: rm %t/prebuilt-cache/Lib.swiftmodule && touch %t/prebuilt-cache/Lib.swiftmodule

test/ParseableInterface/ModuleCache/prebuilt-module-cache.swift

@@ -13,7 +13,9 @@
 // RUN: %empty-directory(%t/MCP)
 // RUN: sed -e 's/FromInterface/FromPrebuilt/g' %S/Inputs/prebuilt-module-cache/Lib.swiftinterface | %target-swift-frontend -parse-stdlib -module-cache-path %t/MCP -emit-module-path %t/prebuilt-cache/Lib.swiftmodule - -module-name Lib
 // RUN: not %target-swift-frontend -typecheck -enable-parseable-module-interface -parse-stdlib -module-cache-path %t/MCP -sdk %S/Inputs -I %S/Inputs/prebuilt-module-cache/ -prebuilt-module-cache-path %t/prebuilt-cache %s 2>&1 | %FileCheck -check-prefix=FROM-PREBUILT %s
-// RUN: ls %t/MCP | not grep swiftmodule
+
+// Make sure we installed a forwarding module.
+// RUN: %{python} %S/Inputs/check-is-forwarding-module.py %t/MCP/Lib-*.swiftmodule
 
 // Try some variations on the detection that the search path is in the SDK:
 // RUN: not %target-swift-frontend -typecheck -enable-parseable-module-interface -parse-stdlib -module-cache-path %t/MCP -sdk %S/Inputs -I %S/Inputs/prebuilt-module-cache/ -prebuilt-module-cache-path %t/prebuilt-cache %s 2>&1 | %FileCheck -check-prefix=FROM-PREBUILT %s

test/ParseableInterface/SmokeTest.swiftinterface

@@ -14,7 +14,7 @@
 // RUN: %target-swift-ide-test -print-module -module-to-print SmokeTest -I %t -source-filename x -print-interface > %t/SmokeTest.txt
 // RUN: %FileCheck %s < %t/SmokeTest.txt
 // RUN: %FileCheck -check-prefix NEGATIVE %s < %t/SmokeTest.txt
-// RUN: llvm-bcanalyzer -dump %t/SmokeTest.swiftmodule | not grep FILE_DEPENDENCY
+// RUN: llvm-bcanalyzer -dump %t/SmokeTest.swiftmodule | grep FILE_DEPENDENCY
 
 // CHECK-LABEL: public class TestClass
 public class TestClass {