This would be needed to reduce overall build times in scenarios when
generating symbols for all binaries is too expensive and/or not needed.
Addresses rdar://76865276
This will make sure that compiler developers are using the new driver when they build the compiler locally and use it.
- Adds a new build-script product category: before_build_script_impl for products we wish to build before the impl products.
- Adds a new EarlySwiftDriver product to that category, which gets built with the host toolchain.
- Adds an escape hatch: --skip-early-swift-driver
- Adjusts the swift CMake configuration with an additional step: swift_create_early_driver_symlinks which (if one was built) creates symlinks in the swift build bin directory to the EarlySwiftDriver swift-driver and swift-help executables.
- Adds a new test subset : only_early_swiftdriver, which will get built into a corresponding CMake test target: check-swift-only_early_swiftdriver-* which runs a small subset of driver-related tests against the Early SwiftDriver.
- This subset is run always when the compiler itself is tested (--test is specified)
- With an escape disable-switch: --skip-test-early-swift-driver
- All tests outside of only_early_swiftdriver are forced to run using the legacy C++ driver (to ensure it gets tested, still).
NOTE: SwiftDriver product (no 'Early') is still the main product used to build the driver for toolchain installation into and for executing the product's own tests. This change does not affect that.
This is meant to support scenarios in which we need to build as little as
LLVM as possible for performance reasons (e.g. when enabling LTO).
While LLVM CMake build system offers options in this sense,
in our investigation they turned out not to be suitable,
since either they are not granular enough (`LLVM_INCLUDE/BUILD` flags)
or they require active opt out for any new tool added
(`*_BUILD_*_TOOL` flags)
When using this mechanism, there is the possibility to specify different
targets to use for cross-compile hosts.
Supports rdar://32019390
Introduce a new parameter `--skip-llbuild-clean` to prevent this from
happening.
This mimicks similar logic in place for `swiftpm`, `swift-driver`
(#33563) and `xctest` (#19512)
Addresses rdar://75057069
This is meant to support parallel CI runs on the same Linux bot, so that
they don't share the module cache and reduce (hopefully) the likelihood
of issues related to invalid signatures.
When the option is enabled, the environment variable `XDG_CACHE_HOME`
is explicitly set in `build-script` and should be inherited by all the
child processes.
Currently we don't check the operating system before setting the
variable (since it should be a noop for other OSes).
Addresses rdar://73887745
This should enable scaling when using machines with large amount of
RAM.
To better support machines with lower spec, process one binary per
dsymutil invocation (reverting #34149).
Add some (limited) facilities to gather the time taken to execute
dsymutil to better assist in tuning the parameter -- these are printed in
JSON format in the log to allow for easier scraping
```
{ "command": "dsymutil", "start": "2020-11-18T18:10:47" }
{ "command": "dsymutil", "end": "2020-11-18T18:14:45" }
```
Addresses rdar://71018443
Along with options to disable the mandatory clean using: `--skip-clean-swift-driver` and `--skip-clean-swiftpm`.
This will ensure that every invocation of build-script will, by default, clean up all build artifacts of these projects and re-build them from scratch.
This is needed for all builds because today arbitrary changes to the compiler can lead to us being unable to incrementally build components that are themselves written in Swift. This causes now-frequent failures in incremental build bots, and is a scenario that is encountered by developers. (For example see: rdar://65006593)
The proper long-term solution is to enable library evolution for these projects. Until this is done, the only safe thing to do is to always rebuild them.
Resolves rdar://65006593
The experimental concurrency model will require a supporting runtime
and possibly end-user-visible library constructs. Introduce a stub of
such a library, enabled by a new `build-script` option
`--enable-experimental-concurrency`, so we have a place to put this
work.
This product supports building and testing swift-format, forwarding the commands to a build script inside of swift-format's repo to handle actually invoking Swift PM. There are new command line options to support the swift-format product:
`--swiftformat`: Enables building swift-format.
`--skip-test-swiftformat': Disables running tests for swift-format after building.
Installing is intentionally not implemented because swift-format isn't ready to be installed as part of the Swift toolchain.
This causes build-script to use the conservative dependency information that I
committed. When one uses this option, it is assumed that one wants to also
install all built products.
Some notes:
1. I included an extra --install-all option so without --infer enabled
one can enable this sort of install everything that we want to
build behavior.
2. I added %cmake as a lit variable. I did this so I could specify in
my build-system unit tests that on Linux they should use the just
built cmake (if we built cmake due to an old cmake being on the
system). Otherwise, the build system unit tests took way too
long. These are meant to be dry-runs, so building this cmake again
is just wasteful and doesn't make sense.
3. I unified how we handle cmark, llvm, swift with the rest of the
build products by making them conditional on build_* variables, but
to preserve current behavior I made it so that they are just
enabled by default unlike things like
llbuild/swiftpm/foundation/etc. This was necessary since previously
we would just pass these flags to build-script-impl and
build-script didn't know about them. Now I taught build-script
about them so I can manipulate these skip-build-{cmark,llvm,swift}
and then just pass them down to build-script-impl if appropriate
rather than relying on build-script-impl to control if these are
built.
Once this lands, I think we are at a good enough place with
build-script until we get rid of build-script-impl in terms of high
value QoI that will imnprove productivity. Once build-script-impl is
destroyed, we can start paring back what build-script itself does.
This is just based off of the defaults that we use on the buildbots as of this
commit.
In order to not break the bots as they are, I left the internal representation
as a semicolon list of components. So this should be NFC from the perspective of
the bots since they all explicitly specify llvm-install-components if needed
(and we do not install without install-llvm anymore).
Support the usual `--enable-*san options`, but also add a
`--test-indexstore-db-santitize-all` that runs the tests once for each
sanitizer. Sanitizing just indexstore-db with a regular toolchain should
be much faster than using sanitized compilers.
Now that the autodifferentiation support is being upstreamed, add an
option to enable building the TensorFlow swift-apis package optionally.
This enables easier development cycles for the engineers working on it.
This migrates the playground support out of the build-script-impl and
into the python based build system. This makes it build more similarly
to the Swift Package Manager and SourceKit-LSP. More importantly, it
reduces the dependency on build-script-impl.
This commit adds initial build system support for macCatalyst,
an Apple technology that enables code targeting iOS
to be recompiled so that it can be executed on macOS while still using
iOS APIs. This is the first in a series of commits building out support for
macCatalyst in the compiler, runtime, standard library, and overlays. Swift
for macCatalyst represents the work of multiple people, including
Devin Coughlin, Ross Bayer, and Brent Royal-Gordon.
Under macCatalyst, compiler-provided shared libraries (including overlays)
are built as one of four kinds (or "flavors") of libraries,
each with different install names and Mach-O load commands. This commit
adds the build system infrastructure to produce these different
library flavors.
**macOS-like Libraries**
A "macOS-like" library (such as the GLKit overlay) is a plain-old macOS library
that can only be loaded into regular macOS processes. It has a macOS slice with
a single load command allowing it to be loaded into normal macOS processes.
**iOS-like Libraries**
An "iOS-like" library, such as the UIKit overlay, is a library with a
macOS slice but with a load command that only allows it be loaded into
macCatalyst processes. iOS-like libraries are produced by passing a new
target tuple to the compiler:
swiftc ... -target x86_64-apple-ios13.0-macabi ...
Here 'ios' (and an iOS version number) is used for OS portion
of the triple, but the 'macabi' environment tells the compiler
that the library is intended for macCatalyst.
**Zippered Libraries**
A "zippered" library can be loaded into either a macCatalyst process or
a standard macOS process. Since macCatalyst does not introduce a new Mach-O
slice, the same code is shared between both processes. Zippered libraries
are usually relatively low level and with an API surface that is similar
between macOS and iOS (for example, both the Foundation overlay and the Swift
Standard Library/Runtime itself are zippered).
Zippered libraries are created by passing both the usual `-target`
flag to the compiler and an additional `-target-variant` flag:
swiftc ... -target x86_64-apple-macos10.15 \
-target-variant x86_64-apple-ios13.0-macabi
Just like the -target flag, -target-variant takes a target tuple.
This tells the compiler to compile the library for the -target tuple but
to add an extra load command, allowing the library to be loaded into processes
of the -target-variant flavor as well.
While a single zippered library and slice is shared between macOS and
macCatalyst, zippered libraries require two separate .swiftinterface/.swiftmodule
files, one for macOS and one for macCatalyst. When a macOS or macCatalyst client
imports the library, it will use module file for its flavor to determine what
symbols are present. This enables a zippered library to expose a subset of its
target APIs to its target-variant.
**Unzippered-Twin Libraries**
"Unzippered Twins" are pairs of libraries with the same name but different
contents and install locations, one for use from macOS processes and one for
use from macCatalyst processes. Unzippered twins are usually libraries that
depend on AppKit on macOS and UIKit on iOS (for example, the MapKit overlay)
and so do not share a common implementation between macOS and macCatalyst.
The macCatalyst version of an unzippered twin is installed in a parallel
directory hierarchy rooted at /System/iOSSupport/. So, for example, while macOS
and zippered Swift overlays are installed in /usr/lib/swift/, iOS-like and
the macCatalyst side of unzippered twins are installed in
/System/iOSSupport/usr/lib/swift. When building for macCatalyst, the build system
passes additional search paths so that the macCatalyst version of libraries is
found before macOS versions.
The add_swift_target_library() funciton now take an
optional MACCATALYST_BUILD_FLAVOR, which enables swift libraries to indicate
which flavor of library they are.
Having the test directory match the module we are testing means we can have scripts in the top level of utils/build_swift which can also have tests. As part of this re-structure the test utilties have been simplified somewhat and all tests no longer use a custom TestCase, rather the standard one exposed by the unittest module.
SwiftSyntax is not part of the standard library and thus should not be
installed in usr/lib/swift.
This also removes the code to install SwiftSyntax's .swiftmodule file
since that code path was never exercised.
