This is where I am going to put the stage 2 swift. The reason why I need to do
this is that I need libdispatch, foundation, etc on Linux to use the stage2
compiler.
The key thing here is that by using this builder, I am going to be able to split
the build-script-impl pipeline in two and put in a build-script pipeline in
between. This is needed so that on Linux, we can build the stage 2 compiler
before we build any of the build-script-impl libraries that depend on having a
swift compiler.
This should be an NFCI change. I am relying on our build system unit tests to
ensure that I am not changing any real behavior.
Add three new flags, '--skip-clean-libdispatch', '--skip-clean-foundation', and
'--skip-clean-xctest', that leave the previous builds of those products in place.
Pass the Swift and CMake flags needed to cross-compile Foundation and so on, with
the first example of Android. Add a new flag, --cross-compile-deps-path, which is
used to search for cross-compiled libraries, like libcurl, that the corelibs
depend on. Also add a new flag, --common-swift-flags, to pass additional Swift
flags to the corelibs.
We should re-enable this once it is clear that there aren't issues with the
Xcode generator/host side swift cmake. The Ninja generator (the default) still
is enabled.
The reason why I am doing this is that we do not yet have on all of the
non-Darwin bots swift host toolchain files. So we can't test this code path on
those platforms and thus can not guarantee correctness. The result is that on
those other platforms if someone /does/ have a host toolchain with a swift
binary in it, failures may result breaking other people's builds.
There are three things going on here (note all on Darwin):
1. If one compiles a swift static library and links the static library into a
cxx executable, the cxx executable will need the -L flags to find the
appropriate libraries in the SDK/toolchain.
2. I fixed an rpath issue where due to old code I didn't understand/propagated
forward, we were setting the rpath to be the -L directory in the appropriate
SDK. After reasoning about this a little bit I realized that this was code
that was actually intended to be used for target libraries (for which the
given rpath would be correct). On the host side though on Darwin, we want to
use the rpath for the standard stabilized stdlib on the system.
3. I added Build System Unittests to ensure that this keeps on working. I also
added test cases that I should have added before. I just had never thought
about how to test this and I realized this method would work and would
prevent regressions while I am waiting for a new swiftc with driver fixes to
land.
In the original implementation (https://github.com/apple/swift/pull/36377), using `--infer` accidentally disables the `earlyswiftdriver` product (`before_impl_product_classes` set to always empty). This change fixes that and makes sure early SwiftDriver is always built, regardless of whether or not `--infer` is used.
This change also ensures that `install_all` setting triggered by `--infer` does not affect products which specify `is_ignore_install_all_product` to return True. This is useful for products which should not be installed into the toolchain (corresponding build products that use the just-built toolchain are the products that get installed, e.g. `swiftdriver` to `earlyswiftdriver`).
...and reenable `llvm-targets-options.test` (previously disabled in #37573).
This will align `skip-local-build.test-sh` with the behaviour of the
other BuildSystem tests, by
* ensuring we use the `cmake` exposed in `lit.cfg`, so that under Linux we
don't attempt to rebuild it
* using a separate build folder for `build-script` invocations, so that
side effects will not affect the main invocation and other lit tests.
I expect these changes to prevent `llvm-targets-options.test` to fail in Linux
presets with an error related to cmake, e.g.
```
build-script: error: argument --cmake: /home/buildnode/jenkins/workspace/
oss-swift-package-linux-ubuntu-18_04/build/cmake-linux-x86_64/bin/cmake is not an executable
```
Addresses rdar://78320684
This will align skip-local-build.test-sh with the behaviour of the
other BuildSystem tests, by
* ensuring we use the cmake exposed in lit.cfg, so that under Linux we
don't attempt to rebuild it
* using a separate build folder for build-script invocations, so that
side effects will not affect the main invocation and other lit tests.
I expect these changes to prevent llvm-targets-options.test to fail in
Linux presets with an error related to cmake, e.g.
```
build-script: error: argument --cmake:
/home/buildnode/jenkins/workspace/
oss-swift-package-linux-ubuntu-18_04/build/cmake-linux-x86_64/bin/cmake
is not an executable
```
Addresses rdar://78320684
... in tests related to symbol generation.
This is to ensure that the dry run invocations of `build-script` use the
same value as `%target-cpu` and prevent failures when running validation
tests for an architecture different from the current one.
Addresses rdar://78141544
This would be needed to reduce overall build times in scenarios when
generating symbols for all binaries is too expensive and/or not needed.
At the same time, introduce tests around the logic that handles symbols.
Addresses rdar://76865276
This would be needed to reduce overall build times in scenarios when
generating symbols for all binaries is too expensive and/or not needed.
Addresses rdar://76865276
...before adding the logic to filter paths
In particular:
* print the list of files that `cpio` copies, so we can test explicitly
which files end up in the symroot (and also see those when asking for
a toolchain in PR testing)
* use `find` instead of `grep` to filter files we want symbols generated
for -- this is to avoid the script failing when there are no symbol to
process, especially in lit tests
* remove an unnecessary check for `swift-api-digester` -- this is now a
symlink to `swift-frontend` and we only process regular files.
Supports rdar://76865276
This will make sure that compiler developers are using the new driver when they build the compiler locally and use it.
- Adds a new build-script product category: before_build_script_impl for products we wish to build before the impl products.
- Adds a new EarlySwiftDriver product to that category, which gets built with the host toolchain.
- Adds an escape hatch: --skip-early-swift-driver
- Adjusts the swift CMake configuration with an additional step: swift_create_early_driver_symlinks which (if one was built) creates symlinks in the swift build bin directory to the EarlySwiftDriver swift-driver and swift-help executables.
- Adds a new test subset : only_early_swiftdriver, which will get built into a corresponding CMake test target: check-swift-only_early_swiftdriver-* which runs a small subset of driver-related tests against the Early SwiftDriver.
- This subset is run always when the compiler itself is tested (--test is specified)
- With an escape disable-switch: --skip-test-early-swift-driver
- All tests outside of only_early_swiftdriver are forced to run using the legacy C++ driver (to ensure it gets tested, still).
NOTE: SwiftDriver product (no 'Early') is still the main product used to build the driver for toolchain installation into and for executing the product's own tests. This change does not affect that.
This is meant to support scenarios in which we need to build as little as
LLVM as possible for performance reasons (e.g. when enabling LTO).
While LLVM CMake build system offers options in this sense,
in our investigation they turned out not to be suitable,
since either they are not granular enough (`LLVM_INCLUDE/BUILD` flags)
or they require active opt out for any new tool added
(`*_BUILD_*_TOOL` flags)
When using this mechanism, there is the possibility to specify different
targets to use for cross-compile hosts.
Supports rdar://32019390
Introduce a new parameter `--skip-llbuild-clean` to prevent this from
happening.
This mimicks similar logic in place for `swiftpm`, `swift-driver`
(#33563) and `xctest` (#19512)
Addresses rdar://75057069
* [Concurrency] Build C only libdispatch before Swift on non-Darwin platforms
_Concurrency depends on libdispatch and since it is not available by
default on non-Darwin platforms, it needs to be built before the Swift
project, so that the dependency can be resolved.
* Fix clibdispatch installation and BuildSystem tests
* Fix build system tests on Darwin and formatting
This is meant to support parallel CI runs on the same Linux bot, so that
they don't share the module cache and reduce (hopefully) the likelihood
of issues related to invalid signatures.
When the option is enabled, the environment variable `XDG_CACHE_HOME`
is explicitly set in `build-script` and should be inherited by all the
child processes.
Currently we don't check the operating system before setting the
variable (since it should be a noop for other OSes).
Addresses rdar://73887745
This should enable scaling when using machines with large amount of
RAM.
To better support machines with lower spec, process one binary per
dsymutil invocation (reverting #34149).
Add some (limited) facilities to gather the time taken to execute
dsymutil to better assist in tuning the parameter -- these are printed in
JSON format in the log to allow for easier scraping
```
{ "command": "dsymutil", "start": "2020-11-18T18:10:47" }
{ "command": "dsymutil", "end": "2020-11-18T18:14:45" }
```
Addresses rdar://71018443
This causes build-script to use the conservative dependency information that I
committed. When one uses this option, it is assumed that one wants to also
install all built products.
Some notes:
1. I included an extra --install-all option so without --infer enabled
one can enable this sort of install everything that we want to
build behavior.
2. I added %cmake as a lit variable. I did this so I could specify in
my build-system unit tests that on Linux they should use the just
built cmake (if we built cmake due to an old cmake being on the
system). Otherwise, the build system unit tests took way too
long. These are meant to be dry-runs, so building this cmake again
is just wasteful and doesn't make sense.
3. I unified how we handle cmark, llvm, swift with the rest of the
build products by making them conditional on build_* variables, but
to preserve current behavior I made it so that they are just
enabled by default unlike things like
llbuild/swiftpm/foundation/etc. This was necessary since previously
we would just pass these flags to build-script-impl and
build-script didn't know about them. Now I taught build-script
about them so I can manipulate these skip-build-{cmark,llvm,swift}
and then just pass them down to build-script-impl if appropriate
rather than relying on build-script-impl to control if these are
built.
Once this lands, I think we are at a good enough place with
build-script until we get rid of build-script-impl in terms of high
value QoI that will imnprove productivity. Once build-script-impl is
destroyed, we can start paring back what build-script itself does.
