Otherwise, we can't successfully dry-run a toolchain on darwin.
I added an option enable-extract-symbol-dry-run-test that can be set to restore
the previous behavior and updating the BuildSystem unit tests that depend on
that behavior to pass that flag.
It makes sense to just set this here since we aren't testing that functionality
and enables us to test this if we aren't using one of the specified Xcodes
(which can be useful).
This is where I am going to put the stage 2 swift. The reason why I need to do
this is that I need libdispatch, foundation, etc on Linux to use the stage2
compiler.
The key thing here is that by using this builder, I am going to be able to split
the build-script-impl pipeline in two and put in a build-script pipeline in
between. This is needed so that on Linux, we can build the stage 2 compiler
before we build any of the build-script-impl libraries that depend on having a
swift compiler.
This should be an NFCI change. I am relying on our build system unit tests to
ensure that I am not changing any real behavior.
Add three new flags, '--skip-clean-libdispatch', '--skip-clean-foundation', and
'--skip-clean-xctest', that leave the previous builds of those products in place.
Pass the Swift and CMake flags needed to cross-compile Foundation and so on, with
the first example of Android. Add a new flag, --cross-compile-deps-path, which is
used to search for cross-compiled libraries, like libcurl, that the corelibs
depend on. Also add a new flag, --common-swift-flags, to pass additional Swift
flags to the corelibs.
We should re-enable this once it is clear that there aren't issues with the
Xcode generator/host side swift cmake. The Ninja generator (the default) still
is enabled.
The reason why I am doing this is that we do not yet have on all of the
non-Darwin bots swift host toolchain files. So we can't test this code path on
those platforms and thus can not guarantee correctness. The result is that on
those other platforms if someone /does/ have a host toolchain with a swift
binary in it, failures may result breaking other people's builds.
There are three things going on here (note all on Darwin):
1. If one compiles a swift static library and links the static library into a
cxx executable, the cxx executable will need the -L flags to find the
appropriate libraries in the SDK/toolchain.
2. I fixed an rpath issue where due to old code I didn't understand/propagated
forward, we were setting the rpath to be the -L directory in the appropriate
SDK. After reasoning about this a little bit I realized that this was code
that was actually intended to be used for target libraries (for which the
given rpath would be correct). On the host side though on Darwin, we want to
use the rpath for the standard stabilized stdlib on the system.
3. I added Build System Unittests to ensure that this keeps on working. I also
added test cases that I should have added before. I just had never thought
about how to test this and I realized this method would work and would
prevent regressions while I am waiting for a new swiftc with driver fixes to
land.
In the original implementation (https://github.com/apple/swift/pull/36377), using `--infer` accidentally disables the `earlyswiftdriver` product (`before_impl_product_classes` set to always empty). This change fixes that and makes sure early SwiftDriver is always built, regardless of whether or not `--infer` is used.
This change also ensures that `install_all` setting triggered by `--infer` does not affect products which specify `is_ignore_install_all_product` to return True. This is useful for products which should not be installed into the toolchain (corresponding build products that use the just-built toolchain are the products that get installed, e.g. `swiftdriver` to `earlyswiftdriver`).
...and reenable `llvm-targets-options.test` (previously disabled in #37573).
This will align `skip-local-build.test-sh` with the behaviour of the
other BuildSystem tests, by
* ensuring we use the `cmake` exposed in `lit.cfg`, so that under Linux we
don't attempt to rebuild it
* using a separate build folder for `build-script` invocations, so that
side effects will not affect the main invocation and other lit tests.
I expect these changes to prevent `llvm-targets-options.test` to fail in Linux
presets with an error related to cmake, e.g.
```
build-script: error: argument --cmake: /home/buildnode/jenkins/workspace/
oss-swift-package-linux-ubuntu-18_04/build/cmake-linux-x86_64/bin/cmake is not an executable
```
Addresses rdar://78320684
This will align skip-local-build.test-sh with the behaviour of the
other BuildSystem tests, by
* ensuring we use the cmake exposed in lit.cfg, so that under Linux we
don't attempt to rebuild it
* using a separate build folder for build-script invocations, so that
side effects will not affect the main invocation and other lit tests.
I expect these changes to prevent llvm-targets-options.test to fail in
Linux presets with an error related to cmake, e.g.
```
build-script: error: argument --cmake:
/home/buildnode/jenkins/workspace/
oss-swift-package-linux-ubuntu-18_04/build/cmake-linux-x86_64/bin/cmake
is not an executable
```
Addresses rdar://78320684
... in tests related to symbol generation.
This is to ensure that the dry run invocations of `build-script` use the
same value as `%target-cpu` and prevent failures when running validation
tests for an architecture different from the current one.
Addresses rdar://78141544
This would be needed to reduce overall build times in scenarios when
generating symbols for all binaries is too expensive and/or not needed.
At the same time, introduce tests around the logic that handles symbols.
Addresses rdar://76865276
This would be needed to reduce overall build times in scenarios when
generating symbols for all binaries is too expensive and/or not needed.
Addresses rdar://76865276
...before adding the logic to filter paths
In particular:
* print the list of files that `cpio` copies, so we can test explicitly
which files end up in the symroot (and also see those when asking for
a toolchain in PR testing)
* use `find` instead of `grep` to filter files we want symbols generated
for -- this is to avoid the script failing when there are no symbol to
process, especially in lit tests
* remove an unnecessary check for `swift-api-digester` -- this is now a
symlink to `swift-frontend` and we only process regular files.
Supports rdar://76865276
This will make sure that compiler developers are using the new driver when they build the compiler locally and use it.
- Adds a new build-script product category: before_build_script_impl for products we wish to build before the impl products.
- Adds a new EarlySwiftDriver product to that category, which gets built with the host toolchain.
- Adds an escape hatch: --skip-early-swift-driver
- Adjusts the swift CMake configuration with an additional step: swift_create_early_driver_symlinks which (if one was built) creates symlinks in the swift build bin directory to the EarlySwiftDriver swift-driver and swift-help executables.
- Adds a new test subset : only_early_swiftdriver, which will get built into a corresponding CMake test target: check-swift-only_early_swiftdriver-* which runs a small subset of driver-related tests against the Early SwiftDriver.
- This subset is run always when the compiler itself is tested (--test is specified)
- With an escape disable-switch: --skip-test-early-swift-driver
- All tests outside of only_early_swiftdriver are forced to run using the legacy C++ driver (to ensure it gets tested, still).
NOTE: SwiftDriver product (no 'Early') is still the main product used to build the driver for toolchain installation into and for executing the product's own tests. This change does not affect that.
This is meant to support scenarios in which we need to build as little as
LLVM as possible for performance reasons (e.g. when enabling LTO).
While LLVM CMake build system offers options in this sense,
in our investigation they turned out not to be suitable,
since either they are not granular enough (`LLVM_INCLUDE/BUILD` flags)
or they require active opt out for any new tool added
(`*_BUILD_*_TOOL` flags)
When using this mechanism, there is the possibility to specify different
targets to use for cross-compile hosts.
Supports rdar://32019390
Introduce a new parameter `--skip-llbuild-clean` to prevent this from
happening.
This mimicks similar logic in place for `swiftpm`, `swift-driver`
(#33563) and `xctest` (#19512)
Addresses rdar://75057069
* [Concurrency] Build C only libdispatch before Swift on non-Darwin platforms
_Concurrency depends on libdispatch and since it is not available by
default on non-Darwin platforms, it needs to be built before the Swift
project, so that the dependency can be resolved.
* Fix clibdispatch installation and BuildSystem tests
* Fix build system tests on Darwin and formatting
This is meant to support parallel CI runs on the same Linux bot, so that
they don't share the module cache and reduce (hopefully) the likelihood
of issues related to invalid signatures.
When the option is enabled, the environment variable `XDG_CACHE_HOME`
is explicitly set in `build-script` and should be inherited by all the
child processes.
Currently we don't check the operating system before setting the
variable (since it should be a noop for other OSes).
Addresses rdar://73887745
This should enable scaling when using machines with large amount of
RAM.
To better support machines with lower spec, process one binary per
dsymutil invocation (reverting #34149).
Add some (limited) facilities to gather the time taken to execute
dsymutil to better assist in tuning the parameter -- these are printed in
JSON format in the log to allow for easier scraping
```
{ "command": "dsymutil", "start": "2020-11-18T18:10:47" }
{ "command": "dsymutil", "end": "2020-11-18T18:14:45" }
```
Addresses rdar://71018443
