Add an option to allow compiling the benchmark test utilities and driver
at -Onone. This is needed because lldb does not support stepping through
optimized swift code.
On Darwin, we pass the *.swiftmodule paths transitively referenced by
the driver executable to ld64. ld64 inserts N_AST references to these
modules into the program, for later use by lldb.
This change should let us exercise much more lldb functionality via
`stepper` testing.
Cleanup and document the configuration of the library path and rpath.
With SWIFT_BENCHMARK_USE_OS_LIBRARIES, it's now possible to directly
build benchmarks for a target device and run those benchmarks on the
device without building or installing Swift.
It's also possible now to specify an absolute SWIFT_LIBRARY_PATH to be
used as an rpath so installation can be skipped.
I also removed the -verify-sil-ownership flag in favor of a disable flag
-disable-sil-ownership-verifier. I used this on only two tests that still need
work to get them to pass with ownership, but whose problems are well understood,
small corner cases. I am going to fix them in follow on commits. I detail them
below:
1. SILOptimizer/definite_init_inout_super_init.swift. This is a test case where
DI is supposed to error. The only problem is that we crash before we error since
the code emitting by SILGen to trigger this error does not pass ownership
invariants. I have spoken with JoeG about this and he suggested that I fix this
earlier in the compiler. Since we do not run the ownership verifier without
asserts enabled, this should not affect compiler users. Given that it has
triggered DI errors previously I think it is safe to disable ownership here.
2. PrintAsObjC/extensions.swift. In this case, the signature generated by type
lowering for one of the thunks here uses an unsafe +0 return value instead of
doing an autorelease return. The ownership checker rightly flags this leak. This
is going to require either an AST level change or a change to TypeLowering. I
think it is safe to turn this off since it is such a corner case that it was
found by a test that has nothing to do with it.
rdar://43398898
I have been meaning to do this change for a minute, but kept on putting it off.
This describes what is actually happening and is a better name for the option.
The build was not specifying the vendor when building the Darwin
benchmarks. Parts of the SDK rely on the vendor macro (`__APPLE__`) to
be defined which requires that the vendor is specified properly. Ensure
that we do so.
Renamed Benchmark_Driver’s `iterations` argument to `independent-samples` to clarify its true meaning and disambiguate it from the concept of `num-iters` used in Benchmark_O. The short form of the argument — `-i` — remains unchanged.
To use this, one needs to first build an installable root for swift (i.e. like
the smoke testbot does). Then use the tool ./benchmark/scripts/build_linux.py
with the appropriate locations of the build-directory, installable snapshot,
and it will build the benchmarks. (There are more arguments, just use --help).
rdar://40541972
I am going to add some support here for Linux. I want to make sure that these
changes are hidden from the main CMakeLists.txt file since we want that file to
be as declarative as possible.
rdar://40541972
We used to have this cmake flag but it seems to have disappeared. Building the
benchmarks with different swiftc flags is central to performance analysis, so
I'm not sure how people were getting this done.
When the SWIFT_BENCHMARK_GENERATE_OPT_VIEW cmake flag is on, the benchmarks are
compiled with -save-optimization-record which generate optimization remarks in
external YAML files. Then the opt-viewer tool from LLVM is invoked to generate
the HTML pages that displays the remarks embedded in the source code.
I've only added it to single-source benchmarks for now.
This can be enabled by
passing --extra-cmake-options='-DSWIFT_BENCHMARK_GENERATE_OPT_VIEW=ON' to
build-script.
This enables us to distinguish in between builds which are triggered by a
subcmake call from the main swift cmake file and one from a user who is trying
to compile the swift benchmark suite against a misc swift installation/use the
ninja file by hand.
This solves problems relating to the SOURCES variable not finding anything
files. module_name expanded to nothing so:
${${module_name}_sources} -> ${_sources} -> ''
Now we handle this properly.
rdar://34556274
I recently broke the out of tree build by mistake [its fixed now ; )]. This
inspired me to make it easy to test this behavior by adding support to
build-script/cmake/etc for running an external benchmark build via
AddExternalProjects.
Now I can just call build-script with --build-external-benchmarks and thats it!
It should just work! It already helped me to avoid breaking the external build
twice!
I hope that eventually we get this on a bot to make sure it keeps working [or
even added to the smoke tests ; )].
*NOTE* This is disabled by default so it will not affect normal builds.
*NOTE* This just builds the external benchmarks. There is an rpath issue that
prevents you from running them (the benchmarks have the rpath set as if they are
next to the stdlib, but they are not. This can be fixed in a few different ways,
but I do not have time to finish implementing it = (. But this commit is a good
first step and at least detects build errors.
Revert "Refactor out the creation of the benchmark libraries into a helper routine swift_benchmark_library. Initially just use it for DriverUtils."
This reverts commit ac539730e4. (+3 squashed commits)
Squashed commits:
[755f057] Revert "[benchmark][cmake] Refactor the swift benchmark helper libraries to use swift_benchmark_library."
This reverts commit a82945e033.
[58e09c6] Revert "[benchmark][cmake] "swift_benchmark_library" => "add_swift_benchmark_library"."
This reverts commit 0e4b7b69cd.
[5f5f2c1] Revert "[benchmark][cmake] Fix typo. This is NFC since this option is usually disabled."
This reverts commit 30f4b1b3e5.
This option is misleading, and never should have been there in the first place. Fortunately it was having no effect because the standard library is already compiled to SIL and the internal checks have been eliminated from any benchmark build.
Some of the benchmarks use Swift 3 APIs. Let’s keep them building that
way to not perturb benchmark numbers.
We should consider adding benchmarks that specifically enable -swift-version 4.
The benchmark bot uses this functionality today to run the benchmarks. By
default build-script only uses 3 samples for each test. Given the noise on our
systems, this is definitely not sufficient for any sort of robust numbers.
Using this patch, I am going to change the benchmarking bot to take the minimum
of 20 samples as we do for our internal benchmarking. This should help make the
benchmark bot give better data. This will have as a cost cause the bot to take
more time. The testing time issue can be solved down the line by changing to a
protocol where we first do tests with a small number of samples (< 5). Then any
benchmark with a delta > 5% is rerun with 20 samples or perhaps until a
statistical criterion is satisfied. But until that is implemented, this at least
makes the bot useful.
There are other things that need to be changed on the benchmarking bot as well,
namely that it should build on a separate machine from which it is running the
benchmarks on. The benchmarking machine should be quiet and not have any work
being done on it. But that is also for another time.
