Renamed Benchmark_Driver’s `iterations` argument to `independent-samples` to clarify its true meaning and disambiguate it from the concept of `num-iters` used in Benchmark_O. The short form of the argument — `-i` — remains unchanged.
To use this, one needs to first build an installable root for swift (i.e. like
the smoke testbot does). Then use the tool ./benchmark/scripts/build_linux.py
with the appropriate locations of the build-directory, installable snapshot,
and it will build the benchmarks. (There are more arguments, just use --help).
rdar://40541972
I am going to add some support here for Linux. I want to make sure that these
changes are hidden from the main CMakeLists.txt file since we want that file to
be as declarative as possible.
rdar://40541972
We used to have this cmake flag but it seems to have disappeared. Building the
benchmarks with different swiftc flags is central to performance analysis, so
I'm not sure how people were getting this done.
When the SWIFT_BENCHMARK_GENERATE_OPT_VIEW cmake flag is on, the benchmarks are
compiled with -save-optimization-record which generate optimization remarks in
external YAML files. Then the opt-viewer tool from LLVM is invoked to generate
the HTML pages that displays the remarks embedded in the source code.
I've only added it to single-source benchmarks for now.
This can be enabled by
passing --extra-cmake-options='-DSWIFT_BENCHMARK_GENERATE_OPT_VIEW=ON' to
build-script.
This enables us to distinguish in between builds which are triggered by a
subcmake call from the main swift cmake file and one from a user who is trying
to compile the swift benchmark suite against a misc swift installation/use the
ninja file by hand.
This solves problems relating to the SOURCES variable not finding anything
files. module_name expanded to nothing so:
${${module_name}_sources} -> ${_sources} -> ''
Now we handle this properly.
rdar://34556274
I recently broke the out of tree build by mistake [its fixed now ; )]. This
inspired me to make it easy to test this behavior by adding support to
build-script/cmake/etc for running an external benchmark build via
AddExternalProjects.
Now I can just call build-script with --build-external-benchmarks and thats it!
It should just work! It already helped me to avoid breaking the external build
twice!
I hope that eventually we get this on a bot to make sure it keeps working [or
even added to the smoke tests ; )].
*NOTE* This is disabled by default so it will not affect normal builds.
*NOTE* This just builds the external benchmarks. There is an rpath issue that
prevents you from running them (the benchmarks have the rpath set as if they are
next to the stdlib, but they are not. This can be fixed in a few different ways,
but I do not have time to finish implementing it = (. But this commit is a good
first step and at least detects build errors.
Revert "Refactor out the creation of the benchmark libraries into a helper routine swift_benchmark_library. Initially just use it for DriverUtils."
This reverts commit ac539730e4. (+3 squashed commits)
Squashed commits:
[755f057] Revert "[benchmark][cmake] Refactor the swift benchmark helper libraries to use swift_benchmark_library."
This reverts commit a82945e033.
[58e09c6] Revert "[benchmark][cmake] "swift_benchmark_library" => "add_swift_benchmark_library"."
This reverts commit 0e4b7b69cd.
[5f5f2c1] Revert "[benchmark][cmake] Fix typo. This is NFC since this option is usually disabled."
This reverts commit 30f4b1b3e5.
This option is misleading, and never should have been there in the first place. Fortunately it was having no effect because the standard library is already compiled to SIL and the internal checks have been eliminated from any benchmark build.
Some of the benchmarks use Swift 3 APIs. Let’s keep them building that
way to not perturb benchmark numbers.
We should consider adding benchmarks that specifically enable -swift-version 4.
The benchmark bot uses this functionality today to run the benchmarks. By
default build-script only uses 3 samples for each test. Given the noise on our
systems, this is definitely not sufficient for any sort of robust numbers.
Using this patch, I am going to change the benchmarking bot to take the minimum
of 20 samples as we do for our internal benchmarking. This should help make the
benchmark bot give better data. This will have as a cost cause the bot to take
more time. The testing time issue can be solved down the line by changing to a
protocol where we first do tests with a small number of samples (< 5). Then any
benchmark with a delta > 5% is rerun with 20 samples or perhaps until a
statistical criterion is satisfied. But until that is implemented, this at least
makes the bot useful.
There are other things that need to be changed on the benchmarking bot as well,
namely that it should build on a separate machine from which it is running the
benchmarks on. The benchmarking machine should be quiet and not have any work
being done on it. But that is also for another time.
This is an intermediate commit in a series of commits towards being able to
build the perf test suite from *.sib files.
Doing so will enable us to:
1. Stress the serialization pipeline.
2. Provide an interesting test case for verifying that executables compiled
from *.sib yield the same object files as if we compiled directly to *.o files.
Keep in mind this is a longer term effort that I am doing on the side.
