Revert "Refactor out the creation of the benchmark libraries into a helper routine swift_benchmark_library. Initially just use it for DriverUtils."
This reverts commit ac539730e4. (+3 squashed commits)
Squashed commits:
[755f057] Revert "[benchmark][cmake] Refactor the swift benchmark helper libraries to use swift_benchmark_library."
This reverts commit a82945e033.
[58e09c6] Revert "[benchmark][cmake] "swift_benchmark_library" => "add_swift_benchmark_library"."
This reverts commit 0e4b7b69cd.
[5f5f2c1] Revert "[benchmark][cmake] Fix typo. This is NFC since this option is usually disabled."
This reverts commit 30f4b1b3e5.
This option is misleading, and never should have been there in the first place. Fortunately it was having no effect because the standard library is already compiled to SIL and the internal checks have been eliminated from any benchmark build.
Some of the benchmarks use Swift 3 APIs. Let’s keep them building that
way to not perturb benchmark numbers.
We should consider adding benchmarks that specifically enable -swift-version 4.
The benchmark bot uses this functionality today to run the benchmarks. By
default build-script only uses 3 samples for each test. Given the noise on our
systems, this is definitely not sufficient for any sort of robust numbers.
Using this patch, I am going to change the benchmarking bot to take the minimum
of 20 samples as we do for our internal benchmarking. This should help make the
benchmark bot give better data. This will have as a cost cause the bot to take
more time. The testing time issue can be solved down the line by changing to a
protocol where we first do tests with a small number of samples (< 5). Then any
benchmark with a delta > 5% is rerun with 20 samples or perhaps until a
statistical criterion is satisfied. But until that is implemented, this at least
makes the bot useful.
There are other things that need to be changed on the benchmarking bot as well,
namely that it should build on a separate machine from which it is running the
benchmarks on. The benchmarking machine should be quiet and not have any work
being done on it. But that is also for another time.
This is an intermediate commit in a series of commits towards being able to
build the perf test suite from *.sib files.
Doing so will enable us to:
1. Stress the serialization pipeline.
2. Provide an interesting test case for verifying that executables compiled
from *.sib yield the same object files as if we compiled directly to *.o files.
Keep in mind this is a longer term effort that I am doing on the side.
