The idea being, we need to decide what benchmarks to run solely based on
tags.
`--tag` allows to list all tags that are required;
`--skip-tags` allows to skip benchmarks that have any of those tags.
By default, skip-tags list contains .unstable and .String, which results
in the same subset of benchmarks as before.
This enables us to distinguish in between builds which are triggered by a
subcmake call from the main swift cmake file and one from a user who is trying
to compile the swift benchmark suite against a misc swift installation/use the
ninja file by hand.
The key thing here is that by providing one of these closures, a benchmark can
inject the initialization/deinitialization of its internal data structures,
outside of the time period where timing is occurring.
The intention is that this will provide us the framework for as we annotate
tests with BenchmarkInfo, to move initialization work out of benchmarks.
It will also allow for more complex benchmarks to be written such as ones that
perform bulk reads from a pipe (my interest in this).
Previously, we both filtered tests and marked tests as not being able to be run.
This commit rips out the marking of tests as being unable to be run (but counted
as part of the indices output by the compiler).
This also ensures benchmarks that are run are always given an ascending index
instead of sometimes having indices being skipped.
*NOTE* We always prefer a registered benchmark if we have one.
I am going to use BenchmarkInfo to solve the "create data for benchmark while we
are already timing" problem. I am going to add a field to BenchmarkInfo that if
it is not-null is called before we start measuring time. This closure can be
used to initialize any global data structures/etc.
But to do this, I need to be able to combine the registered and legacy
not-registered benchmarks.
Only use the .abstraction tag for benchmarks that are primarilly stressing the
optimization of a specific kind of abstraction. Then I can use these as CPU
benchmarks.
Previously this tag was applied to any benchmark that happened to involve
classes or protocols. When a compiler engineer changes something that might
affect codegen in these cases, they should simply benchmark the entire stable
suite. There's no value in having a separate set of 97 "abstraction" benchmarks
that are primarilly testing something else, like a stdlib API.
This solves problems relating to the SOURCES variable not finding anything
files. module_name expanded to nothing so:
${${module_name}_sources} -> ${_sources} -> ''
Now we handle this properly.
rdar://34556274
I recently broke the out of tree build by mistake [its fixed now ; )]. This
inspired me to make it easy to test this behavior by adding support to
build-script/cmake/etc for running an external benchmark build via
AddExternalProjects.
Now I can just call build-script with --build-external-benchmarks and thats it!
It should just work! It already helped me to avoid breaking the external build
twice!
I hope that eventually we get this on a bot to make sure it keeps working [or
even added to the smoke tests ; )].
*NOTE* This is disabled by default so it will not affect normal builds.
*NOTE* This just builds the external benchmarks. There is an rpath issue that
prevents you from running them (the benchmarks have the rpath set as if they are
next to the stdlib, but they are not. This can be fixed in a few different ways,
but I do not have time to finish implementing it = (. But this commit is a good
first step and at least detects build errors.
When a benchmark is added, it's name will be manually added to CMakeLists and
main.swift.
Adding a benchmark no longer requires discovering a nonobvious script.
If the build works locally, then it will work everywhere else.
Note that generate_harness.py should now be renamed to generate_benchmarks.py,
but I think it's name may be hard-coded somewhere in the compiler tests. Anyway,
this script should go away as soon as someone fixes the cmake configuration to
automatically run gyb.
Revert "Refactor out the creation of the benchmark libraries into a helper routine swift_benchmark_library. Initially just use it for DriverUtils."
This reverts commit ac539730e4. (+3 squashed commits)
Squashed commits:
[755f057] Revert "[benchmark][cmake] Refactor the swift benchmark helper libraries to use swift_benchmark_library."
This reverts commit a82945e033.
[58e09c6] Revert "[benchmark][cmake] "swift_benchmark_library" => "add_swift_benchmark_library"."
This reverts commit 0e4b7b69cd.
[5f5f2c1] Revert "[benchmark][cmake] Fix typo. This is NFC since this option is usually disabled."
This reverts commit 30f4b1b3e5.
