Report only the total number of executed tests.
Aggregating MIN, MAX and MEAN values for all executed benchmarks together (with microsecond precision!) has no statistical relevance.
Reintroduced feature lost during `BenchmarkInfo` modernization: All registered benchmarks are ordered alphabetically and assigned an index. This number can be used as a shortcut to invoke the test instead of its full name. (Adding and removing tests from the suite will naturally reassign the indices, but they are stable for a given build.)
The `--list` parameter now prints the test *number*, *name* and *tags* separated by delimiter.
The `--list` output format is modified from:
````
Enabled Tests,Tags
AngryPhonebook,[String, api, validation]
...
````
to this:
````
\#,Test,[Tags]
2,AngryPhonebook,[String, api, validation]
…
````
(There isn’t a backslash before the #, git was eating the whole line without it.)
Note: Test number 1 is Ackermann, which is marked as “skip”, so it’s not listed with the default `skip-tags` value.
Fixes the issue where running tests via `Benchmark_Driver` always reported each test as number 1. Each test is run independently, therefore every invocation was “first”. Restoring test numbers resolves this issue back to original state: The number reported in the first column when executing the tests is its ordinal number in the Swift Benchmark Suite.
When listing benchmarks with `--list` parameter, present the tags in format that is actually accepted by the `--tags` and `--skip-tags` parameters.
Changes the `--list` output from
````
Enabled Tests,Tags
AngryPhonebook,[TestsUtils.BenchmarkCategory.validation, TestsUtils.BenchmarkCategory.api, TestsUtils.BenchmarkCategory.String]
...
````
into
````
Enabled Tests,Tags
AngryPhonebook,[String, api, validation]
…
````
Today, one can not completely disable a benchmark depending on the platform
without changing the source of main.swift. We would like to be able to disable
benchmarks locally in a benchmark's file without needing to modify the rest of
the infrastructure. The closest that one can get to such behavior is to just
conditionally compile out the file locally. But one still will have the test
run.
This commit adds support for not-running the benchmark on specific
platforms. This in combination with conditional compilation of benchmark bodies,
allows us to not have to comment out module's in main.swift or have to
conditionally compile testinfo.
rdar://40541972
Include the initial implementation of _StringGuts, a 2-word
replacement for _LegacyStringCore. 64-bit Darwin supported, 32-bit and
Linux support in subsequent commits.
The idea being, we need to decide what benchmarks to run solely based on
tags.
`--tag` allows to list all tags that are required;
`--skip-tags` allows to skip benchmarks that have any of those tags.
By default, skip-tags list contains .unstable and .String, which results
in the same subset of benchmarks as before.
The key thing here is that by providing one of these closures, a benchmark can
inject the initialization/deinitialization of its internal data structures,
outside of the time period where timing is occurring.
The intention is that this will provide us the framework for as we annotate
tests with BenchmarkInfo, to move initialization work out of benchmarks.
It will also allow for more complex benchmarks to be written such as ones that
perform bulk reads from a pipe (my interest in this).
Previously, we both filtered tests and marked tests as not being able to be run.
This commit rips out the marking of tests as being unable to be run (but counted
as part of the indices output by the compiler).
This also ensures benchmarks that are run are always given an ascending index
instead of sometimes having indices being skipped.
*NOTE* We always prefer a registered benchmark if we have one.
I am going to use BenchmarkInfo to solve the "create data for benchmark while we
are already timing" problem. I am going to add a field to BenchmarkInfo that if
it is not-null is called before we start measuring time. This closure can be
used to initialize any global data structures/etc.
But to do this, I need to be able to combine the registered and legacy
not-registered benchmarks.
SR-4780 Can not run performance tests that are not in precommit suite
Modified driver to honor command line arguments when listing enabled tests. Fixed interaction between filters (positional arguments) and --run-all option.
Benchmark_Driver lists available benchmarks with --run-all option when benchmarks or filters are specified.
From the Swift documentation:
"If you define an optional variable without providing a default value,
the variable is automatically set to nil for you."
* Migrate from `UnsafePointer<Void>` to `UnsafeRawPointer`.
As proposed in SE-0107: UnsafeRawPointer.
`void*` imports as `UnsafeMutableRawPointer`.
`const void*` imports as `UnsafeRawPointer`.
Occurrences of `UnsafePointer<Void>` are replaced with UnsafeRawPointer.
* Migrate overlays from UnsafePointer<Void> to UnsafeRawPointer.
This requires explicit memory binding in several places,
particularly in NSData and CoreAudio.
* Fix a bunch of test cases for Void->Raw migration.
* qsort takes IUO values
* Bridge `Unsafe[Mutable]RawPointer as `void [const] *`.
* Parse #dsohandle as UnsafeMutableRawPointer
* Update a bunch of test cases for Void->Raw migration.
* Trivial fix for the SceneKit test case.
* Add an UnsafeRawPointer self initializer.
This is unfortunately necessary for assignment between types imported from C.
* Tiny simplification of the initializer.
* Migrate from `UnsafePointer<Void>` to `UnsafeRawPointer`.
As proposed in SE-0107: UnsafeRawPointer.
`void*` imports as `UnsafeMutableRawPointer`.
`const void*` imports as `UnsafeRawPointer`.
Occurrences of `UnsafePointer<Void>` are replaced with UnsafeRawPointer.
* Migrate overlays from UnsafePointer<Void> to UnsafeRawPointer.
This requires explicit memory binding in several places,
particularly in NSData and CoreAudio.
* Fix a bunch of test cases for Void->Raw migration.
* qsort takes IUO values
* Bridge `Unsafe[Mutable]RawPointer as `void [const] *`.
* Parse #dsohandle as UnsafeMutableRawPointer
* Update a bunch of test cases for Void->Raw migration.
* Trivial fix for the SceneKit test case.
* Add an UnsafeRawPointer self initializer.
This is unfortunately necessary for assignment between types imported from C.
* Tiny simplification of the initializer.
