setUpFunction code should be located together with the corresponding runFunction code, so that we can verify them at a glance. However, we can't really do that yet, and doing the setUp for all tests in bulk makes it less likely that we make a mistake in matching up tests with their corresponding setups. :/
Disable the random hash seed while benchmarking. By its nature, it makes the number of hash collisions fluctuate between runs, adding unnecessary noise to benchmark results.
I expect we'll be able to re-enable random seeding here once we have made hash collisions cheaper -- they are currently always resolved by calling the Key's Equatable implementation, which can be expensive.
WordCount shows significant variance between iterations, which is apparently caused by the initialization of someAlphanumerics and asciiWords/utf16Words leaking into the measured part of the benchmark.
Making sure these variables are initialized before we start measuring elapsed time stabilizes the results.
This benchmark script is similar to the guard malloc/runtime runner, but it only
runs the tests. The intention is that one can use this to quickly in a
multithreaded way verify that all benchmarks run successfully. In contrast, the
normal driver will run only single threaded since it is meant to test
performance, so is not able to take advantage of all cores on a system.
I wrote this quickly to verify some benchmark tests still worked. No point in
not sharing with everyone else.
* Benchmark {Float,Double,Float80}.description
This just tests how fast we can format the standard
floating-point types. It also includes a test that
uses the result, to ensure that future optimized
_creation_ of the string doesn't incidentally pessimize
_use_ of the results.
* Benchmark {Float,Double,Float80}.description
This just tests how fast we can format the standard
floating-point types. It also includes a test that
uses the result, to ensure that future optimized
_creation_ of the string doesn't incidentally pessimize
_use_ of the results.
* Add benchmarks for values close to 1 + other review suggestions
Dictionary and Set currently exhibit O(n^2) behavior for certain operations involving copying elements in bulk. Add benchmarks to verify an upcoming fix and to catch regressions later.
https://bugs.swift.org/browse/SR-3268
This is necessary to build standalone benchmarks with leak checking
enabled. This is useful if you want to debug an internal benchmark failure using
a public swift.
We used to have this cmake flag but it seems to have disappeared. Building the
benchmarks with different swiftc flags is central to performance analysis, so
I'm not sure how people were getting this done.
This takes more than 15 minutes to compile and makes the benchmark suite
unusable. There are multiple severe compile issues that need to be fixed
properly before we can support the DoubleWidth API.
See [SR-6947] DoubleWidth compile time.
On the bots, we have a timeout without output of 60 minutes for the entire test.
This should ensure that we are able to kill mis-behaving tests and give a good
error instead of just getting a jenkins timeout error.
For those confused, this is for the guard malloc/leaks test.
rdar://36874229
