Turns out that both the old code in `DriverUtils` that computed median, as well as newer quartiles in `PerformanceTestSamples` had off-by-1 error.
It trully is the 3rd of the 2 hard things in computer science!
Improve conformance to Swift Naming Guidelines
https://swift.org/documentation/api-design-guidelines/#parameter-names
Removed the gross undescores and ticks from parameter names. Ticks are ectoparasites feeding on the blood. We are just measuring time and there is also no [mysterious ticking noise](http://bit.ly/TickNoise) here either…
It pays off of to be a nice process and yield the processor at regular intervals, to prevent having measured samples corrupted by preemptive multitasking.
When the scheduled time slice (10ms on Mac OS) is probably about to expire during the next measurement, we voluntarily yield and resume the measurement within a fresh scheduler quantum.
This cooperative approach to multitasking improves the sample quality and robustness of the measurement process.
Clean up after removing bogus agregate statistics from last line of the log. It makes more sense to report the total number of executed benchmarks as a sentence that trying to fit into the format of preceding table.
Added test assertion that `run_benchmarks` return csv formatted log, as it is used to write the log into file in `log_results`.
This benchmark was added to test if the compiler crashes.
For some reason it was added as benchmark and not as lit test.
It has no value as benchmark anyway because the compiler optimizes away pretty much everything.
The `--help` option now prints standard usage description with documentaion for all arguments:
````
$ Benchmark_O --help
usage: Benchmark_O [--argument=VALUE] [TEST [TEST ...]]
positional arguments:
TEST name or number of the benchmark to measure
optional arguments:
--help show this help message and exit
--num-samples number of samples to take per benchmark; default: 1
--num-iters number of iterations averaged in the sample;
default: auto-scaled to measure for 1 second
--iter-scale number of seconds used for num-iters calculation
default: 1
--verbose increase output verbosity
--delim value delimiter used for log output; default: ,
--tags run tests matching all the specified categories
--skip-tags don't run tests matching any of the specified
categories; default: unstable,skip
--sleep number of seconds to sleep after benchmarking
--list don't run the tests, just log the list of test
numbers, names and tags (respects specified filters)
````
The `parseArgs` funtion is now a private method on `ArgumentParser`.
Removed `Arguments` struct and moved the `Argument` as a nested struct into the parser.
Adjusted error messages and the corresponding checks.
In case of invalid command line arguments, there is no reasonable recovery, the `ArgumentParser` can exit the program itself. It is therefore no longer necessary to propagate the `ArgumentError`s outside of the parser.
The `ArgumentParser` now has a configuration phase which specifies the supported arguments and their handling. The configured parser is then executed using the `parse` method which returns the parsed result.
Renamed `optionalArg` to `parseArg` because it can now also handle the positional arguments. Now all the command line arguments are handled through this function.
Minor naming improvement of the `filterTests` parameter.
The `TestConfig` is now completely immutable. Arguments that are not necessary for running benchmarks were moved inside the config initialization, which also subsumed the `printRunInfo` function for displaying the configuration details in verbose mode.
Moving towards immutable TestConfig.
The pattern used is inspired by the`PartialBenchmarkConfig` from [criterion.rs](https://github.com/japaric/criterion.rs/blob/master/src/benchmark.rs).
The idea is to have a temporary mutable struct which collects the command line arguments and is filled by the “parser” (which will be extracted later). The partial configuration is used to initialize the immutable `TestConfig`.
The optionality of the command line parameters is represented by the corresponding properties being `Optional`. (Accidentaly, all our arguments are optional… but that’s beside the point.) Null coalescing operator is used to fill in the defaults during initialization.
For some reason, the compiler found `optionalArg` calls for `tags` and `skip-tags` ambiguous, when types changed to `Set<BenchmarkCategory>?`, which was resolved by providing fully qualified key paths for them (`\PartialTestConfig.tags`).
Added handling for arguments that don’t have a value (flags), or whose values are optional (they can be empty). This covers all the argument types currently used in Benchmark_O with the single `optionalArg` function.
Refactored to use Swift’s idiomatic error handling.
In case of invalid argument errors, the message is printed to `stderr` and we exit gracefully with error code 1. We no longer crash the app in most cases.
