The `--help` option now prints standard usage description with documentaion for all arguments:
````
$ Benchmark_O --help
usage: Benchmark_O [--argument=VALUE] [TEST [TEST ...]]
positional arguments:
TEST name or number of the benchmark to measure
optional arguments:
--help show this help message and exit
--num-samples number of samples to take per benchmark; default: 1
--num-iters number of iterations averaged in the sample;
default: auto-scaled to measure for 1 second
--iter-scale number of seconds used for num-iters calculation
default: 1
--verbose increase output verbosity
--delim value delimiter used for log output; default: ,
--tags run tests matching all the specified categories
--skip-tags don't run tests matching any of the specified
categories; default: unstable,skip
--sleep number of seconds to sleep after benchmarking
--list don't run the tests, just log the list of test
numbers, names and tags (respects specified filters)
````
In case of invalid command line arguments, there is no reasonable recovery, the `ArgumentParser` can exit the program itself. It is therefore no longer necessary to propagate the `ArgumentError`s outside of the parser.
The `ArgumentParser` now has a configuration phase which specifies the supported arguments and their handling. The configured parser is then executed using the `parse` method which returns the parsed result.
Renamed `optionalArg` to `parseArg` because it can now also handle the positional arguments. Now all the command line arguments are handled through this function.
Minor naming improvement of the `filterTests` parameter.
The `TestConfig` is now completely immutable. Arguments that are not necessary for running benchmarks were moved inside the config initialization, which also subsumed the `printRunInfo` function for displaying the configuration details in verbose mode.
Moving towards immutable TestConfig.
The pattern used is inspired by the`PartialBenchmarkConfig` from [criterion.rs](https://github.com/japaric/criterion.rs/blob/master/src/benchmark.rs).
The idea is to have a temporary mutable struct which collects the command line arguments and is filled by the “parser” (which will be extracted later). The partial configuration is used to initialize the immutable `TestConfig`.
The optionality of the command line parameters is represented by the corresponding properties being `Optional`. (Accidentaly, all our arguments are optional… but that’s beside the point.) Null coalescing operator is used to fill in the defaults during initialization.
For some reason, the compiler found `optionalArg` calls for `tags` and `skip-tags` ambiguous, when types changed to `Set<BenchmarkCategory>?`, which was resolved by providing fully qualified key paths for them (`\PartialTestConfig.tags`).
Added handling for arguments that don’t have a value (flags), or whose values are optional (they can be empty). This covers all the argument types currently used in Benchmark_O with the single `optionalArg` function.
Refactored to use Swift’s idiomatic error handling.
In case of invalid argument errors, the message is printed to `stderr` and we exit gracefully with error code 1. We no longer crash the app in most cases.
* Fix: flushing stdout before crashing to enable testing.
* Added tests that verify reporting of errors when the parsing of command line arguments fails.
Measure and report system environment indicators during benchmark execution:
* Memory usage with maximum resident set size (MAX_RSS) in bytes
Proxy indicators of system load level:
* Number of Involuntary Context Switches (ICS)
* Number of Voluntary Context Switches (VCS)
MAX_RSS delta is always reported in the last column of the log report.
The `--verbose` mode additionaly reports full values measured before and after the benchmark execution as well as their delta for MAX_RSS, ICS and VCS.
The indices (test numbers) are strings on both ends of the IO — in user input as well as when printed to the console. We can spare few conversions and just store them directly as `String`.
Moved the formatting of`BenchmarkResults` into `runBenchmarks` function which already contained the logging of header and the special case of unsupported benchmark.
Report only the total number of executed tests.
Aggregating MIN, MAX and MEAN values for all executed benchmarks together (with microsecond precision!) has no statistical relevance.
Reintroduced feature lost during `BenchmarkInfo` modernization: All registered benchmarks are ordered alphabetically and assigned an index. This number can be used as a shortcut to invoke the test instead of its full name. (Adding and removing tests from the suite will naturally reassign the indices, but they are stable for a given build.)
The `--list` parameter now prints the test *number*, *name* and *tags* separated by delimiter.
The `--list` output format is modified from:
````
Enabled Tests,Tags
AngryPhonebook,[String, api, validation]
...
````
to this:
````
\#,Test,[Tags]
2,AngryPhonebook,[String, api, validation]
…
````
(There isn’t a backslash before the #, git was eating the whole line without it.)
Note: Test number 1 is Ackermann, which is marked as “skip”, so it’s not listed with the default `skip-tags` value.
Fixes the issue where running tests via `Benchmark_Driver` always reported each test as number 1. Each test is run independently, therefore every invocation was “first”. Restoring test numbers resolves this issue back to original state: The number reported in the first column when executing the tests is its ordinal number in the Swift Benchmark Suite.
When listing benchmarks with `--list` parameter, present the tags in format that is actually accepted by the `--tags` and `--skip-tags` parameters.
Changes the `--list` output from
````
Enabled Tests,Tags
AngryPhonebook,[TestsUtils.BenchmarkCategory.validation, TestsUtils.BenchmarkCategory.api, TestsUtils.BenchmarkCategory.String]
...
````
into
````
Enabled Tests,Tags
AngryPhonebook,[String, api, validation]
…
````
Today, one can not completely disable a benchmark depending on the platform
without changing the source of main.swift. We would like to be able to disable
benchmarks locally in a benchmark's file without needing to modify the rest of
the infrastructure. The closest that one can get to such behavior is to just
conditionally compile out the file locally. But one still will have the test
run.
This commit adds support for not-running the benchmark on specific
platforms. This in combination with conditional compilation of benchmark bodies,
allows us to not have to comment out module's in main.swift or have to
conditionally compile testinfo.
rdar://40541972
Include the initial implementation of _StringGuts, a 2-word
replacement for _LegacyStringCore. 64-bit Darwin supported, 32-bit and
Linux support in subsequent commits.
The idea being, we need to decide what benchmarks to run solely based on
tags.
`--tag` allows to list all tags that are required;
`--skip-tags` allows to skip benchmarks that have any of those tags.
By default, skip-tags list contains .unstable and .String, which results
in the same subset of benchmarks as before.
