The Swift benchmarking harness now has two distinct output formats:
* Default: Formatted text that's intended for human consumption.
Right now, this is just the minimum value, but we can augment that.
* `--json`: each output line is a JSON-encoded object that contains raw data
This information is intended for use by python scripts that aggregate
or compare multiple independent tests.
Previously, we tried to use the same output for both purposes. This required
the python scripts to do more complex parsing of textual layouts, and also meant
that the python scripts had only summary data to work with instead of full raw
sample information. This in turn made it almost impossible to derive meaningful
comparisons between runs or to aggregate multiple runs.
Typical output in the new JSON format looks like this:
```
{"number":89, "name":"PerfTest", "samples":[1.23, 2.35], "max_rss":16384}
{"number":91, "name":"OtherTest", "samples":[14.8, 19.7]}
```
This format is easy to parse in Python. Just iterate over
lines and decode each one separately. Also note that the
optional fields (`"max_rss"` above) are trivial to handle:
```
import json
for l in lines:
j = json.loads(l)
# Default 0 if not present
max_rss = j.get("max_rss", 0)
```
Note the `"samples"` array includes the runtime for each individual run.
Because optional fields are so much easier to handle in this form, I reworked
the Python logic to translate old formats into this JSON format for more
uniformity. Hopefully, we can simplify the code in a year or so by stripping
out the old log formats entirely, along with some of the redundant statistical
calculations. In particular, the python logic still makes an effort to preserve
mean, median, max, min, stdev, and other statistical data whenever the full set
of samples is not present. Once we've gotten to a point where we're always
keeping full samples, we can compute any such information on the fly as needed,
eliminating the need to record it.
This is a pretty big rearchitecture of the core benchmarking logic. In order to
try to keep things a bit more manageable, I have not taken this opportunity to
replace any of the actual statistics used in the higher level code or to change
how the actual samples are measured. (But I expect this rearchitecture will make
such changes simpler.) In particular, this should not actually change any
benchmark results.
For the future, please keep this general principle in mind: Statistical
summaries (averages, medians, etc) should as a rule be computed for immediate
output and rarely if ever stored or used as input for other processing. Instead,
aim to store and transfer raw data from which statistics can be recomputed as
necessary.
The `getFalse` utility function was not excluded from cross-module-optimization, which led the optimizer to completely eliminate the main loop body
Also, the passed `N` was shadowed by a local `n`.
* [SILOptimizer] Add prespecialization for arbitray reference types
* Fix benchmark Package.swift
* Move SimpleArray to utils
* Fix multiple indirect result case
* Remove leftover code from previous attempt
* Fix test after rebase
* Move code to compute type replacements to SpecializedFunction
* Fix ownership when OSSA is enabled
* Fixes after rebase
* Changes after rebasing
* Add feature flag for layout pre-specialization
* Fix pre_specialize-macos.swift
* Add compiler flag to benchmark build
* Fix benchmark SwiftPM flags
The logic here was apparently intended to omit literal zeros from deltas
to save a few bytes, but it instead drops all zeros from all columns.
Remove the condition that drops zeros in order to avoid confusing
the many scripts that consume this data.
Alternatives Considered
I'm probably going to entirely drop the delta form in an upcoming
PR, so I didn't think it was worthwhile to do something more complex,
such as:
* Fixing this logic to only omit zeros from actual delta columns
* Rewriting all the client scripts to treat any empty column as zero
* Add benchmarks that measure KeyPath read and write performance.
* Added setUpFunctions. Revised number of iterations per benchmark.
* Include n as a factor in the number of iterations.
* Increased number of iterations for KeyPathDirectAccess by a factor of 25.
* One last tweak to the number of iterations on testDirectAccess to get them above 20 us.
* Made revisions based on feedback. Added three new benchmarks.
* Added benchmarks to exhaustively benchmark all KeyPathComponent types. Removed benchmarks dealing with an inlining issue.
* Wrapped additional keypaths with identity() where needed. More cleanup and documentation.
* Moved KeyPaths for KeyPathRead and Write into FixedSizeArrayHolder. Renamed GetSet to Getset.
* Added inline(never) to both versions of getKeypathToElement().
* Moved identity() wraps so that they're called once per variable per benchmark.
* Moving destinationKeyPaths into FixedSizeArrayHolder to try to reduce long setup overhead errors.
* Additional moving of the identity() wrapping into the singleton's init() to try to reduce setup time errors.
The script defaulted to a mode that no one uses without checking
whether the input was compatible with that mode.
This is the script used for run-to-run comparison of benchmark
results. The in-tree benchmarks happened to work with the script only
because of a fragile string comparison burried deep within the
script. Other out-of-tree benchmark scripts that generate results were
silently broken when using this script for comparison.
Function bodies of `blackHole`, `identity`, etc. must not be visible in the benchmark modules.
Enabling CMO by default broke this. Since then we need to explicitly exclude those functions from cross-module-optimization.
