averello/swift-mirror
1
0
Fork 0
mirror of https://github.com/apple/swift.git synced 2025-12-14 20:36:38 +01:00
Code Issues Packages Projects Releases Wiki Activity
184,973 Commits 622 Branches 2,942 Tags
maxd/async-build-script
Commit Graph

492 Commits

Author SHA1 Message Date
Karoy Lorentey
f93dcf3dfa [benchmark] Add benchmark for [AnyHashable: Any] with String keys 2018-10-26 11:56:41 +01:00
Patrick Balestra
1c0778bb5b [benchmark] Add insert(_:Character) benchmark with ASCII and non-ASCII characters 
Adds insert character benchmark with ASCII and non-ASCII characters
 2018-10-24 18:55:20 -07:00
Pavol Vaskovic
67b489dcb1 [benchmark] Auto-determine number of samples 
When measuring with specified number of iterations (generally, `--num-iters=1` makes sense), automaticially determine the number of samples to take, so that the overall measurement duration comes close to `sample-time`.

This is the same technique used to scale `num-iters` before, but for `num-samples`.
 2018-10-11 18:56:27 +02:00
Pavol Vaskovic
9d9200e9eb [benchmark] Measure setUpFunction 
Measure the duration of the `setUpFunction` and report it in verbose mode.

This will be used by `BenchmarkDoctor`, to ensure there isn’t unreasonably big imbalance between the time it takes to set up and run the actual benchmark.
 2018-10-02 14:34:43 +02:00
Pavol Vaskovic
a9f0ce4338 [benchmark] Fix quantile estimation type 
The correct quantile estimation type for printing all measurements in the summary report while `quantile == num-samples - 1` is R-1, SAS-3.  It's the inverse of empirical distribution function.

References:
* https://en.wikipedia.org/wiki/Quantile#Estimating_quantiles_from_a_sample
* discussion in https://github.com/apple/swift/pull/19097#issuecomment-421238197
 2018-09-20 09:19:07 +02:00
Pavol Vaskovic
f0e7b8737a [benchmark] Round quantile idx to nearest or even 
Explicitly use round-half-to-even rounding algorithm to match the behavior of numpy's quantile(interpolation='nearest') and quantile estimate type R-3, SAS-2. See:
https://en.wikipedia.org/wiki/Quantile#Estimating_quantiles_from_a_sample
 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
8b3b1f695a [benchmark] Option: delta encoded quantiles format 
Added `--delta` argument to print the quantiles in delta encoded format, that ommits 0s.

This results in machine and human readable output that highlights modes and is easily digestible, giving you the feel for the underlying probability distribution of the samples in the reported results:

````
$ ./Benchmark_O --num-iters=1 --num-samples=20 --quantile=20 --delta 170 171 184 185 198 199 418 419 432 433 619 620
#,TEST,SAMPLES,MIN(μs),𝚫V1,𝚫V2,𝚫V3,𝚫V4,𝚫V5,𝚫V6,𝚫V7,𝚫V8,𝚫V9,𝚫VA,𝚫VB,𝚫VC,𝚫VD,𝚫VE,𝚫VF,𝚫VG,𝚫VH,𝚫VI,𝚫VJ,𝚫MAX
170,DropFirstArray,20,171,,,,,,,,,,,,,,,,,,,2,29
171,DropFirstArrayLazy,20,168,,,,,,,,,,,,,,,,,,,,8
184,DropLastArray,20,55,,,,,,,,,,,,,,,,,,,,26
185,DropLastArrayLazy,20,65,,,,,,,,,,,,,,,,,,,1,90
198,DropWhileArray,20,214,1,,,,,,,,,,,,,,,,,1,27,2
199,DropWhileArrayLazy,20,464,,,,1,,,,,,,,1,1,1,4,9,1,9,113,2903
418,PrefixArray,20,132,,,,,,,,,,,,,,,,,1,1,32,394
419,PrefixArrayLazy,20,168,,,,,,,,,,,,1,,2,9,1,15,8,88,3338
432,PrefixWhileArray,20,252,1,,,,1,,,,,,,,,,,1,,,,30
433,PrefixWhileArrayLazy,20,168,,,,,,,,,,,,,1,,6,6,14,43,28,10200
619,SuffixArray,20,68,,,,,,,,,,,,,1,,,,22,1,1,4
620,SuffixArrayLazy,20,65,,,,,,,,,,,,,,,,,,1,9,340
````
 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
72e960457b [benchmark] Gardening maxRSS as Int? 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
022e1111a9 [benchmark] Report quantiles from samples 
The default benchmark result reports statistics of a normal distribution — mean and standard deviation. Unfortunately the samples from our benchmarks are *not normally distributed*. To get a better picture of the underlying probability distribution, this adds support for reporting quantiles.

See https://en.wikipedia.org/wiki/Quantile

This gives better subsample of the measurements in the summary, without need to resort to the use of a full verbose mode, which might be unnecessarily slow.
 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
219a5d9290 [benchmark] Rename SampleRunner -> TestRunner 
It is now running all the benchmarks, so it’s a TestRunner.
 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
0e751e2717 [benchmark] Gardening: Even nicer microseconds 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
d704557c88 [benchmark] Gardening: Fixed method indentation 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
12c6e39a20 [benchmark] Refactor run runBenchmarks logVerbose 
Extracted nested func logVerbose as instance method on SampleRunner.

Internalized the free functions `runBech` and `runBenchmarks` into SampleRunner as methods `run` and `runBenchmarks`.
 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
e7d1d482d8 [benchmark] Extract yield & add resetMeasurements 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
331c0bf772 [benchmark] Refactor numIters computation 
The spaghetti if-else code was untangled into nested function that computes `iterationsPerSampleTime` and a single constant `numIters` expression that takes care of the overflow capping as well as the choice between fixed and computed `numIters` value.

The `numIters` is now computed and logged only once per benchmark measurement instead of on every sample.

The sampling loop is now just a single line. Hurrah!

Modified test to verify that the `LogParser` maintains `num-iters` derived from the `Measuring with scale` message across samples.
 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
29b2cc7397 [benchmark] Refactor sampling loop with addSample 
Extracted sample saving to inner func `addSample`.
Used it to save the `oneIter` sample from `numIters` calibration when it comes out as 1 and continue the for loop to next sample.

This simplified following code that can now always measure the sample with `numIters` and save it.
 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
b762f80a64 [benchmark] Gardening: Documentation of numIters 
Clarified the need for capping `numIters` according to the discussion at https://github.com/apple/swift/pull/17268#issuecomment-404831035

The sampling loop is a hairy piece of code, because it’s trying to reuse the calibration measurement as a regular sample, in case the computed `numIters` turns out to be 1. But it conflicts with the case when `fixedNumIters` is 1, necessitating a separate measurement in the else branch… That was a quick fix back then, but its hard to make it clean. More thinking is required…
 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
75604a285d [benchmark] Gardening: Sensibly rename variables 
To make sense of this spaghetti code, let’s first use reasonable variable names:
* scale -> numIters
* elapsed_time -> time
 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
a169606e60 [benchmark] Gardening: DRYer verbose log 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
9ae69908b0 [benchmark] Refactor to currency type Int 
Removed unnecessary use of UInt64, where appropriate, following the advice from Swift Language Guide:

> Use the `Int` type for all general-purpose integer constants and variables in your code, even if they’re known to be nonnegative. Using the default integer type in everyday situations means that integer constants and variables are immediately interoperable in your code and will match the inferred type for integer literal values.
https://docs.swift.org/swift-book/LanguageGuide/TheBasics.html#ID324
 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
28eb79819b [benchmark] Refactor to report samples in μs 
Moved the adjustment of `lastSampleTime` to account for the `scale` (`numIters`) and conversion to microseconds into SampleRunner’s `measure` method.
 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
beabad86f4 [benchmark] Gardening: scale was always Int 
Since the `scale` (or `numIters`) is passed to the `test.runFunction` as `Int`, the whole type-casting dance here was just silly!
 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
e775b8fc60 [benchmark] Gardening: numSamples UInt vs Int 
Type check command line argument to be non-negative, but store value in currency type `Int`.
 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
79d7730be8 [benchmark] Gardening: afterRunSleep is UInt32 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
7768cb3295 [benchmark] Move stats computation to BenchResults 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
e48b5fdb34 [benchmark] Fix index computation for quantiles 
Turns out that both the old code in `DriverUtils` that computed median, as well as newer quartiles in `PerformanceTestSamples` had off-by-1 error.

It trully is the 3rd of the 2 hard things in computer science!
 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
ab3e6122c0 [benchmark] Refactor min max median computation 
We can spare 2 array passes (for min and max), if we just sort first.
 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
ed53fe8f71 [benchmark] Refactor mean and stdev computation 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
1a0a7b3d0f [benchmark] Gardening: Timer Parasite Control 
Improve conformance to Swift Naming Guidelines
https://swift.org/documentation/api-design-guidelines/#parameter-names

Removed the gross undescores and ticks from parameter names. Ticks are ectoparasites feeding on the blood. We are just measuring time and there is also no [mysterious ticking noise](http://bit.ly/TickNoise) here either…
 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
7bbf2e1d0a [benchmark] Gardening: Extract constant oneSecond 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
1ba2ee41d9 [benchmark] Gardening: Code format class Timer 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
496a277419 [benchmark] Gardening: Indentation of .listTests 2018-09-14 23:40:43 +02:00
Ben Langmuir
423e145b0c Revert "[benchmark] Report Quantiles from Benchmark_O and a TON of Gardening" 2018-09-14 13:24:01 -07:00
swift-ci
8ce03faa76 Merge pull request #19097 from palimondo/fluctuation-of-the-pupil 2018-09-14 09:24:00 -07:00
Soroush Khanlou
5736cacc9a Add count(where:) and tests (#16099) 
* add count(where:) and tests

* Revise count(where:) documentation

* Remove errant word in abstract

* add a benchmark for ranges and strings with help from @natecook1000

* update benchmark to use Array instead of Range
 2018-09-13 12:37:06 -05:00
Pavol Vaskovic
a56c55c8e4 [benchmark] Round quantile idx to nearest or even 
Explicitly use round-half-to-even rounding algorithm to match the behavior of numpy's quantile(interpolation='nearest') and quantile estimate type R-3, SAS-2. See:
https://en.wikipedia.org/wiki/Quantile#Estimating_quantiles_from_a_sample
 2018-09-10 10:45:00 +02:00
Pavol Vaskovic
313dfda5a4 [benchmark] Option: delta encoded quantiles format 
Added `--delta` argument to print the quantiles in delta encoded format, that ommits 0s.

This results in machine and human readable output that highlights modes and is easily digestible, giving you the feel for the underlying probability distribution of the samples in the reported results:

````
$ ./Benchmark_O --num-iters=1 --num-samples=20 --quantile=20 --delta 170 171 184 185 198 199 418 419 432 433 619 620
#,TEST,SAMPLES,MIN(μs),𝚫V1,𝚫V2,𝚫V3,𝚫V4,𝚫V5,𝚫V6,𝚫V7,𝚫V8,𝚫V9,𝚫VA,𝚫VB,𝚫VC,𝚫VD,𝚫VE,𝚫VF,𝚫VG,𝚫VH,𝚫VI,𝚫VJ,𝚫MAX
170,DropFirstArray,20,171,,,,,,,,,,,,,,,,,,,2,29
171,DropFirstArrayLazy,20,168,,,,,,,,,,,,,,,,,,,,8
184,DropLastArray,20,55,,,,,,,,,,,,,,,,,,,,26
185,DropLastArrayLazy,20,65,,,,,,,,,,,,,,,,,,,1,90
198,DropWhileArray,20,214,1,,,,,,,,,,,,,,,,,1,27,2
199,DropWhileArrayLazy,20,464,,,,1,,,,,,,,1,1,1,4,9,1,9,113,2903
418,PrefixArray,20,132,,,,,,,,,,,,,,,,,1,1,32,394
419,PrefixArrayLazy,20,168,,,,,,,,,,,,1,,2,9,1,15,8,88,3338
432,PrefixWhileArray,20,252,1,,,,1,,,,,,,,,,,1,,,,30
433,PrefixWhileArrayLazy,20,168,,,,,,,,,,,,,1,,6,6,14,43,28,10200
619,SuffixArray,20,68,,,,,,,,,,,,,1,,,,22,1,1,4
620,SuffixArrayLazy,20,65,,,,,,,,,,,,,,,,,,1,9,340
````
 2018-09-03 16:00:05 +02:00
Pavol Vaskovic
13e7c3faaf [benchmark] Gardening maxRSS as Int? 2018-09-02 11:47:43 +02:00
Pavol Vaskovic
1f465b9bf7 [benchmark] Report quantiles from samples 
The default benchmark result reports statistics of a normal distribution — mean and standard deviation. Unfortunately the samples from our benchmarks are *not normally distributed*. To get a better picture of the underlying probability distribution, this adds support for reporting quantiles.

See https://en.wikipedia.org/wiki/Quantile

This gives better subsample of the measurements in the summary, without need to resort to the use of a full verbose mode, which might be unnecessarily slow.
 2018-08-31 23:16:34 +02:00
Pavol Vaskovic
6079d4ff03 [benchmark] Rename SampleRunner -> TestRunner 
It is now running all the benchmarks, so it’s a TestRunner.
 2018-08-31 18:05:26 +02:00
Pavol Vaskovic
ae7d82b607 [benchmark] Gardening: Even nicer microseconds 2018-08-31 18:05:20 +02:00
Pavol Vaskovic
df3b38589e [benchmark] Gardening: Fixed method indentation 2018-08-31 17:25:16 +02:00
Pavol Vaskovic
cdcb631469 [benchmark] Refactor run runBenchmarks logVerbose 
Extracted nested func logVerbose as instance method on SampleRunner.

Internalized the free functions `runBech` and `runBenchmarks` into SampleRunner as methods `run` and `runBenchmarks`.
 2018-08-31 17:17:48 +02:00
Pavol Vaskovic
265f537d10 [benchmark] Extract yield & add resetMeasurements 2018-08-31 17:17:48 +02:00
Pavol Vaskovic
be39c02001 [benchmark] Refactor numIters computation 
The spaghetti if-else code was untangled into nested function that computes `iterationsPerSampleTime` and a single constant `numIters` expression that takes care of the overflow capping as well as the choice between fixed and computed `numIters` value.

The `numIters` is now computed and logged only once per benchmark measurement instead of on every sample.

The sampling loop is now just a single line. Hurrah!

Modified test to verify that the `LogParser` maintains `num-iters` derived from the `Measuring with scale` message across samples.
 2018-08-31 17:17:48 +02:00
Pavol Vaskovic
46ee2a4bd8 [benchmark] Refactor sampling loop with addSample 
Extracted sample saving to inner func `addSample`.
Used it to save the `oneIter` sample from `numIters` calibration when it comes out as 1 and continue the for loop to next sample.

This simplified following code that can now always measure the sample with `numIters` and save it.
 2018-08-31 07:32:23 +02:00
Pavol Vaskovic
3c55e30382 [benchmark] Gardening: Documentation of numIters 
Clarified the need for capping `numIters` according to the discussion at https://github.com/apple/swift/pull/17268#issuecomment-404831035

The sampling loop is a hairy piece of code, because it’s trying to reuse the calibration measurement as a regular sample, in case the computed `numIters` turns out to be 1. But it conflicts with the case when `fixedNumIters` is 1, necessitating a separate measurement in the else branch… That was a quick fix back then, but its hard to make it clean. More thinking is required…
 2018-08-31 07:32:22 +02:00
Pavol Vaskovic
6e27af7142 [benchmark] Gardening: Sensibly rename variables 
To make sense of this spaghetti code, let’s first use reasonable variable names:
* scale -> numIters
* elapsed_time -> time
 2018-08-31 07:32:21 +02:00
Pavol Vaskovic
46bef893d0 [benchmark] Gardening: DRYer verbose log 2018-08-31 07:32:20 +02:00
Pavol Vaskovic
9c4876eabd [benchmark] Refactor to currency type Int 
Removed unnecessary use of UInt64, where appropriate, following the advice from Swift Language Guide:

> Use the `Int` type for all general-purpose integer constants and variables in your code, even if they’re known to be nonnegative. Using the default integer type in everyday situations means that integer constants and variables are immediately interoperable in your code and will match the inferred type for integer literal values.
https://docs.swift.org/swift-book/LanguageGuide/TheBasics.html#ID324
 2018-08-31 07:32:19 +02:00