averello/swift-mirror
1
0
Fork 0
mirror of https://github.com/apple/swift.git synced 2025-12-21 12:14:44 +01:00
Code Issues Packages Projects Releases Wiki Activity
104,175 Commits 623 Branches 2,949 Tags
103630296d3ebcc41d7c48f0ab34b697efbfb51a
Commit Graph

408 Commits

Author SHA1 Message Date
David Smith
f85e1fee22 Generalize the JSON perf tests to Codable 2018-12-03 11:16:47 -08:00
David Smith
c129066b39 Add simple benchmarks for JSON coding 2018-11-29 17:41:50 -08:00
Karoy Lorentey
74eaea2a6a [benchmark] Add some benchmarks for String breadcrumbs (#20769) 
[benchmark] Add some benchmarks for String breadcrumbs
 2018-11-28 16:24:29 -08:00
Pavol Vaskovic
0bdd3ef275 [benchmark] Equalize memory usage (w&w/o verbose) 
The use of `--verbose` parameter was affecting the reported memory usage (`--memory`), because it front-loads initialization of string interpolation and printing.

By always computing the configuration string and always calling print, the baseline memory measurement no longer includes this constant overhead.
 2018-11-28 21:34:12 +01:00
Andreas Jönsson
09768f29c5 [benchmark] fixes index(of:) deprecation warnings in ArgParse 2018-11-27 22:57:22 +01:00
swift-ci
f093925036 Merge pull request #20452 from Catfish-Man/stringtrampolines 2018-11-15 17:55:34 -08:00
David Smith
8bb6b78e36 Use super trampolines to let us override several more NSString methods (-UTF8String, -cStringUsingEncoding:, and -getCString:maxLength:encoding:) for performance 2018-11-15 17:10:51 -08:00
Maxim Moiseev
cbf83ac04f [NFC][stdlib] Add FIXME markers to simplify audit 2018-11-14 11:58:42 -08:00
Slava Pestov
f6c2caf64b stdlib: Add @inlinable to @inline(__always) declarations 
These should be audited since some might not actually need to be
@inlinable, but for now:

- Anything public and @inline(__always) is now also @inlinable
- Anything @usableFromInline and @inline(__always) is now @inlinable
 2018-11-13 15:15:07 -05:00
David Smith
634e6bc584 Add benchmarks for various ObjC NSString operations on bridged Swift Strings 2018-11-06 13:43:24 -08:00
eeckstein
f64f02bfde Merge pull request #20212 from palimondo/fluctuation-of-the-pupil 
[benchmark] Legacy Factor
 2018-11-06 11:31:08 -08:00
Lance Parker
7376009ccc Add benchmarks and tests for the normalized iterator (#32) 
Add benchmarks and tests for the normalized iterator
 2018-11-04 10:42:41 -08:00
Pavol Vaskovic
a7f832fb57 [benchmark] Legacy factor 
This adds optional `legacyFactor` to the `BenchmarkInfo`, which allows for linear modification of constants that unnecesarily inflate the base workload of benchmarks, while maintaining the continuity of log-term benchmark tracking.

For example, if a benchmark uses `for _ in N*10_000` in its run function, we could lower this to `for _ in N*1_000` and adding a `legacyFactor: 10` to its `BenchmarkInfo`.

Note that this doesn’t affect the real measurements gathered from the `--verbose` output. The `BenchmarkDoctor` has been slightly adjusted to work with these real samples, therefore `Benchmark_Driver check` will not flag these benchmarks for slow run time reported in the summary, if their real runtimes fall into the recommended range.
 2018-11-01 06:24:27 +01:00
eeckstein
cd920b69f4 Merge pull request #19910 from palimondo/fluctuation-of-the-pupil 
[benchmark] More Robust Benchmark_Driver
 2018-10-29 15:02:07 -07:00
Michael Gottesman
ba7815b663 [benchmark] Fix swiftpm based benchmark build on Linux. 2018-10-29 12:15:20 -07:00
Michael Gottesman
4d76ff9681 [benchmark] Add two benchmarks that show performance of flattening an array. 
The first is a naive imperative approach using appends in a loop. The second
uses flatMap. We would like both of these to have equivalent performance.
 2018-10-28 15:55:26 -07:00
Karoy Lorentey
f93dcf3dfa [benchmark] Add benchmark for [AnyHashable: Any] with String keys 2018-10-26 11:56:41 +01:00
Patrick Balestra
1c0778bb5b [benchmark] Add insert(_:Character) benchmark with ASCII and non-ASCII characters 
Adds insert character benchmark with ASCII and non-ASCII characters
 2018-10-24 18:55:20 -07:00
Pavol Vaskovic
67b489dcb1 [benchmark] Auto-determine number of samples 
When measuring with specified number of iterations (generally, `--num-iters=1` makes sense), automaticially determine the number of samples to take, so that the overall measurement duration comes close to `sample-time`.

This is the same technique used to scale `num-iters` before, but for `num-samples`.
 2018-10-11 18:56:27 +02:00
Pavol Vaskovic
9d9200e9eb [benchmark] Measure setUpFunction 
Measure the duration of the `setUpFunction` and report it in verbose mode.

This will be used by `BenchmarkDoctor`, to ensure there isn’t unreasonably big imbalance between the time it takes to set up and run the actual benchmark.
 2018-10-02 14:34:43 +02:00
Pavol Vaskovic
a9f0ce4338 [benchmark] Fix quantile estimation type 
The correct quantile estimation type for printing all measurements in the summary report while `quantile == num-samples - 1` is R-1, SAS-3.  It's the inverse of empirical distribution function.

References:
* https://en.wikipedia.org/wiki/Quantile#Estimating_quantiles_from_a_sample
* discussion in https://github.com/apple/swift/pull/19097#issuecomment-421238197
 2018-09-20 09:19:07 +02:00
Pavol Vaskovic
f0e7b8737a [benchmark] Round quantile idx to nearest or even 
Explicitly use round-half-to-even rounding algorithm to match the behavior of numpy's quantile(interpolation='nearest') and quantile estimate type R-3, SAS-2. See:
https://en.wikipedia.org/wiki/Quantile#Estimating_quantiles_from_a_sample
 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
8b3b1f695a [benchmark] Option: delta encoded quantiles format 
Added `--delta` argument to print the quantiles in delta encoded format, that ommits 0s.

This results in machine and human readable output that highlights modes and is easily digestible, giving you the feel for the underlying probability distribution of the samples in the reported results:

````
$ ./Benchmark_O --num-iters=1 --num-samples=20 --quantile=20 --delta 170 171 184 185 198 199 418 419 432 433 619 620
#,TEST,SAMPLES,MIN(μs),𝚫V1,𝚫V2,𝚫V3,𝚫V4,𝚫V5,𝚫V6,𝚫V7,𝚫V8,𝚫V9,𝚫VA,𝚫VB,𝚫VC,𝚫VD,𝚫VE,𝚫VF,𝚫VG,𝚫VH,𝚫VI,𝚫VJ,𝚫MAX
170,DropFirstArray,20,171,,,,,,,,,,,,,,,,,,,2,29
171,DropFirstArrayLazy,20,168,,,,,,,,,,,,,,,,,,,,8
184,DropLastArray,20,55,,,,,,,,,,,,,,,,,,,,26
185,DropLastArrayLazy,20,65,,,,,,,,,,,,,,,,,,,1,90
198,DropWhileArray,20,214,1,,,,,,,,,,,,,,,,,1,27,2
199,DropWhileArrayLazy,20,464,,,,1,,,,,,,,1,1,1,4,9,1,9,113,2903
418,PrefixArray,20,132,,,,,,,,,,,,,,,,,1,1,32,394
419,PrefixArrayLazy,20,168,,,,,,,,,,,,1,,2,9,1,15,8,88,3338
432,PrefixWhileArray,20,252,1,,,,1,,,,,,,,,,,1,,,,30
433,PrefixWhileArrayLazy,20,168,,,,,,,,,,,,,1,,6,6,14,43,28,10200
619,SuffixArray,20,68,,,,,,,,,,,,,1,,,,22,1,1,4
620,SuffixArrayLazy,20,65,,,,,,,,,,,,,,,,,,1,9,340
````
 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
72e960457b [benchmark] Gardening maxRSS as Int? 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
022e1111a9 [benchmark] Report quantiles from samples 
The default benchmark result reports statistics of a normal distribution — mean and standard deviation. Unfortunately the samples from our benchmarks are *not normally distributed*. To get a better picture of the underlying probability distribution, this adds support for reporting quantiles.

See https://en.wikipedia.org/wiki/Quantile

This gives better subsample of the measurements in the summary, without need to resort to the use of a full verbose mode, which might be unnecessarily slow.
 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
219a5d9290 [benchmark] Rename SampleRunner -> TestRunner 
It is now running all the benchmarks, so it’s a TestRunner.
 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
0e751e2717 [benchmark] Gardening: Even nicer microseconds 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
d704557c88 [benchmark] Gardening: Fixed method indentation 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
12c6e39a20 [benchmark] Refactor run runBenchmarks logVerbose 
Extracted nested func logVerbose as instance method on SampleRunner.

Internalized the free functions `runBech` and `runBenchmarks` into SampleRunner as methods `run` and `runBenchmarks`.
 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
e7d1d482d8 [benchmark] Extract yield & add resetMeasurements 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
331c0bf772 [benchmark] Refactor numIters computation 
The spaghetti if-else code was untangled into nested function that computes `iterationsPerSampleTime` and a single constant `numIters` expression that takes care of the overflow capping as well as the choice between fixed and computed `numIters` value.

The `numIters` is now computed and logged only once per benchmark measurement instead of on every sample.

The sampling loop is now just a single line. Hurrah!

Modified test to verify that the `LogParser` maintains `num-iters` derived from the `Measuring with scale` message across samples.
 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
29b2cc7397 [benchmark] Refactor sampling loop with addSample 
Extracted sample saving to inner func `addSample`.
Used it to save the `oneIter` sample from `numIters` calibration when it comes out as 1 and continue the for loop to next sample.

This simplified following code that can now always measure the sample with `numIters` and save it.
 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
b762f80a64 [benchmark] Gardening: Documentation of numIters 
Clarified the need for capping `numIters` according to the discussion at https://github.com/apple/swift/pull/17268#issuecomment-404831035

The sampling loop is a hairy piece of code, because it’s trying to reuse the calibration measurement as a regular sample, in case the computed `numIters` turns out to be 1. But it conflicts with the case when `fixedNumIters` is 1, necessitating a separate measurement in the else branch… That was a quick fix back then, but its hard to make it clean. More thinking is required…
 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
75604a285d [benchmark] Gardening: Sensibly rename variables 
To make sense of this spaghetti code, let’s first use reasonable variable names:
* scale -> numIters
* elapsed_time -> time
 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
a169606e60 [benchmark] Gardening: DRYer verbose log 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
9ae69908b0 [benchmark] Refactor to currency type Int 
Removed unnecessary use of UInt64, where appropriate, following the advice from Swift Language Guide:

> Use the `Int` type for all general-purpose integer constants and variables in your code, even if they’re known to be nonnegative. Using the default integer type in everyday situations means that integer constants and variables are immediately interoperable in your code and will match the inferred type for integer literal values.
https://docs.swift.org/swift-book/LanguageGuide/TheBasics.html#ID324
 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
28eb79819b [benchmark] Refactor to report samples in μs 
Moved the adjustment of `lastSampleTime` to account for the `scale` (`numIters`) and conversion to microseconds into SampleRunner’s `measure` method.
 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
beabad86f4 [benchmark] Gardening: scale was always Int 
Since the `scale` (or `numIters`) is passed to the `test.runFunction` as `Int`, the whole type-casting dance here was just silly!
 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
e775b8fc60 [benchmark] Gardening: numSamples UInt vs Int 
Type check command line argument to be non-negative, but store value in currency type `Int`.
 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
79d7730be8 [benchmark] Gardening: afterRunSleep is UInt32 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
7768cb3295 [benchmark] Move stats computation to BenchResults 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
e48b5fdb34 [benchmark] Fix index computation for quantiles 
Turns out that both the old code in `DriverUtils` that computed median, as well as newer quartiles in `PerformanceTestSamples` had off-by-1 error.

It trully is the 3rd of the 2 hard things in computer science!
 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
ab3e6122c0 [benchmark] Refactor min max median computation 
We can spare 2 array passes (for min and max), if we just sort first.
 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
ed53fe8f71 [benchmark] Refactor mean and stdev computation 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
1a0a7b3d0f [benchmark] Gardening: Timer Parasite Control 
Improve conformance to Swift Naming Guidelines
https://swift.org/documentation/api-design-guidelines/#parameter-names

Removed the gross undescores and ticks from parameter names. Ticks are ectoparasites feeding on the blood. We are just measuring time and there is also no [mysterious ticking noise](http://bit.ly/TickNoise) here either…
 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
7bbf2e1d0a [benchmark] Gardening: Extract constant oneSecond 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
1ba2ee41d9 [benchmark] Gardening: Code format class Timer 2018-09-14 23:40:43 +02:00
Pavol Vaskovic
496a277419 [benchmark] Gardening: Indentation of .listTests 2018-09-14 23:40:43 +02:00
Ben Langmuir
423e145b0c Revert "[benchmark] Report Quantiles from Benchmark_O and a TON of Gardening" 2018-09-14 13:24:01 -07:00
swift-ci
8ce03faa76 Merge pull request #19097 from palimondo/fluctuation-of-the-pupil 2018-09-14 09:24:00 -07:00