[benchmark] Fix quantile estimation type

The correct quantile estimation type for printing all measurements in the summary report while `quantile == num-samples - 1` is R-1, SAS-3.  It's the inverse of empirical distribution function.

References:
* https://en.wikipedia.org/wiki/Quantile#Estimating_quantiles_from_a_sample
* discussion in https://github.com/apple/swift/pull/19097#issuecomment-421238197
This commit is contained in:
Pavol Vaskovic
2018-09-19 22:42:40 +02:00
parent 541c48f9e4
commit a9f0ce4338
3 changed files with 21 additions and 16 deletions

View File

@@ -34,8 +34,7 @@ import re
import sys
from bisect import bisect, bisect_left, bisect_right
from collections import namedtuple
from decimal import Decimal, ROUND_HALF_EVEN
from math import sqrt
from math import ceil, sqrt
class Sample(namedtuple('Sample', 'i num_iters runtime')):
@@ -143,15 +142,12 @@ class PerformanceTestSamples(object):
return self.samples[-1].runtime
def quantile(self, q):
"""Return runtime of a sample nearest to the quantile.
"""Return runtime for given quantile.
Explicitly uses round-half-to-even rounding algorithm to match the
behavior of numpy's quantile(interpolation='nearest') and quantile
estimate type R-3, SAS-2. See:
Equivalent to quantile estimate type R-1, SAS-3. See:
https://en.wikipedia.org/wiki/Quantile#Estimating_quantiles_from_a_sample
"""
index = int(Decimal((self.count - 1) * Decimal(q))
.quantize(0, ROUND_HALF_EVEN))
index = max(0, int(ceil(self.count * float(q))) - 1)
return self.samples[index].runtime
@property