criterion performance measurements

overview

want to understand this report?

head_tail/scott/m = 2

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.8735103765898078e-8 1.876259668005157e-8 1.8804613642850978e-8
Standard deviation 7.71013892816398e-11 1.1343407499861462e-10 1.5253098273220006e-10

Outlying measurements have no (3.4013209239480992e-3%) effect on estimated standard deviation.

head_tail/scott/m = 10

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.882859466585859e-8 1.8844098378515133e-8 1.8877413607076468e-8
Standard deviation 2.5047959103470282e-11 7.307828166240072e-11 1.2003707227410141e-10

Outlying measurements have no (3.4013209239480173e-3%) effect on estimated standard deviation.

head_tail/scott/m = 100

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.8848198695007207e-8 1.8982167324562876e-8 1.960558444300565e-8
Standard deviation 9.878996542974863e-11 8.057214474963083e-10 1.8360609514211954e-9

Outlying measurements have severe (0.6611391225027786%) effect on estimated standard deviation.

head_tail/scott/m = 1000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.8817386826137104e-8 1.88327344566295e-8 1.8879002148740895e-8
Standard deviation 3.1393991353881706e-11 8.169372237557771e-11 1.7198791040695662e-10

Outlying measurements have no (3.4013209239478668e-3%) effect on estimated standard deviation.

head_tail/scott/m = 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.881514916612667e-8 1.882608252529518e-8 1.884466481795051e-8
Standard deviation 2.759481076002515e-11 4.6298534612244236e-11 7.219596610903846e-11

Outlying measurements have no (3.401320923947762e-3%) effect on estimated standard deviation.

head_tail/scott/m = 100000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.8839222665027268e-8 1.8873297885818753e-8 1.8994606112241827e-8
Standard deviation 3.566692566046142e-11 1.9733217219984344e-10 4.110690238831713e-10

Outlying measurements have moderate (0.10339643829878599%) effect on estimated standard deviation.

head_tail/builtin/m = 2

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.502080205254298e-9 5.503184734246324e-9 5.505410558326612e-9
Standard deviation 3.0513643472417135e-12 5.0546946836134814e-12 9.232629071451311e-12

Outlying measurements have no (3.1347652387166375e-3%) effect on estimated standard deviation.

head_tail/builtin/m = 10

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.503000056814007e-9 5.50539124662046e-9 5.512783003009627e-9
Standard deviation 4.77724799288804e-12 1.2673526623241116e-11 2.6547314218550125e-11

Outlying measurements have no (3.1347652387167164e-3%) effect on estimated standard deviation.

head_tail/builtin/m = 100

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.5392590502121105e-9 5.595447224258604e-9 5.728004663621247e-9
Standard deviation 1.357170516337328e-10 2.7176866894734e-10 4.949094568103412e-10

Outlying measurements have severe (0.735954958980834%) effect on estimated standard deviation.

head_tail/builtin/m = 1000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.509415451543502e-9 5.513532713365771e-9 5.522220234827989e-9
Standard deviation 1.1361815769157322e-11 1.9404959256581075e-11 3.6490337857719263e-11

Outlying measurements have no (3.1347652387167797e-3%) effect on estimated standard deviation.

head_tail/builtin/m = 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.507344037169744e-9 5.518218850264039e-9 5.545974983085144e-9
Standard deviation 1.3438337835775204e-11 5.254650245354666e-11 9.907659731208492e-11

Outlying measurements have slight (9.51417555162737e-2%) effect on estimated standard deviation.

head_tail/builtin/m = 100000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.507808273017074e-9 5.512897441343986e-9 5.5228043182696955e-9
Standard deviation 1.2984684913946645e-11 2.358928640389431e-11 3.6937317696432425e-11

Outlying measurements have no (3.134765238716496e-3%) effect on estimated standard deviation.

sum/scott/m = 1

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.77185964565311e-8 1.7747422409524133e-8 1.7793381621463522e-8
Standard deviation 7.672189756329175e-11 1.1859303276122327e-10 1.7427432984151867e-10

Outlying measurements have no (3.3897912906660226e-3%) effect on estimated standard deviation.

sum/scott/m = 10

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.362156335170868e-7 1.3673483689003684e-7 1.387311506746756e-7
Standard deviation 5.271806183580735e-10 3.0935379663120288e-9 6.999969713676189e-9

Outlying measurements have moderate (0.3198739727323602%) effect on estimated standard deviation.

sum/scott/m = 100

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.2739455620845268e-6 1.2752128473726142e-6 1.278606452414163e-6
Standard deviation 2.8274274389303605e-9 6.3196950070693445e-9 1.2453474242326059e-8

Outlying measurements have no (4.8308040343103984e-3%) effect on estimated standard deviation.

sum/scott/m = 1000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.3116657897975359e-5 1.312472895310691e-5 1.3141752719664551e-5
Standard deviation 2.1682082495627586e-8 3.754653314475114e-8 6.909554775075602e-8

Outlying measurements have no (6.289056240987021e-3%) effect on estimated standard deviation.

sum/scott/m = 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.955678663582531e-4 1.95743473495007e-4 1.9613101563937453e-4
Standard deviation 2.675754603229487e-7 8.306229090226399e-7 1.398447433082035e-6

Outlying measurements have no (9.614478273164102e-3%) effect on estimated standard deviation.

sum/scott/m = 100000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.8756321875786964e-3 2.881692944654911e-3 2.888480188264263e-3
Standard deviation 1.7124086180275214e-5 2.1360800337598063e-5 2.8405138350179166e-5

Outlying measurements have slight (1.9223375624759707e-2%) effect on estimated standard deviation.

sum/builtin/m = 1

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 7.616281599530407e-9 7.621740608842343e-9 7.62827237959788e-9
Standard deviation 1.614863158660443e-11 1.9875783162855838e-11 2.769680073791901e-11

Outlying measurements have no (3.205095067255155e-3%) effect on estimated standard deviation.

sum/builtin/m = 10

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.3988180769797853e-8 3.43462776887264e-8 3.513385417301273e-8
Standard deviation 8.121870180591853e-10 1.715643226379403e-9 2.785731164016383e-9

Outlying measurements have severe (0.7236087011302299%) effect on estimated standard deviation.

sum/builtin/m = 100

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.9238673991515e-7 3.925756794301654e-7 3.9312822780714077e-7
Standard deviation 4.17640572415247e-10 9.64128350214557e-10 1.999326557456315e-9

Outlying measurements have no (4.328922495274311e-3%) effect on estimated standard deviation.

sum/builtin/m = 1000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.878696833580143e-6 3.880080071555723e-6 3.882357664564203e-6
Standard deviation 3.944308141933337e-9 5.847817709685808e-9 9.333537888459357e-9

Outlying measurements have no (5.4346203230911634e-3%) effect on estimated standard deviation.

sum/builtin/m = 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 9.396769929818203e-5 9.40628650300239e-5 9.420496011732352e-5
Standard deviation 2.7337911239367083e-7 3.843888574967289e-7 5.536737337486636e-7

Outlying measurements have no (8.402757828210285e-3%) effect on estimated standard deviation.

sum/builtin/m = 100000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.418365255289337e-3 1.4260028755078912e-3 1.4361778301657338e-3
Standard deviation 2.3146736015895446e-5 2.9714001332912097e-5 4.293949468430892e-5

Outlying measurements have moderate (0.10264841383648761%) effect on estimated standard deviation.

quicksort/scott/m = 1

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.385726139794326e-8 5.395215662640341e-8 5.4209347262911444e-8
Standard deviation 1.9298160637138926e-10 4.792662098162156e-10 9.956425598936096e-10

Outlying measurements have slight (7.310329904250613e-2%) effect on estimated standard deviation.

quicksort/scott/m = 10

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.893703254903301e-7 2.8997154032033057e-7 2.919774747893664e-7
Standard deviation 1.0155621980759122e-9 3.322792155350185e-9 7.1932115323058266e-9

Outlying measurements have slight (9.989847993539235e-2%) effect on estimated standard deviation.

quicksort/scott/m = 100

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.679407327077383e-6 2.6806703782027097e-6 2.6822146097601975e-6
Standard deviation 3.692015691276549e-9 4.7002146752445924e-9 6.681134514545089e-9

Outlying measurements have no (5.208190564951599e-3%) effect on estimated standard deviation.

quicksort/scott/m = 1000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.316849261362858e-5 3.325760326029437e-5 3.3385348761677415e-5
Standard deviation 2.4617703237943294e-7 3.525963554096806e-7 4.750196173530088e-7

Outlying measurements have no (7.142487448889792e-3%) effect on estimated standard deviation.

quicksort/scott/m = 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.8535447005397103e-4 3.8600137323875167e-4 3.866973011637479e-4
Standard deviation 1.7520001981477607e-6 2.247327333543461e-6 2.932076779873416e-6

Outlying measurements have slight (1.0987654320987826e-2%) effect on estimated standard deviation.

quicksort/scott/m = 100000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.8373735485233525e-3 5.866821467672924e-3 5.897905902360164e-3
Standard deviation 7.05669240201353e-5 9.123434932253806e-5 1.2493666903852654e-4

Outlying measurements have slight (2.5623268698060937e-2%) effect on estimated standard deviation.

quicksort/builtin/m = 1

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.1157046596676875e-8 2.1311085053924353e-8 2.167843947181255e-8
Standard deviation 9.932958936109794e-11 7.192792143883034e-10 1.2719793013482185e-9

Outlying measurements have severe (0.5503910415822382%) effect on estimated standard deviation.

quicksort/builtin/m = 10

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.4028886770318488e-7 1.407248871241875e-7 1.4192113788877615e-7
Standard deviation 1.040928639108847e-9 2.2295906851599556e-9 4.578336618219922e-9

Outlying measurements have moderate (0.1857680191834429%) effect on estimated standard deviation.

quicksort/builtin/m = 100

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.5820941293236924e-6 1.5832123175178207e-6 1.5863258742266194e-6
Standard deviation 2.4840051170463227e-9 5.797985380565039e-9 1.1783352569436739e-8

Outlying measurements have no (4.925987648269566e-3%) effect on estimated standard deviation.

quicksort/builtin/m = 1000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.9780358601822367e-5 1.9799966928720453e-5 1.9838917536736912e-5
Standard deviation 5.4228021805331376e-8 8.818666674322688e-8 1.5183556259760886e-7

Outlying measurements have no (6.622222222222132e-3%) effect on estimated standard deviation.

quicksort/builtin/m = 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.3543081230425052e-4 2.3571025041911495e-4 2.3635510889773599e-4
Standard deviation 7.05105105336345e-7 1.35184393130913e-6 2.6841930463791244e-6

Outlying measurements have no (9.90000000000013e-3%) effect on estimated standard deviation.

quicksort/builtin/m = 100000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 4.638133131297349e-3 4.6619972717691395e-3 4.685940343226301e-3
Standard deviation 6.145406681257957e-5 7.404104199148272e-5 9.07199579015445e-5

Outlying measurements have slight (2.3242630385487847e-2%) effect on estimated standard deviation.

understanding this report

In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.

Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.

We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)

A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.