criterion performance measurements

overview

want to understand this report?

head_tail/scott/m = 2

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.8317455347588395e-8 1.8329026299438925e-8 1.8346713922128228e-8
Standard deviation 3.1707590258415066e-11 4.688075960314497e-11 7.458044286024987e-11

Outlying measurements have no (3.4013209239479557e-3%) effect on estimated standard deviation.

head_tail/scott/m = 10

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.8355973047018138e-8 1.8376128000345814e-8 1.840231824114884e-8
Standard deviation 5.7862718055048546e-11 7.554847967014789e-11 9.817849147820448e-11

Outlying measurements have no (3.4013209239479704e-3%) effect on estimated standard deviation.

head_tail/scott/m = 100

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.8381716011111435e-8 1.839537223338751e-8 1.8421251643563507e-8
Standard deviation 3.206208397789981e-11 6.06573470435254e-11 9.662886160598635e-11

Outlying measurements have no (3.401320923947867e-3%) effect on estimated standard deviation.

head_tail/scott/m = 1000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.8360362208297194e-8 1.8376903417279845e-8 1.8407032813105295e-8
Standard deviation 4.47900402939963e-11 7.197574248004588e-11 1.2619173400444298e-10

Outlying measurements have no (3.4013209239478976e-3%) effect on estimated standard deviation.

head_tail/scott/m = 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.8382894023732562e-8 1.8398751969538447e-8 1.8433789664357703e-8
Standard deviation 4.1305040085694286e-11 7.496071402501056e-11 1.429174496070281e-10

Outlying measurements have no (3.401320923947818e-3%) effect on estimated standard deviation.

head_tail/scott/m = 100000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.837139402366988e-8 1.8392918465460005e-8 1.844701043631125e-8
Standard deviation 5.123938405631586e-11 1.0540841898044369e-10 2.0698023761729001e-10

Outlying measurements have no (3.4013209239479895e-3%) effect on estimated standard deviation.

head_tail/builtin/m = 2

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 6.149498880245353e-9 6.349259806769664e-9 6.7362490679636596e-9
Standard deviation 5.407051579689931e-10 9.193650404349146e-10 1.366778120577357e-9

Outlying measurements have severe (0.9629928803679509%) effect on estimated standard deviation.

head_tail/builtin/m = 10

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 6.03058595022534e-9 6.179882264311594e-9 6.524181593988086e-9
Standard deviation 2.390065642134543e-10 7.369829329425332e-10 1.2432473410053387e-9

Outlying measurements have severe (0.9462334650106085%) effect on estimated standard deviation.

head_tail/builtin/m = 100

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.98166939924769e-9 5.995768055934259e-9 6.010860176183618e-9
Standard deviation 3.920177169722342e-11 4.9651437038892294e-11 6.494367443108838e-11

Outlying measurements have slight (7.179019141467817e-2%) effect on estimated standard deviation.

head_tail/builtin/m = 1000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.970362233498697e-9 5.980752773324106e-9 5.9916191697248606e-9
Standard deviation 2.712201078956763e-11 3.550128297084257e-11 4.798655062926567e-11

Outlying measurements have no (3.154542541259519e-3%) effect on estimated standard deviation.

head_tail/builtin/m = 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.981241130575278e-9 5.9941709336375995e-9 6.014214237727256e-9
Standard deviation 3.6912408844426326e-11 5.3209513749792786e-11 8.895811394243148e-11

Outlying measurements have slight (8.368091320560879e-2%) effect on estimated standard deviation.

head_tail/builtin/m = 100000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.977319255234244e-9 5.988402155144584e-9 6.013324518217005e-9
Standard deviation 2.6873803997428834e-11 5.2163158070623076e-11 1.022618765821441e-10

Outlying measurements have slight (8.068551965651109e-2%) effect on estimated standard deviation.

sum/scott/m = 1

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.773834170854132e-8 1.7760633674062392e-8 1.7785420712965553e-8
Standard deviation 6.602282005304246e-11 8.024354315218903e-11 1.035307509478594e-10

Outlying measurements have no (3.3897912906661735e-3%) effect on estimated standard deviation.

sum/scott/m = 10

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.3585796557455063e-7 1.3593160216946076e-7 1.3605294722805505e-7
Standard deviation 2.1982263555182165e-10 3.1287833101393206e-10 4.466124173619707e-10

Outlying measurements have no (3.952506928697657e-3%) effect on estimated standard deviation.

sum/scott/m = 100

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.2725489862867813e-6 1.273753674430297e-6 1.2759457094687127e-6
Standard deviation 2.63833198616187e-9 5.3225213356334506e-9 8.348112830896785e-9

Outlying measurements have no (4.830804034310436e-3%) effect on estimated standard deviation.

sum/scott/m = 1000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.3111714604708351e-5 1.3126655929311088e-5 1.315607502857838e-5
Standard deviation 3.887422843636672e-8 6.69553874303272e-8 1.0215433707540577e-7

Outlying measurements have no (6.2890562409872e-3%) effect on estimated standard deviation.

sum/scott/m = 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.9589853452858775e-4 1.9620851334383282e-4 1.9677867079831413e-4
Standard deviation 8.243411287233852e-7 1.3827885286387037e-6 2.1380559860839225e-6

Outlying measurements have no (9.614478273164244e-3%) effect on estimated standard deviation.

sum/scott/m = 100000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.035954390654272e-3 3.127558509990536e-3 3.2756705325785785e-3
Standard deviation 2.6215354631652234e-4 3.906710121889856e-4 5.69013720249885e-4

Outlying measurements have severe (0.748431801900329%) effect on estimated standard deviation.

sum/builtin/m = 1

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 7.613348422447887e-9 7.619714215693527e-9 7.630070700156978e-9
Standard deviation 1.7661251671045534e-11 2.678915444859432e-11 4.164616705961723e-11

Outlying measurements have no (3.2050950672552997e-3%) effect on estimated standard deviation.

sum/builtin/m = 10

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.385531831130291e-8 3.396604244394515e-8 3.438026892891463e-8
Standard deviation 8.977564360073974e-11 6.57996976601104e-10 1.380629996771589e-9

Outlying measurements have moderate (0.27594466441760335%) effect on estimated standard deviation.

sum/builtin/m = 100

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.918102065609985e-7 3.9223832820325206e-7 3.9329011569367577e-7
Standard deviation 6.008030324937385e-10 2.046447693743886e-9 3.4887275979615146e-9

Outlying measurements have no (4.328922495274087e-3%) effect on estimated standard deviation.

sum/builtin/m = 1000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.889314030717771e-6 3.894016521508655e-6 3.906591832709436e-6
Standard deviation 9.452567279075394e-9 2.3425523929765535e-8 4.442511251596527e-8

Outlying measurements have no (5.434620323090979e-3%) effect on estimated standard deviation.

sum/builtin/m = 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 9.371482745904527e-5 9.384406630607336e-5 9.436822252808289e-5
Standard deviation 1.8785691588163924e-7 7.414463492038744e-7 1.6535735205883103e-6

Outlying measurements have no (8.40275782821042e-3%) effect on estimated standard deviation.

sum/builtin/m = 100000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.4143835108333916e-3 1.4225333891757287e-3 1.4331195324790982e-3
Standard deviation 2.455372913778819e-5 3.090774802626537e-5 4.453884319963516e-5

Outlying measurements have moderate (0.10318819545911052%) effect on estimated standard deviation.

quicksort/scott/m = 1

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.290248229805763e-8 5.3005484932900826e-8 5.345083373007246e-8
Standard deviation 1.3703065719155554e-10 5.822785826579732e-10 1.3789380189118206e-9

Outlying measurements have moderate (0.10484600245693733%) effect on estimated standard deviation.

quicksort/scott/m = 10

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.490411106263977e-7 3.498753584157975e-7 3.5107046920903533e-7
Standard deviation 2.5029179483139876e-9 3.37106277339552e-9 4.5290282824703325e-9

Outlying measurements have slight (7.29420969496471e-2%) effect on estimated standard deviation.

quicksort/scott/m = 100

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.5711531558938038e-6 3.573644631630152e-6 3.577045693969109e-6
Standard deviation 7.1233013454481455e-9 9.579930016788613e-9 1.2549091482917058e-8

Outlying measurements have no (5.376186997808619e-3%) effect on estimated standard deviation.

quicksort/scott/m = 1000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 4.0617553190466146e-5 4.075553282514836e-5 4.0989238676948245e-5
Standard deviation 3.7982146043753325e-7 5.844199372027303e-7 8.128129782736926e-7

Outlying measurements have slight (9.107680077450808e-2%) effect on estimated standard deviation.

quicksort/scott/m = 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.884507223667916e-4 6.053212116138697e-4 6.380131605639804e-4
Standard deviation 4.846983218776066e-5 7.476884401303821e-5 1.272042888934873e-4

Outlying measurements have severe (0.8305954435913395%) effect on estimated standard deviation.

quicksort/scott/m = 100000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 8.301680646115452e-3 8.345332234864892e-3 8.394054683270473e-3
Standard deviation 9.915946590767151e-5 1.3165541845672527e-4 1.8117244809254536e-4

Outlying measurements have slight (2.9384756657483867e-2%) effect on estimated standard deviation.

quicksort/builtin/m = 1

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.91646709829139e-8 1.9259323480960707e-8 1.9586770220635266e-8
Standard deviation 1.5284651751625595e-10 5.27633870110985e-10 1.1515404104934764e-9

Outlying measurements have moderate (0.4438900618874906%) effect on estimated standard deviation.

quicksort/builtin/m = 10

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.8299435448311098e-7 1.8319896554589638e-7 1.8372229791810682e-7
Standard deviation 4.662803536300546e-10 1.022879970120367e-9 1.906989557716886e-9

Outlying measurements have no (4.0485160949170745e-3%) effect on estimated standard deviation.

quicksort/builtin/m = 100

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.9857913449003965e-6 1.9907969267613006e-6 2.0006420574508023e-6
Standard deviation 1.3337827699155558e-8 2.290571024048849e-8 3.918044239457236e-8

Outlying measurements have slight (8.636868427759813e-2%) effect on estimated standard deviation.

quicksort/builtin/m = 1000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.450452402112994e-5 2.4550759244138077e-5 2.4727625404976652e-5
Standard deviation 5.000027632447137e-8 2.6884311547524746e-7 5.961930562263348e-7

Outlying measurements have slight (6.430624214232909e-2%) effect on estimated standard deviation.

quicksort/builtin/m = 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.1149965423244125e-4 3.1189716094354994e-4 3.1307946166464837e-4
Standard deviation 8.348470663712479e-7 2.09809428977642e-6 4.4011742703625025e-6

Outlying measurements have slight (1.0525124490719946e-2%) effect on estimated standard deviation.

quicksort/builtin/m = 100000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 6.91530081171258e-3 7.019979359421306e-3 7.319895324988944e-3
Standard deviation 1.2961856272171018e-4 4.8610672862694603e-4 9.78326931394474e-4

Outlying measurements have moderate (0.40095206246445564%) effect on estimated standard deviation.

understanding this report

In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.

Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.

We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)

A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.