criterion performance measurements
overview
want to understand this report?
head_tail/scott/m = 2
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 1.8317455347588395e-8 | 1.8329026299438925e-8 | 1.8346713922128228e-8 |
Standard deviation | 3.1707590258415066e-11 | 4.688075960314497e-11 | 7.458044286024987e-11 |
Outlying measurements have no (3.4013209239479557e-3%) effect on estimated standard deviation.
head_tail/scott/m = 10
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 1.8355973047018138e-8 | 1.8376128000345814e-8 | 1.840231824114884e-8 |
Standard deviation | 5.7862718055048546e-11 | 7.554847967014789e-11 | 9.817849147820448e-11 |
Outlying measurements have no (3.4013209239479704e-3%) effect on estimated standard deviation.
head_tail/scott/m = 100
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 1.8381716011111435e-8 | 1.839537223338751e-8 | 1.8421251643563507e-8 |
Standard deviation | 3.206208397789981e-11 | 6.06573470435254e-11 | 9.662886160598635e-11 |
Outlying measurements have no (3.401320923947867e-3%) effect on estimated standard deviation.
head_tail/scott/m = 1000
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 1.8360362208297194e-8 | 1.8376903417279845e-8 | 1.8407032813105295e-8 |
Standard deviation | 4.47900402939963e-11 | 7.197574248004588e-11 | 1.2619173400444298e-10 |
Outlying measurements have no (3.4013209239478976e-3%) effect on estimated standard deviation.
head_tail/scott/m = 10000
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 1.8382894023732562e-8 | 1.8398751969538447e-8 | 1.8433789664357703e-8 |
Standard deviation | 4.1305040085694286e-11 | 7.496071402501056e-11 | 1.429174496070281e-10 |
Outlying measurements have no (3.401320923947818e-3%) effect on estimated standard deviation.
head_tail/scott/m = 100000
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 1.837139402366988e-8 | 1.8392918465460005e-8 | 1.844701043631125e-8 |
Standard deviation | 5.123938405631586e-11 | 1.0540841898044369e-10 | 2.0698023761729001e-10 |
Outlying measurements have no (3.4013209239479895e-3%) effect on estimated standard deviation.
head_tail/builtin/m = 2
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 6.149498880245353e-9 | 6.349259806769664e-9 | 6.7362490679636596e-9 |
Standard deviation | 5.407051579689931e-10 | 9.193650404349146e-10 | 1.366778120577357e-9 |
Outlying measurements have severe (0.9629928803679509%) effect on estimated standard deviation.
head_tail/builtin/m = 10
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 6.03058595022534e-9 | 6.179882264311594e-9 | 6.524181593988086e-9 |
Standard deviation | 2.390065642134543e-10 | 7.369829329425332e-10 | 1.2432473410053387e-9 |
Outlying measurements have severe (0.9462334650106085%) effect on estimated standard deviation.
head_tail/builtin/m = 100
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 5.98166939924769e-9 | 5.995768055934259e-9 | 6.010860176183618e-9 |
Standard deviation | 3.920177169722342e-11 | 4.9651437038892294e-11 | 6.494367443108838e-11 |
Outlying measurements have slight (7.179019141467817e-2%) effect on estimated standard deviation.
head_tail/builtin/m = 1000
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 5.970362233498697e-9 | 5.980752773324106e-9 | 5.9916191697248606e-9 |
Standard deviation | 2.712201078956763e-11 | 3.550128297084257e-11 | 4.798655062926567e-11 |
Outlying measurements have no (3.154542541259519e-3%) effect on estimated standard deviation.
head_tail/builtin/m = 10000
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 5.981241130575278e-9 | 5.9941709336375995e-9 | 6.014214237727256e-9 |
Standard deviation | 3.6912408844426326e-11 | 5.3209513749792786e-11 | 8.895811394243148e-11 |
Outlying measurements have slight (8.368091320560879e-2%) effect on estimated standard deviation.
head_tail/builtin/m = 100000
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 5.977319255234244e-9 | 5.988402155144584e-9 | 6.013324518217005e-9 |
Standard deviation | 2.6873803997428834e-11 | 5.2163158070623076e-11 | 1.022618765821441e-10 |
Outlying measurements have slight (8.068551965651109e-2%) effect on estimated standard deviation.
sum/scott/m = 1
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 1.773834170854132e-8 | 1.7760633674062392e-8 | 1.7785420712965553e-8 |
Standard deviation | 6.602282005304246e-11 | 8.024354315218903e-11 | 1.035307509478594e-10 |
Outlying measurements have no (3.3897912906661735e-3%) effect on estimated standard deviation.
sum/scott/m = 10
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 1.3585796557455063e-7 | 1.3593160216946076e-7 | 1.3605294722805505e-7 |
Standard deviation | 2.1982263555182165e-10 | 3.1287833101393206e-10 | 4.466124173619707e-10 |
Outlying measurements have no (3.952506928697657e-3%) effect on estimated standard deviation.
sum/scott/m = 100
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 1.2725489862867813e-6 | 1.273753674430297e-6 | 1.2759457094687127e-6 |
Standard deviation | 2.63833198616187e-9 | 5.3225213356334506e-9 | 8.348112830896785e-9 |
Outlying measurements have no (4.830804034310436e-3%) effect on estimated standard deviation.
sum/scott/m = 1000
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 1.3111714604708351e-5 | 1.3126655929311088e-5 | 1.315607502857838e-5 |
Standard deviation | 3.887422843636672e-8 | 6.69553874303272e-8 | 1.0215433707540577e-7 |
Outlying measurements have no (6.2890562409872e-3%) effect on estimated standard deviation.
sum/scott/m = 10000
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 1.9589853452858775e-4 | 1.9620851334383282e-4 | 1.9677867079831413e-4 |
Standard deviation | 8.243411287233852e-7 | 1.3827885286387037e-6 | 2.1380559860839225e-6 |
Outlying measurements have no (9.614478273164244e-3%) effect on estimated standard deviation.
sum/scott/m = 100000
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 3.035954390654272e-3 | 3.127558509990536e-3 | 3.2756705325785785e-3 |
Standard deviation | 2.6215354631652234e-4 | 3.906710121889856e-4 | 5.69013720249885e-4 |
Outlying measurements have severe (0.748431801900329%) effect on estimated standard deviation.
sum/builtin/m = 1
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 7.613348422447887e-9 | 7.619714215693527e-9 | 7.630070700156978e-9 |
Standard deviation | 1.7661251671045534e-11 | 2.678915444859432e-11 | 4.164616705961723e-11 |
Outlying measurements have no (3.2050950672552997e-3%) effect on estimated standard deviation.
sum/builtin/m = 10
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 3.385531831130291e-8 | 3.396604244394515e-8 | 3.438026892891463e-8 |
Standard deviation | 8.977564360073974e-11 | 6.57996976601104e-10 | 1.380629996771589e-9 |
Outlying measurements have moderate (0.27594466441760335%) effect on estimated standard deviation.
sum/builtin/m = 100
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 3.918102065609985e-7 | 3.9223832820325206e-7 | 3.9329011569367577e-7 |
Standard deviation | 6.008030324937385e-10 | 2.046447693743886e-9 | 3.4887275979615146e-9 |
Outlying measurements have no (4.328922495274087e-3%) effect on estimated standard deviation.
sum/builtin/m = 1000
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 3.889314030717771e-6 | 3.894016521508655e-6 | 3.906591832709436e-6 |
Standard deviation | 9.452567279075394e-9 | 2.3425523929765535e-8 | 4.442511251596527e-8 |
Outlying measurements have no (5.434620323090979e-3%) effect on estimated standard deviation.
sum/builtin/m = 10000
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 9.371482745904527e-5 | 9.384406630607336e-5 | 9.436822252808289e-5 |
Standard deviation | 1.8785691588163924e-7 | 7.414463492038744e-7 | 1.6535735205883103e-6 |
Outlying measurements have no (8.40275782821042e-3%) effect on estimated standard deviation.
sum/builtin/m = 100000
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 1.4143835108333916e-3 | 1.4225333891757287e-3 | 1.4331195324790982e-3 |
Standard deviation | 2.455372913778819e-5 | 3.090774802626537e-5 | 4.453884319963516e-5 |
Outlying measurements have moderate (0.10318819545911052%) effect on estimated standard deviation.
quicksort/scott/m = 1
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 5.290248229805763e-8 | 5.3005484932900826e-8 | 5.345083373007246e-8 |
Standard deviation | 1.3703065719155554e-10 | 5.822785826579732e-10 | 1.3789380189118206e-9 |
Outlying measurements have moderate (0.10484600245693733%) effect on estimated standard deviation.
quicksort/scott/m = 10
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 3.490411106263977e-7 | 3.498753584157975e-7 | 3.5107046920903533e-7 |
Standard deviation | 2.5029179483139876e-9 | 3.37106277339552e-9 | 4.5290282824703325e-9 |
Outlying measurements have slight (7.29420969496471e-2%) effect on estimated standard deviation.
quicksort/scott/m = 100
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 3.5711531558938038e-6 | 3.573644631630152e-6 | 3.577045693969109e-6 |
Standard deviation | 7.1233013454481455e-9 | 9.579930016788613e-9 | 1.2549091482917058e-8 |
Outlying measurements have no (5.376186997808619e-3%) effect on estimated standard deviation.
quicksort/scott/m = 1000
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 4.0617553190466146e-5 | 4.075553282514836e-5 | 4.0989238676948245e-5 |
Standard deviation | 3.7982146043753325e-7 | 5.844199372027303e-7 | 8.128129782736926e-7 |
Outlying measurements have slight (9.107680077450808e-2%) effect on estimated standard deviation.
quicksort/scott/m = 10000
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 5.884507223667916e-4 | 6.053212116138697e-4 | 6.380131605639804e-4 |
Standard deviation | 4.846983218776066e-5 | 7.476884401303821e-5 | 1.272042888934873e-4 |
Outlying measurements have severe (0.8305954435913395%) effect on estimated standard deviation.
quicksort/scott/m = 100000
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 8.301680646115452e-3 | 8.345332234864892e-3 | 8.394054683270473e-3 |
Standard deviation | 9.915946590767151e-5 | 1.3165541845672527e-4 | 1.8117244809254536e-4 |
Outlying measurements have slight (2.9384756657483867e-2%) effect on estimated standard deviation.
quicksort/builtin/m = 1
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 1.91646709829139e-8 | 1.9259323480960707e-8 | 1.9586770220635266e-8 |
Standard deviation | 1.5284651751625595e-10 | 5.27633870110985e-10 | 1.1515404104934764e-9 |
Outlying measurements have moderate (0.4438900618874906%) effect on estimated standard deviation.
quicksort/builtin/m = 10
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 1.8299435448311098e-7 | 1.8319896554589638e-7 | 1.8372229791810682e-7 |
Standard deviation | 4.662803536300546e-10 | 1.022879970120367e-9 | 1.906989557716886e-9 |
Outlying measurements have no (4.0485160949170745e-3%) effect on estimated standard deviation.
quicksort/builtin/m = 100
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 1.9857913449003965e-6 | 1.9907969267613006e-6 | 2.0006420574508023e-6 |
Standard deviation | 1.3337827699155558e-8 | 2.290571024048849e-8 | 3.918044239457236e-8 |
Outlying measurements have slight (8.636868427759813e-2%) effect on estimated standard deviation.
quicksort/builtin/m = 1000
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 2.450452402112994e-5 | 2.4550759244138077e-5 | 2.4727625404976652e-5 |
Standard deviation | 5.000027632447137e-8 | 2.6884311547524746e-7 | 5.961930562263348e-7 |
Outlying measurements have slight (6.430624214232909e-2%) effect on estimated standard deviation.
quicksort/builtin/m = 10000
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 3.1149965423244125e-4 | 3.1189716094354994e-4 | 3.1307946166464837e-4 |
Standard deviation | 8.348470663712479e-7 | 2.09809428977642e-6 | 4.4011742703625025e-6 |
Outlying measurements have slight (1.0525124490719946e-2%) effect on estimated standard deviation.
quicksort/builtin/m = 100000
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 6.91530081171258e-3 | 7.019979359421306e-3 | 7.319895324988944e-3 |
Standard deviation | 1.2961856272171018e-4 | 4.8610672862694603e-4 | 9.78326931394474e-4 |
Outlying measurements have moderate (0.40095206246445564%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.