criterion performance measurements
overview
want to understand this report?
head_tail/scott/m = 2
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.8735103765898078e-8 | 1.876259668005157e-8 | 1.8804613642850978e-8 |
| Standard deviation | 7.71013892816398e-11 | 1.1343407499861462e-10 | 1.5253098273220006e-10 |
Outlying measurements have no (3.4013209239480992e-3%) effect on estimated standard deviation.
head_tail/scott/m = 10
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.882859466585859e-8 | 1.8844098378515133e-8 | 1.8877413607076468e-8 |
| Standard deviation | 2.5047959103470282e-11 | 7.307828166240072e-11 | 1.2003707227410141e-10 |
Outlying measurements have no (3.4013209239480173e-3%) effect on estimated standard deviation.
head_tail/scott/m = 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.8848198695007207e-8 | 1.8982167324562876e-8 | 1.960558444300565e-8 |
| Standard deviation | 9.878996542974863e-11 | 8.057214474963083e-10 | 1.8360609514211954e-9 |
Outlying measurements have severe (0.6611391225027786%) effect on estimated standard deviation.
head_tail/scott/m = 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.8817386826137104e-8 | 1.88327344566295e-8 | 1.8879002148740895e-8 |
| Standard deviation | 3.1393991353881706e-11 | 8.169372237557771e-11 | 1.7198791040695662e-10 |
Outlying measurements have no (3.4013209239478668e-3%) effect on estimated standard deviation.
head_tail/scott/m = 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.881514916612667e-8 | 1.882608252529518e-8 | 1.884466481795051e-8 |
| Standard deviation | 2.759481076002515e-11 | 4.6298534612244236e-11 | 7.219596610903846e-11 |
Outlying measurements have no (3.401320923947762e-3%) effect on estimated standard deviation.
head_tail/scott/m = 100000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.8839222665027268e-8 | 1.8873297885818753e-8 | 1.8994606112241827e-8 |
| Standard deviation | 3.566692566046142e-11 | 1.9733217219984344e-10 | 4.110690238831713e-10 |
Outlying measurements have moderate (0.10339643829878599%) effect on estimated standard deviation.
head_tail/builtin/m = 2
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.502080205254298e-9 | 5.503184734246324e-9 | 5.505410558326612e-9 |
| Standard deviation | 3.0513643472417135e-12 | 5.0546946836134814e-12 | 9.232629071451311e-12 |
Outlying measurements have no (3.1347652387166375e-3%) effect on estimated standard deviation.
head_tail/builtin/m = 10
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.503000056814007e-9 | 5.50539124662046e-9 | 5.512783003009627e-9 |
| Standard deviation | 4.77724799288804e-12 | 1.2673526623241116e-11 | 2.6547314218550125e-11 |
Outlying measurements have no (3.1347652387167164e-3%) effect on estimated standard deviation.
head_tail/builtin/m = 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.5392590502121105e-9 | 5.595447224258604e-9 | 5.728004663621247e-9 |
| Standard deviation | 1.357170516337328e-10 | 2.7176866894734e-10 | 4.949094568103412e-10 |
Outlying measurements have severe (0.735954958980834%) effect on estimated standard deviation.
head_tail/builtin/m = 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.509415451543502e-9 | 5.513532713365771e-9 | 5.522220234827989e-9 |
| Standard deviation | 1.1361815769157322e-11 | 1.9404959256581075e-11 | 3.6490337857719263e-11 |
Outlying measurements have no (3.1347652387167797e-3%) effect on estimated standard deviation.
head_tail/builtin/m = 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.507344037169744e-9 | 5.518218850264039e-9 | 5.545974983085144e-9 |
| Standard deviation | 1.3438337835775204e-11 | 5.254650245354666e-11 | 9.907659731208492e-11 |
Outlying measurements have slight (9.51417555162737e-2%) effect on estimated standard deviation.
head_tail/builtin/m = 100000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.507808273017074e-9 | 5.512897441343986e-9 | 5.5228043182696955e-9 |
| Standard deviation | 1.2984684913946645e-11 | 2.358928640389431e-11 | 3.6937317696432425e-11 |
Outlying measurements have no (3.134765238716496e-3%) effect on estimated standard deviation.
sum/scott/m = 1
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.77185964565311e-8 | 1.7747422409524133e-8 | 1.7793381621463522e-8 |
| Standard deviation | 7.672189756329175e-11 | 1.1859303276122327e-10 | 1.7427432984151867e-10 |
Outlying measurements have no (3.3897912906660226e-3%) effect on estimated standard deviation.
sum/scott/m = 10
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.362156335170868e-7 | 1.3673483689003684e-7 | 1.387311506746756e-7 |
| Standard deviation | 5.271806183580735e-10 | 3.0935379663120288e-9 | 6.999969713676189e-9 |
Outlying measurements have moderate (0.3198739727323602%) effect on estimated standard deviation.
sum/scott/m = 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.2739455620845268e-6 | 1.2752128473726142e-6 | 1.278606452414163e-6 |
| Standard deviation | 2.8274274389303605e-9 | 6.3196950070693445e-9 | 1.2453474242326059e-8 |
Outlying measurements have no (4.8308040343103984e-3%) effect on estimated standard deviation.
sum/scott/m = 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.3116657897975359e-5 | 1.312472895310691e-5 | 1.3141752719664551e-5 |
| Standard deviation | 2.1682082495627586e-8 | 3.754653314475114e-8 | 6.909554775075602e-8 |
Outlying measurements have no (6.289056240987021e-3%) effect on estimated standard deviation.
sum/scott/m = 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.955678663582531e-4 | 1.95743473495007e-4 | 1.9613101563937453e-4 |
| Standard deviation | 2.675754603229487e-7 | 8.306229090226399e-7 | 1.398447433082035e-6 |
Outlying measurements have no (9.614478273164102e-3%) effect on estimated standard deviation.
sum/scott/m = 100000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.8756321875786964e-3 | 2.881692944654911e-3 | 2.888480188264263e-3 |
| Standard deviation | 1.7124086180275214e-5 | 2.1360800337598063e-5 | 2.8405138350179166e-5 |
Outlying measurements have slight (1.9223375624759707e-2%) effect on estimated standard deviation.
sum/builtin/m = 1
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.616281599530407e-9 | 7.621740608842343e-9 | 7.62827237959788e-9 |
| Standard deviation | 1.614863158660443e-11 | 1.9875783162855838e-11 | 2.769680073791901e-11 |
Outlying measurements have no (3.205095067255155e-3%) effect on estimated standard deviation.
sum/builtin/m = 10
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.3988180769797853e-8 | 3.43462776887264e-8 | 3.513385417301273e-8 |
| Standard deviation | 8.121870180591853e-10 | 1.715643226379403e-9 | 2.785731164016383e-9 |
Outlying measurements have severe (0.7236087011302299%) effect on estimated standard deviation.
sum/builtin/m = 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.9238673991515e-7 | 3.925756794301654e-7 | 3.9312822780714077e-7 |
| Standard deviation | 4.17640572415247e-10 | 9.64128350214557e-10 | 1.999326557456315e-9 |
Outlying measurements have no (4.328922495274311e-3%) effect on estimated standard deviation.
sum/builtin/m = 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.878696833580143e-6 | 3.880080071555723e-6 | 3.882357664564203e-6 |
| Standard deviation | 3.944308141933337e-9 | 5.847817709685808e-9 | 9.333537888459357e-9 |
Outlying measurements have no (5.4346203230911634e-3%) effect on estimated standard deviation.
sum/builtin/m = 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.396769929818203e-5 | 9.40628650300239e-5 | 9.420496011732352e-5 |
| Standard deviation | 2.7337911239367083e-7 | 3.843888574967289e-7 | 5.536737337486636e-7 |
Outlying measurements have no (8.402757828210285e-3%) effect on estimated standard deviation.
sum/builtin/m = 100000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.418365255289337e-3 | 1.4260028755078912e-3 | 1.4361778301657338e-3 |
| Standard deviation | 2.3146736015895446e-5 | 2.9714001332912097e-5 | 4.293949468430892e-5 |
Outlying measurements have moderate (0.10264841383648761%) effect on estimated standard deviation.
quicksort/scott/m = 1
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.385726139794326e-8 | 5.395215662640341e-8 | 5.4209347262911444e-8 |
| Standard deviation | 1.9298160637138926e-10 | 4.792662098162156e-10 | 9.956425598936096e-10 |
Outlying measurements have slight (7.310329904250613e-2%) effect on estimated standard deviation.
quicksort/scott/m = 10
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.893703254903301e-7 | 2.8997154032033057e-7 | 2.919774747893664e-7 |
| Standard deviation | 1.0155621980759122e-9 | 3.322792155350185e-9 | 7.1932115323058266e-9 |
Outlying measurements have slight (9.989847993539235e-2%) effect on estimated standard deviation.
quicksort/scott/m = 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.679407327077383e-6 | 2.6806703782027097e-6 | 2.6822146097601975e-6 |
| Standard deviation | 3.692015691276549e-9 | 4.7002146752445924e-9 | 6.681134514545089e-9 |
Outlying measurements have no (5.208190564951599e-3%) effect on estimated standard deviation.
quicksort/scott/m = 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.316849261362858e-5 | 3.325760326029437e-5 | 3.3385348761677415e-5 |
| Standard deviation | 2.4617703237943294e-7 | 3.525963554096806e-7 | 4.750196173530088e-7 |
Outlying measurements have no (7.142487448889792e-3%) effect on estimated standard deviation.
quicksort/scott/m = 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.8535447005397103e-4 | 3.8600137323875167e-4 | 3.866973011637479e-4 |
| Standard deviation | 1.7520001981477607e-6 | 2.247327333543461e-6 | 2.932076779873416e-6 |
Outlying measurements have slight (1.0987654320987826e-2%) effect on estimated standard deviation.
quicksort/scott/m = 100000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.8373735485233525e-3 | 5.866821467672924e-3 | 5.897905902360164e-3 |
| Standard deviation | 7.05669240201353e-5 | 9.123434932253806e-5 | 1.2493666903852654e-4 |
Outlying measurements have slight (2.5623268698060937e-2%) effect on estimated standard deviation.
quicksort/builtin/m = 1
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.1157046596676875e-8 | 2.1311085053924353e-8 | 2.167843947181255e-8 |
| Standard deviation | 9.932958936109794e-11 | 7.192792143883034e-10 | 1.2719793013482185e-9 |
Outlying measurements have severe (0.5503910415822382%) effect on estimated standard deviation.
quicksort/builtin/m = 10
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.4028886770318488e-7 | 1.407248871241875e-7 | 1.4192113788877615e-7 |
| Standard deviation | 1.040928639108847e-9 | 2.2295906851599556e-9 | 4.578336618219922e-9 |
Outlying measurements have moderate (0.1857680191834429%) effect on estimated standard deviation.
quicksort/builtin/m = 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.5820941293236924e-6 | 1.5832123175178207e-6 | 1.5863258742266194e-6 |
| Standard deviation | 2.4840051170463227e-9 | 5.797985380565039e-9 | 1.1783352569436739e-8 |
Outlying measurements have no (4.925987648269566e-3%) effect on estimated standard deviation.
quicksort/builtin/m = 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.9780358601822367e-5 | 1.9799966928720453e-5 | 1.9838917536736912e-5 |
| Standard deviation | 5.4228021805331376e-8 | 8.818666674322688e-8 | 1.5183556259760886e-7 |
Outlying measurements have no (6.622222222222132e-3%) effect on estimated standard deviation.
quicksort/builtin/m = 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.3543081230425052e-4 | 2.3571025041911495e-4 | 2.3635510889773599e-4 |
| Standard deviation | 7.05105105336345e-7 | 1.35184393130913e-6 | 2.6841930463791244e-6 |
Outlying measurements have no (9.90000000000013e-3%) effect on estimated standard deviation.
quicksort/builtin/m = 100000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.638133131297349e-3 | 4.6619972717691395e-3 | 4.685940343226301e-3 |
| Standard deviation | 6.145406681257957e-5 | 7.404104199148272e-5 | 9.07199579015445e-5 |
Outlying measurements have slight (2.3242630385487847e-2%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.