criterion performance measurements
overview
want to understand this report?
head_tail/scott/m = 2
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 1.7992850178506668e-8 | 1.804730771607263e-8 | 1.829081917733283e-8 |
Standard deviation | 5.661107224389133e-11 | 3.3419976977269315e-10 | 7.62620097173348e-10 |
Outlying measurements have moderate (0.2668551467150451%) effect on estimated standard deviation.
head_tail/scott/m = 10
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 1.7708729002473197e-8 | 1.7789106194031686e-8 | 1.7956837895901793e-8 |
Standard deviation | 1.9388860199689512e-10 | 3.863320341663348e-10 | 6.300494201365577e-10 |
Outlying measurements have moderate (0.3365095909288198%) effect on estimated standard deviation.
head_tail/scott/m = 100
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 1.7683977208235944e-8 | 1.7886831959184684e-8 | 1.8957544479791642e-8 |
Standard deviation | 4.322026812443842e-11 | 1.0735109495717544e-9 | 2.6632365646807677e-9 |
Outlying measurements have severe (0.7998547141106558%) effect on estimated standard deviation.
head_tail/scott/m = 1000
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 1.7653073024890006e-8 | 1.7657787138584573e-8 | 1.766651711503798e-8 |
Standard deviation | 1.1012468589570872e-11 | 2.0401863910287583e-11 | 3.227869237607742e-11 |
Outlying measurements have no (3.3897912906659263e-3%) effect on estimated standard deviation.
head_tail/scott/m = 10000
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 1.7891665094304186e-8 | 1.799257826259344e-8 | 1.8392464610187854e-8 |
Standard deviation | 5.304144522638525e-11 | 6.329516783618791e-10 | 1.3434564084313105e-9 |
Outlying measurements have severe (0.5735652042606535%) effect on estimated standard deviation.
head_tail/scott/m = 100000
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 1.789546942325028e-8 | 1.7997615387223256e-8 | 1.8391024166552373e-8 |
Standard deviation | 6.265723333694516e-11 | 6.383855819401488e-10 | 1.3515717054732625e-9 |
Outlying measurements have severe (0.5770408708920007%) effect on estimated standard deviation.
head_tail/builtin/m = 2
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 5.829464108711504e-9 | 5.8314727200384325e-9 | 5.834308312264796e-9 |
Standard deviation | 5.1531849528833655e-12 | 7.859446533308033e-12 | 1.1827230711358077e-11 |
Outlying measurements have no (3.1446227945347255e-3%) effect on estimated standard deviation.
head_tail/builtin/m = 10
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 5.825640030015009e-9 | 5.8280060209627366e-9 | 5.833719413060676e-9 |
Standard deviation | 5.3583935949442525e-12 | 1.1370125198109252e-11 | 2.0185427259733204e-11 |
Outlying measurements have no (3.144622794535031e-3%) effect on estimated standard deviation.
head_tail/builtin/m = 100
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 5.824634350688852e-9 | 5.8257951632758414e-9 | 5.828262886457277e-9 |
Standard deviation | 1.5638674284156012e-12 | 5.514831985629767e-12 | 9.673348104208956e-12 |
Outlying measurements have no (3.144622794534725e-3%) effect on estimated standard deviation.
head_tail/builtin/m = 1000
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 5.844698294239764e-9 | 5.876762040556132e-9 | 5.949709249735585e-9 |
Standard deviation | 8.466202177137128e-11 | 1.488270458581433e-10 | 2.420536193832449e-10 |
Outlying measurements have moderate (0.4249557521210125%) effect on estimated standard deviation.
head_tail/builtin/m = 10000
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 5.836901221673448e-9 | 5.840023749373232e-9 | 5.844155277433244e-9 |
Standard deviation | 9.691528528513555e-12 | 1.211669956623533e-11 | 1.4790729745597775e-11 |
Outlying measurements have no (3.144622794534853e-3%) effect on estimated standard deviation.
head_tail/builtin/m = 100000
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 5.8325310349525534e-9 | 5.841595465998009e-9 | 5.8608957740847956e-9 |
Standard deviation | 1.2369318831363987e-11 | 4.16363147915444e-11 | 6.958441912911937e-11 |
Outlying measurements have slight (3.271720566076735e-2%) effect on estimated standard deviation.
sum/scott/m = 1
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 1.7366962946106725e-8 | 1.738852764809992e-8 | 1.7426780355690542e-8 |
Standard deviation | 5.971465278217075e-11 | 8.986021717510579e-11 | 1.297908889846156e-10 |
Outlying measurements have no (3.389791290666176e-3%) effect on estimated standard deviation.
sum/scott/m = 10
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 1.310423593189949e-7 | 1.3112166695565418e-7 | 1.3131027639898203e-7 |
Standard deviation | 1.8435144913849632e-10 | 3.7381350898663985e-10 | 6.968314678827698e-10 |
Outlying measurements have no (3.9369463669170525e-3%) effect on estimated standard deviation.
sum/scott/m = 100
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 1.2415364488426404e-6 | 1.2427428650006464e-6 | 1.2485694900200447e-6 |
Standard deviation | 1.0467001890437262e-9 | 7.0512549114404885e-9 | 1.588605162893079e-8 |
Outlying measurements have no (4.8075801068869235e-3%) effect on estimated standard deviation.
sum/scott/m = 1000
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 1.2869968580977384e-5 | 1.2885875790548907e-5 | 1.2942534134024437e-5 |
Standard deviation | 1.4048548587524178e-8 | 9.481145048302703e-8 | 2.0127989148175397e-7 |
Outlying measurements have no (6.249752778766577e-3%) effect on estimated standard deviation.
sum/scott/m = 10000
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 1.9850043922738003e-4 | 1.9878581654022407e-4 | 1.998796580208583e-4 |
Standard deviation | 3.1627765528893965e-7 | 1.7612968251475953e-6 | 3.6989738965777894e-6 |
Outlying measurements have no (9.614478273164315e-3%) effect on estimated standard deviation.
sum/scott/m = 100000
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 3.1327075284520987e-3 | 3.1397131413121184e-3 | 3.1464198424318015e-3 |
Standard deviation | 1.8751873306651784e-5 | 2.1711987401602704e-5 | 2.6040050375413755e-5 |
Outlying measurements have slight (1.99916701374427e-2%) effect on estimated standard deviation.
sum/scott/m = 1000000
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 3.2915689798806116e-2 | 3.333277182970768e-2 | 3.4301262161189366e-2 |
Standard deviation | 4.821204280394248e-4 | 1.306561949588707e-3 | 2.2977506644716963e-3 |
Outlying measurements have moderate (0.11506877283931725%) effect on estimated standard deviation.
sum/scott/m = 10000000
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 0.3137484718754422 | 0.3464537786882526 | 0.3721711414400488 |
Standard deviation | 2.6276852596803457e-2 | 3.530241080188055e-2 | 4.2414818135637786e-2 |
Outlying measurements have moderate (0.22648624828191072%) effect on estimated standard deviation.
sum/builtin/m = 1
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 7.879859201299036e-9 | 7.881902368671846e-9 | 7.885852764752944e-9 |
Standard deviation | 5.257001883986502e-12 | 9.197787670433515e-12 | 1.604716059286634e-11 |
Outlying measurements have no (3.2050950672550226e-3%) effect on estimated standard deviation.
sum/builtin/m = 10
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 3.469357375425575e-8 | 3.538379955279587e-8 | 3.700777940428144e-8 |
Standard deviation | 1.7445223217271666e-9 | 3.198814103691838e-9 | 6.0411964766268115e-9 |
Outlying measurements have severe (0.8979967747656372%) effect on estimated standard deviation.
sum/builtin/m = 100
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 3.8834308205647174e-7 | 3.884394571507781e-7 | 3.8867522739818326e-7 |
Standard deviation | 2.2204904805392038e-10 | 4.959305218356728e-10 | 1.018138084320866e-9 |
Outlying measurements have no (4.310264050523791e-3%) effect on estimated standard deviation.
sum/builtin/m = 1000
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 3.89089360102956e-6 | 3.892951812400013e-6 | 3.901416063051244e-6 |
Standard deviation | 3.1523030114520307e-9 | 1.1782448936638624e-8 | 2.6205888379099268e-8 |
Outlying measurements have no (5.434620323091123e-3%) effect on estimated standard deviation.
sum/builtin/m = 10000
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 9.678954872605523e-5 | 9.701601089106193e-5 | 9.759081154400457e-5 |
Standard deviation | 2.1306462296189328e-7 | 1.1782617435057598e-6 | 2.161440822738816e-6 |
Outlying measurements have slight (6.367034389026507e-2%) effect on estimated standard deviation.
sum/builtin/m = 100000
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 1.5114787162840942e-3 | 1.5207315984847804e-3 | 1.5330503053168575e-3 |
Standard deviation | 2.6210402834204435e-5 | 3.5478908075241624e-5 | 4.881612068647569e-5 |
Outlying measurements have moderate (0.12132340831396626%) effect on estimated standard deviation.
sum/builtin/m = 1000000
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 2.60458861232147e-2 | 2.754663803794803e-2 | 2.8980704911014635e-2 |
Standard deviation | 1.9017554709458958e-3 | 3.0244295953289453e-3 | 4.693940755486009e-3 |
Outlying measurements have moderate (0.48199729453917683%) effect on estimated standard deviation.
sum/builtin/m = 10000000
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 0.22843750339816324 | 0.27258929516305214 | 0.30298865341581405 |
Standard deviation | 2.546487753692674e-2 | 4.882184085264587e-2 | 7.429059203265195e-2 |
Outlying measurements have moderate (0.3844131162513758%) effect on estimated standard deviation.
quicksort/scott/m = 1
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 4.981099463429329e-8 | 4.984842020432251e-8 | 4.991031562411637e-8 |
Standard deviation | 9.945022123150577e-11 | 1.5909180706682726e-10 | 2.310323909242628e-10 |
Outlying measurements have no (3.6495860671683306e-3%) effect on estimated standard deviation.
quicksort/scott/m = 10
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 1.620289681217871e-7 | 1.621788053256447e-7 | 1.6272554908984616e-7 |
Standard deviation | 1.362214411538339e-10 | 9.025794150509482e-10 | 1.900346076472408e-9 |
Outlying measurements have no (3.9999354849115185e-3%) effect on estimated standard deviation.
quicksort/scott/m = 100
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 2.108863793450608e-6 | 2.1103796612086627e-6 | 2.1136718164900823e-6 |
Standard deviation | 3.796821257355922e-9 | 6.914615389819679e-9 | 1.250990380916559e-8 |
Outlying measurements have no (5.076009995835182e-3%) effect on estimated standard deviation.
quicksort/scott/m = 1000
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 1.8995153087749136e-5 | 1.9024512119694283e-5 | 1.913194774651349e-5 |
Standard deviation | 1.791592498112745e-8 | 1.7605172706959822e-7 | 3.704917407918162e-7 |
Outlying measurements have no (6.578658830753057e-3%) effect on estimated standard deviation.
quicksort/scott/m = 10000
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 2.0721706574598897e-4 | 2.099096663966401e-4 | 2.153109440609313e-4 |
Standard deviation | 7.153839619119676e-6 | 1.1619133407440622e-5 | 1.756042079501959e-5 |
Outlying measurements have severe (0.533949617804793%) effect on estimated standard deviation.
quicksort/scott/m = 100000
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 3.113330815555108e-3 | 3.1337005465993995e-3 | 3.1712151902602167e-3 |
Standard deviation | 4.7751279598046136e-5 | 8.635515107193482e-5 | 1.6104865906099462e-4 |
Outlying measurements have moderate (0.13428742830848256%) effect on estimated standard deviation.
quicksort/scott/m = 1000000
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 2.9464058294469894e-2 | 3.0666805762288005e-2 | 3.198353549339779e-2 |
Standard deviation | 1.7852831809553194e-3 | 2.6857462081709416e-3 | 4.45009235606469e-3 |
Outlying measurements have moderate (0.3380138199649157%) effect on estimated standard deviation.
quicksort/scott/m = 10000000
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 0.26683292431407607 | 0.2967008492889969 | 0.3249069805703281 |
Standard deviation | 3.1931725144684675e-2 | 3.945196924616204e-2 | 4.1941847483795765e-2 |
Outlying measurements have moderate (0.37172059865961216%) effect on estimated standard deviation.
quicksort/builtin/m = 1
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 2.2919002845299586e-8 | 2.2959474815014476e-8 | 2.3150009114236364e-8 |
Standard deviation | 2.4043979434680056e-11 | 2.5446108557099367e-10 | 5.829323817555849e-10 |
Outlying measurements have moderate (0.11465156057718688%) effect on estimated standard deviation.
quicksort/builtin/m = 10
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 8.375050374669949e-8 | 8.377228002559787e-8 | 8.381569177518925e-8 |
Standard deviation | 5.624801646466131e-11 | 9.761798036078041e-11 | 1.607289633563907e-10 |
Outlying measurements have no (3.8022259775071384e-3%) effect on estimated standard deviation.
quicksort/builtin/m = 100
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 1.1314934454117767e-6 | 1.1316289081724167e-6 | 1.1318041960116447e-6 |
Standard deviation | 4.2645096914031376e-10 | 5.224876940817762e-10 | 6.840319776544779e-10 |
Outlying measurements have no (4.761795746434375e-3%) effect on estimated standard deviation.
quicksort/builtin/m = 1000
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 1.3105947809105735e-5 | 1.3379238801462811e-5 | 1.392958568588321e-5 |
Standard deviation | 7.075037500605585e-7 | 1.172151435687937e-6 | 2.014128637433561e-6 |
Outlying measurements have severe (0.8202569225233002%) effect on estimated standard deviation.
quicksort/builtin/m = 10000
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 1.4372446330376384e-4 | 1.437986780712871e-4 | 1.438836535880085e-4 |
Standard deviation | 2.390890655796052e-7 | 2.7836646539629375e-7 | 3.411127158247468e-7 |
Outlying measurements have no (9.008264462809909e-3%) effect on estimated standard deviation.
quicksort/builtin/m = 100000
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 2.2227721079104558e-3 | 2.4220771756984214e-3 | 2.9065115826626777e-3 |
Standard deviation | 4.585096201218045e-4 | 9.457142062543907e-4 | 1.8021998224659718e-3 |
Outlying measurements have severe (0.9796531850391975%) effect on estimated standard deviation.
quicksort/builtin/m = 1000000
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 2.107352910096478e-2 | 2.3295837315281e-2 | 2.6638754608844715e-2 |
Standard deviation | 4.473708384207577e-3 | 6.124490722972341e-3 | 8.69296596847458e-3 |
Outlying measurements have severe (0.833667722554653%) effect on estimated standard deviation.
quicksort/builtin/m = 10000000
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | xxx | xxx | xxx |
R² goodness-of-fit | xxx | xxx | xxx |
Mean execution time | 0.2056626216831824 | 0.23229039711780688 | 0.258409362727414 |
Standard deviation | 2.7903331857832045e-2 | 3.761469698554845e-2 | 4.8730976419746615e-2 |
Outlying measurements have moderate (0.47517117531404995%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.