criterion performance measurements

overview

want to understand this report?

head_tail/scott/m = 2

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.7992850178506668e-8 1.804730771607263e-8 1.829081917733283e-8
Standard deviation 5.661107224389133e-11 3.3419976977269315e-10 7.62620097173348e-10

Outlying measurements have moderate (0.2668551467150451%) effect on estimated standard deviation.

head_tail/scott/m = 10

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.7708729002473197e-8 1.7789106194031686e-8 1.7956837895901793e-8
Standard deviation 1.9388860199689512e-10 3.863320341663348e-10 6.300494201365577e-10

Outlying measurements have moderate (0.3365095909288198%) effect on estimated standard deviation.

head_tail/scott/m = 100

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.7683977208235944e-8 1.7886831959184684e-8 1.8957544479791642e-8
Standard deviation 4.322026812443842e-11 1.0735109495717544e-9 2.6632365646807677e-9

Outlying measurements have severe (0.7998547141106558%) effect on estimated standard deviation.

head_tail/scott/m = 1000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.7653073024890006e-8 1.7657787138584573e-8 1.766651711503798e-8
Standard deviation 1.1012468589570872e-11 2.0401863910287583e-11 3.227869237607742e-11

Outlying measurements have no (3.3897912906659263e-3%) effect on estimated standard deviation.

head_tail/scott/m = 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.7891665094304186e-8 1.799257826259344e-8 1.8392464610187854e-8
Standard deviation 5.304144522638525e-11 6.329516783618791e-10 1.3434564084313105e-9

Outlying measurements have severe (0.5735652042606535%) effect on estimated standard deviation.

head_tail/scott/m = 100000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.789546942325028e-8 1.7997615387223256e-8 1.8391024166552373e-8
Standard deviation 6.265723333694516e-11 6.383855819401488e-10 1.3515717054732625e-9

Outlying measurements have severe (0.5770408708920007%) effect on estimated standard deviation.

head_tail/builtin/m = 2

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.829464108711504e-9 5.8314727200384325e-9 5.834308312264796e-9
Standard deviation 5.1531849528833655e-12 7.859446533308033e-12 1.1827230711358077e-11

Outlying measurements have no (3.1446227945347255e-3%) effect on estimated standard deviation.

head_tail/builtin/m = 10

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.825640030015009e-9 5.8280060209627366e-9 5.833719413060676e-9
Standard deviation 5.3583935949442525e-12 1.1370125198109252e-11 2.0185427259733204e-11

Outlying measurements have no (3.144622794535031e-3%) effect on estimated standard deviation.

head_tail/builtin/m = 100

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.824634350688852e-9 5.8257951632758414e-9 5.828262886457277e-9
Standard deviation 1.5638674284156012e-12 5.514831985629767e-12 9.673348104208956e-12

Outlying measurements have no (3.144622794534725e-3%) effect on estimated standard deviation.

head_tail/builtin/m = 1000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.844698294239764e-9 5.876762040556132e-9 5.949709249735585e-9
Standard deviation 8.466202177137128e-11 1.488270458581433e-10 2.420536193832449e-10

Outlying measurements have moderate (0.4249557521210125%) effect on estimated standard deviation.

head_tail/builtin/m = 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.836901221673448e-9 5.840023749373232e-9 5.844155277433244e-9
Standard deviation 9.691528528513555e-12 1.211669956623533e-11 1.4790729745597775e-11

Outlying measurements have no (3.144622794534853e-3%) effect on estimated standard deviation.

head_tail/builtin/m = 100000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.8325310349525534e-9 5.841595465998009e-9 5.8608957740847956e-9
Standard deviation 1.2369318831363987e-11 4.16363147915444e-11 6.958441912911937e-11

Outlying measurements have slight (3.271720566076735e-2%) effect on estimated standard deviation.

sum/scott/m = 1

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.7366962946106725e-8 1.738852764809992e-8 1.7426780355690542e-8
Standard deviation 5.971465278217075e-11 8.986021717510579e-11 1.297908889846156e-10

Outlying measurements have no (3.389791290666176e-3%) effect on estimated standard deviation.

sum/scott/m = 10

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.310423593189949e-7 1.3112166695565418e-7 1.3131027639898203e-7
Standard deviation 1.8435144913849632e-10 3.7381350898663985e-10 6.968314678827698e-10

Outlying measurements have no (3.9369463669170525e-3%) effect on estimated standard deviation.

sum/scott/m = 100

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.2415364488426404e-6 1.2427428650006464e-6 1.2485694900200447e-6
Standard deviation 1.0467001890437262e-9 7.0512549114404885e-9 1.588605162893079e-8

Outlying measurements have no (4.8075801068869235e-3%) effect on estimated standard deviation.

sum/scott/m = 1000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.2869968580977384e-5 1.2885875790548907e-5 1.2942534134024437e-5
Standard deviation 1.4048548587524178e-8 9.481145048302703e-8 2.0127989148175397e-7

Outlying measurements have no (6.249752778766577e-3%) effect on estimated standard deviation.

sum/scott/m = 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.9850043922738003e-4 1.9878581654022407e-4 1.998796580208583e-4
Standard deviation 3.1627765528893965e-7 1.7612968251475953e-6 3.6989738965777894e-6

Outlying measurements have no (9.614478273164315e-3%) effect on estimated standard deviation.

sum/scott/m = 100000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.1327075284520987e-3 3.1397131413121184e-3 3.1464198424318015e-3
Standard deviation 1.8751873306651784e-5 2.1711987401602704e-5 2.6040050375413755e-5

Outlying measurements have slight (1.99916701374427e-2%) effect on estimated standard deviation.

sum/scott/m = 1000000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.2915689798806116e-2 3.333277182970768e-2 3.4301262161189366e-2
Standard deviation 4.821204280394248e-4 1.306561949588707e-3 2.2977506644716963e-3

Outlying measurements have moderate (0.11506877283931725%) effect on estimated standard deviation.

sum/scott/m = 10000000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.3137484718754422 0.3464537786882526 0.3721711414400488
Standard deviation 2.6276852596803457e-2 3.530241080188055e-2 4.2414818135637786e-2

Outlying measurements have moderate (0.22648624828191072%) effect on estimated standard deviation.

sum/builtin/m = 1

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 7.879859201299036e-9 7.881902368671846e-9 7.885852764752944e-9
Standard deviation 5.257001883986502e-12 9.197787670433515e-12 1.604716059286634e-11

Outlying measurements have no (3.2050950672550226e-3%) effect on estimated standard deviation.

sum/builtin/m = 10

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.469357375425575e-8 3.538379955279587e-8 3.700777940428144e-8
Standard deviation 1.7445223217271666e-9 3.198814103691838e-9 6.0411964766268115e-9

Outlying measurements have severe (0.8979967747656372%) effect on estimated standard deviation.

sum/builtin/m = 100

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.8834308205647174e-7 3.884394571507781e-7 3.8867522739818326e-7
Standard deviation 2.2204904805392038e-10 4.959305218356728e-10 1.018138084320866e-9

Outlying measurements have no (4.310264050523791e-3%) effect on estimated standard deviation.

sum/builtin/m = 1000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.89089360102956e-6 3.892951812400013e-6 3.901416063051244e-6
Standard deviation 3.1523030114520307e-9 1.1782448936638624e-8 2.6205888379099268e-8

Outlying measurements have no (5.434620323091123e-3%) effect on estimated standard deviation.

sum/builtin/m = 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 9.678954872605523e-5 9.701601089106193e-5 9.759081154400457e-5
Standard deviation 2.1306462296189328e-7 1.1782617435057598e-6 2.161440822738816e-6

Outlying measurements have slight (6.367034389026507e-2%) effect on estimated standard deviation.

sum/builtin/m = 100000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.5114787162840942e-3 1.5207315984847804e-3 1.5330503053168575e-3
Standard deviation 2.6210402834204435e-5 3.5478908075241624e-5 4.881612068647569e-5

Outlying measurements have moderate (0.12132340831396626%) effect on estimated standard deviation.

sum/builtin/m = 1000000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.60458861232147e-2 2.754663803794803e-2 2.8980704911014635e-2
Standard deviation 1.9017554709458958e-3 3.0244295953289453e-3 4.693940755486009e-3

Outlying measurements have moderate (0.48199729453917683%) effect on estimated standard deviation.

sum/builtin/m = 10000000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.22843750339816324 0.27258929516305214 0.30298865341581405
Standard deviation 2.546487753692674e-2 4.882184085264587e-2 7.429059203265195e-2

Outlying measurements have moderate (0.3844131162513758%) effect on estimated standard deviation.

quicksort/scott/m = 1

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 4.981099463429329e-8 4.984842020432251e-8 4.991031562411637e-8
Standard deviation 9.945022123150577e-11 1.5909180706682726e-10 2.310323909242628e-10

Outlying measurements have no (3.6495860671683306e-3%) effect on estimated standard deviation.

quicksort/scott/m = 10

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.620289681217871e-7 1.621788053256447e-7 1.6272554908984616e-7
Standard deviation 1.362214411538339e-10 9.025794150509482e-10 1.900346076472408e-9

Outlying measurements have no (3.9999354849115185e-3%) effect on estimated standard deviation.

quicksort/scott/m = 100

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.108863793450608e-6 2.1103796612086627e-6 2.1136718164900823e-6
Standard deviation 3.796821257355922e-9 6.914615389819679e-9 1.250990380916559e-8

Outlying measurements have no (5.076009995835182e-3%) effect on estimated standard deviation.

quicksort/scott/m = 1000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.8995153087749136e-5 1.9024512119694283e-5 1.913194774651349e-5
Standard deviation 1.791592498112745e-8 1.7605172706959822e-7 3.704917407918162e-7

Outlying measurements have no (6.578658830753057e-3%) effect on estimated standard deviation.

quicksort/scott/m = 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.0721706574598897e-4 2.099096663966401e-4 2.153109440609313e-4
Standard deviation 7.153839619119676e-6 1.1619133407440622e-5 1.756042079501959e-5

Outlying measurements have severe (0.533949617804793%) effect on estimated standard deviation.

quicksort/scott/m = 100000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.113330815555108e-3 3.1337005465993995e-3 3.1712151902602167e-3
Standard deviation 4.7751279598046136e-5 8.635515107193482e-5 1.6104865906099462e-4

Outlying measurements have moderate (0.13428742830848256%) effect on estimated standard deviation.

quicksort/scott/m = 1000000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.9464058294469894e-2 3.0666805762288005e-2 3.198353549339779e-2
Standard deviation 1.7852831809553194e-3 2.6857462081709416e-3 4.45009235606469e-3

Outlying measurements have moderate (0.3380138199649157%) effect on estimated standard deviation.

quicksort/scott/m = 10000000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.26683292431407607 0.2967008492889969 0.3249069805703281
Standard deviation 3.1931725144684675e-2 3.945196924616204e-2 4.1941847483795765e-2

Outlying measurements have moderate (0.37172059865961216%) effect on estimated standard deviation.

quicksort/builtin/m = 1

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.2919002845299586e-8 2.2959474815014476e-8 2.3150009114236364e-8
Standard deviation 2.4043979434680056e-11 2.5446108557099367e-10 5.829323817555849e-10

Outlying measurements have moderate (0.11465156057718688%) effect on estimated standard deviation.

quicksort/builtin/m = 10

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 8.375050374669949e-8 8.377228002559787e-8 8.381569177518925e-8
Standard deviation 5.624801646466131e-11 9.761798036078041e-11 1.607289633563907e-10

Outlying measurements have no (3.8022259775071384e-3%) effect on estimated standard deviation.

quicksort/builtin/m = 100

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.1314934454117767e-6 1.1316289081724167e-6 1.1318041960116447e-6
Standard deviation 4.2645096914031376e-10 5.224876940817762e-10 6.840319776544779e-10

Outlying measurements have no (4.761795746434375e-3%) effect on estimated standard deviation.

quicksort/builtin/m = 1000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.3105947809105735e-5 1.3379238801462811e-5 1.392958568588321e-5
Standard deviation 7.075037500605585e-7 1.172151435687937e-6 2.014128637433561e-6

Outlying measurements have severe (0.8202569225233002%) effect on estimated standard deviation.

quicksort/builtin/m = 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.4372446330376384e-4 1.437986780712871e-4 1.438836535880085e-4
Standard deviation 2.390890655796052e-7 2.7836646539629375e-7 3.411127158247468e-7

Outlying measurements have no (9.008264462809909e-3%) effect on estimated standard deviation.

quicksort/builtin/m = 100000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.2227721079104558e-3 2.4220771756984214e-3 2.9065115826626777e-3
Standard deviation 4.585096201218045e-4 9.457142062543907e-4 1.8021998224659718e-3

Outlying measurements have severe (0.9796531850391975%) effect on estimated standard deviation.

quicksort/builtin/m = 1000000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.107352910096478e-2 2.3295837315281e-2 2.6638754608844715e-2
Standard deviation 4.473708384207577e-3 6.124490722972341e-3 8.69296596847458e-3

Outlying measurements have severe (0.833667722554653%) effect on estimated standard deviation.

quicksort/builtin/m = 10000000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.2056626216831824 0.23229039711780688 0.258409362727414
Standard deviation 2.7903331857832045e-2 3.761469698554845e-2 4.8730976419746615e-2

Outlying measurements have moderate (0.47517117531404995%) effect on estimated standard deviation.

understanding this report

In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.

Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.

We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)

A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.