criterion performance measurements

overview

want to understand this report?

head_tail/scott/m = 2

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.502692096499578e-8 2.5083931226921062e-8 2.5309698755561055e-8
Standard deviation 5.2920930029778725e-11 3.3351187846939437e-10 7.529989041123306e-10

Outlying measurements have moderate (0.15550385367719996%) effect on estimated standard deviation.

head_tail/scott/m = 10

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.497716610439414e-8 2.5020325572352206e-8 2.5179919858362048e-8
Standard deviation 5.6019361924005273e-11 2.4781619899466506e-10 5.48017621025603e-10

Outlying measurements have slight (9.22787502164525e-2%) effect on estimated standard deviation.

head_tail/scott/m = 100

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.4973732252843656e-8 2.4998482023235323e-8 2.5057632407616097e-8
Standard deviation 5.366224145124094e-11 1.2310911464884628e-10 2.288707845037391e-10

Outlying measurements have no (3.472180067743891e-3%) effect on estimated standard deviation.

head_tail/scott/m = 1000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.5079088640944055e-8 2.5147315939551972e-8 2.5362951044869513e-8
Standard deviation 1.292341872916161e-10 3.716192594934904e-10 7.829045552867413e-10

Outlying measurements have moderate (0.18563039905651924%) effect on estimated standard deviation.

head_tail/scott/m = 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.506350230354071e-8 2.5096057998136204e-8 2.5162522620667635e-8
Standard deviation 8.900877481803543e-11 1.530681749821353e-10 2.7160797378535365e-10

Outlying measurements have no (3.4721800677439394e-3%) effect on estimated standard deviation.

head_tail/scott/m = 100000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.5006249443983558e-8 2.502167849519524e-8 2.5048965411286096e-8
Standard deviation 4.102491363876129e-11 6.708999135912257e-11 1.0994466785020072e-10

Outlying measurements have no (3.472180067743937e-3%) effect on estimated standard deviation.

head_tail/builtin/m = 2

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 7.459817732838883e-9 7.479394385969968e-9 7.548636803671931e-9
Standard deviation 3.252952055412365e-11 1.0905531066531079e-10 2.367403418742641e-10

Outlying measurements have moderate (0.1923185368608657%) effect on estimated standard deviation.

head_tail/builtin/m = 10

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 7.44932825198303e-9 7.452682686363629e-9 7.456828240219058e-9
Standard deviation 1.0278807807644676e-11 1.2493424789847422e-11 1.6782465718975615e-11

Outlying measurements have no (3.194855358316837e-3%) effect on estimated standard deviation.

head_tail/builtin/m = 100

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 7.458155727460366e-9 7.461556025081729e-9 7.4659667516518e-9
Standard deviation 9.861139898048307e-12 1.2902321112502434e-11 1.804054858558882e-11

Outlying measurements have no (3.1948553583171103e-3%) effect on estimated standard deviation.

head_tail/builtin/m = 1000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 7.472215130930422e-9 7.539601211912847e-9 7.723691250886432e-9
Standard deviation 1.3669016933489196e-10 3.4580328862406695e-10 7.064964255043795e-10

Outlying measurements have severe (0.7078671747560081%) effect on estimated standard deviation.

head_tail/builtin/m = 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 7.467802723115954e-9 7.474321215737475e-9 7.489783273647602e-9
Standard deviation 1.5769430516844746e-11 3.176652216684969e-11 6.248213948510986e-11

Outlying measurements have no (3.1948553583170444e-3%) effect on estimated standard deviation.

head_tail/builtin/m = 100000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 7.468275843800141e-9 7.50366189612358e-9 7.64273046552581e-9
Standard deviation 9.39276469111258e-12 2.2592959945008773e-10 4.809636418443974e-10

Outlying measurements have severe (0.5103778750325372%) effect on estimated standard deviation.

sum/scott/m = 1

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.4482160027427458e-8 2.454611823348937e-8 2.4672656948901892e-8
Standard deviation 1.6679076219275186e-10 2.890337303289019e-10 4.88978551618987e-10

Outlying measurements have moderate (0.12552066850726334%) effect on estimated standard deviation.

sum/scott/m = 10

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.0210065028064758e-7 2.02689923167831e-7 2.0504472350386877e-7
Standard deviation 2.2846483066055218e-10 3.6803080243555203e-9 7.823736755916304e-9

Outlying measurements have moderate (0.22672996897475423%) effect on estimated standard deviation.

sum/scott/m = 100

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.9254868296666653e-6 1.9271633931936864e-6 1.9316817262686095e-6
Standard deviation 3.3116779263250967e-9 8.714533751409524e-9 1.800744337418105e-8

Outlying measurements have no (5.024997449240138e-3%) effect on estimated standard deviation.

sum/scott/m = 1000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.6396578049921185e-5 2.6427638801332984e-5 2.6498235438172643e-5
Standard deviation 7.337571670627947e-8 1.474208945113255e-7 2.8005784685305525e-7

Outlying measurements have no (6.8962191358025345e-3%) effect on estimated standard deviation.

sum/scott/m = 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.0537111511332634e-4 3.075288175028507e-4 3.1382197143662606e-4
Standard deviation 2.9809667846479994e-6 1.1399053439905687e-5 2.418690801137118e-5

Outlying measurements have moderate (0.3264048066597953%) effect on estimated standard deviation.

sum/scott/m = 100000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 8.260166086912197e-3 8.327124649046457e-3 8.391164299403068e-3
Standard deviation 1.4986687520201933e-4 1.8463094586949823e-4 2.3494019182459242e-4

Outlying measurements have slight (5.706837414218139e-2%) effect on estimated standard deviation.

sum/builtin/m = 1

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.3919128295065323e-8 1.392605178373867e-8 1.3939502927357205e-8
Standard deviation 1.9692025824931506e-11 3.1747490950196256e-11 5.425706194867483e-11

Outlying measurements have no (3.333296048142493e-3%) effect on estimated standard deviation.

sum/builtin/m = 10

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.1033380463094953e-7 1.1066364761122834e-7 1.1218983670042338e-7
Standard deviation 1.7274247032183386e-10 2.0204761047335895e-9 4.615161479694522e-9

Outlying measurements have moderate (0.23500497034292245%) effect on estimated standard deviation.

sum/builtin/m = 100

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.0311682419816378e-6 1.0378626285323937e-6 1.0530740396680884e-6
Standard deviation 1.3813515065908383e-8 3.242191623039945e-8 5.6385957169351515e-8

Outlying measurements have moderate (0.4279040707203486%) effect on estimated standard deviation.

sum/builtin/m = 1000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.7114177425096207e-5 1.7128004708931198e-5 1.715417373893765e-5
Standard deviation 3.406607315617793e-8 6.209450098774435e-8 9.64728661186352e-8

Outlying measurements have no (6.493229099918523e-3%) effect on estimated standard deviation.

sum/builtin/m = 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.0311549765897338e-4 2.038761289702128e-4 2.0517600436782576e-4
Standard deviation 1.8516141501652293e-6 3.291723057141546e-6 5.119691454188461e-6

Outlying measurements have slight (9.160215257921459e-2%) effect on estimated standard deviation.

sum/builtin/m = 100000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 6.663107392152159e-3 6.706197321660968e-3 6.756234604604107e-3
Standard deviation 1.1121767386583366e-4 1.3414368731996676e-4 1.689637273853729e-4

Outlying measurements have slight (2.700617283950618e-2%) effect on estimated standard deviation.

quicksort/scott/m = 1

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.0254502989479755e-7 1.0461635145543386e-7 1.0790406940195964e-7
Standard deviation 5.7012094697604596e-9 8.455055253011012e-9 1.1221497189036935e-8

Outlying measurements have severe (0.8645242966313802%) effect on estimated standard deviation.

quicksort/scott/m = 10

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 4.634737658271707e-7 4.638499215268375e-7 4.646020136207387e-7
Standard deviation 9.926680213284586e-10 1.6774945208728104e-9 3.122681595110072e-9

Outlying measurements have no (4.385879795843268e-3%) effect on estimated standard deviation.

quicksort/scott/m = 100

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 4.1225403183262075e-6 4.181471648035946e-6 4.2526273287477105e-6
Standard deviation 1.7886590322803427e-7 2.1845389276188672e-7 2.5864364126826737e-7

Outlying measurements have severe (0.6509319466118542%) effect on estimated standard deviation.

quicksort/scott/m = 1000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.919954064200528e-5 3.922223821924593e-5 3.925420433011818e-5
Standard deviation 6.857886201629363e-8 9.094800672159128e-8 1.3186949241593575e-7

Outlying measurements have no (7.298875432525951e-3%) effect on estimated standard deviation.

quicksort/scott/m = 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 4.415147358118668e-4 4.4197624484200795e-4 4.426884410908955e-4
Standard deviation 1.2497696303734096e-6 1.885786220238779e-6 2.770142692008369e-6

Outlying measurements have slight (1.1362135024441802e-2%) effect on estimated standard deviation.

quicksort/scott/m = 100000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 6.386822055809849e-3 6.443014348017542e-3 6.506640033431784e-3
Standard deviation 1.3817138299649474e-4 1.7707621980847223e-4 2.353978086588588e-4

Outlying measurements have moderate (0.10157390621540756%) effect on estimated standard deviation.

quicksort/builtin/m = 1

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.980327337498838e-8 6.00220662024098e-8 6.092879891971576e-8
Standard deviation 2.610571878947542e-10 1.2565911964795707e-9 2.805959321832343e-9

Outlying measurements have moderate (0.29934488796835296%) effect on estimated standard deviation.

quicksort/builtin/m = 10

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.7860729726267537e-7 2.8118756219140503e-7 2.8988061100510077e-7
Standard deviation 1.8651754774889862e-9 1.3735665326588482e-8 2.771903114231017e-8

Outlying measurements have severe (0.6787532548199441%) effect on estimated standard deviation.

quicksort/builtin/m = 100

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.525125394111828e-6 2.531189291710657e-6 2.5474665514780765e-6
Standard deviation 1.4288872077902284e-8 3.0624516204065374e-8 6.198361624468145e-8

Outlying measurements have slight (9.35141302436831e-2%) effect on estimated standard deviation.

quicksort/builtin/m = 1000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.5634814025232626e-5 2.5664262965940344e-5 2.5728433055173517e-5
Standard deviation 6.638034724567904e-8 1.3885240465124802e-7 2.3749804458671368e-7

Outlying measurements have no (6.848989298454407e-3%) effect on estimated standard deviation.

quicksort/builtin/m = 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.826631454077876e-4 2.831906793312553e-4 2.841837462129455e-4
Standard deviation 1.3374622980149708e-6 2.3386261875074165e-6 3.525551543754914e-6

Outlying measurements have slight (1.0308159722222114e-2%) effect on estimated standard deviation.

quicksort/builtin/m = 100000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 4.705796921380206e-3 4.72833452537e-3 4.758143629847545e-3
Standard deviation 5.9581147978449147e-5 7.886179614585633e-5 1.1147627586812446e-4

Outlying measurements have slight (2.3242630385487607e-2%) effect on estimated standard deviation.

understanding this report

In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.

Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.

We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)

A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.