criterion performance measurements

overview

want to understand this report?

sort . nub/1000(1->536870911)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 4.449127422239928e-3 4.467718588530757e-3 4.4822556672707925e-3
Standard deviation 3.56005662342857e-5 4.5177301915570863e-5 5.282499624756689e-5

Outlying measurements have slight (3.329369797859691e-2%) effect on estimated standard deviation.

sort . nub/1000(1->100000)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 4.3645285547580384e-3 4.376037876267016e-3 4.390950427259955e-3
Standard deviation 2.361689658244095e-5 3.561800964698775e-5 4.994363526778487e-5

Outlying measurements have slight (3.329369797859691e-2%) effect on estimated standard deviation.

sort . nub/1000(1->8000)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 4.086077134228165e-3 4.092736261052948e-3 4.104667363093675e-3
Standard deviation 1.9571057737632654e-5 2.5343915979275506e-5 3.1131945951610504e-5

Outlying measurements have slight (3.222222222222222e-2%) effect on estimated standard deviation.

sort . nub/1000(1->5000)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.921240317674046e-3 3.940286288556364e-3 3.960616295651621e-3
Standard deviation 3.590058528350664e-5 4.631608669192931e-5 5.656542134628966e-5

Outlying measurements have slight (3.222222222222222e-2%) effect on estimated standard deviation.

sort . nub/1000(1->2000)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.267946590972865e-3 3.2771502972043843e-3 3.298839796799492e-3
Standard deviation 2.079801461826313e-5 2.6488968477933393e-5 3.2370136513007594e-5

Outlying measurements have slight (2.9384756657483812e-2%) effect on estimated standard deviation.

sort . nub/1000(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.926539427193961e-3 2.933046151912885e-3 2.9441473140883693e-3
Standard deviation 1.8326185492795542e-5 2.4553549702156607e-5 3.130913718262209e-5

Outlying measurements have slight (2.7755102040816323e-2%) effect on estimated standard deviation.

sort . nub/1000(1->500)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.690933635869525e-3 1.6934282770664038e-3 1.6988756351646855e-3
Standard deviation 6.982194540448961e-6 9.7113793068901e-6 1.3741010059837093e-5

Outlying measurements have slight (2.221074380165289e-2%) effect on estimated standard deviation.

sort . nub/1000(1->100)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.792513322906108e-4 3.803298194830379e-4 3.828698608649089e-4
Standard deviation 3.0862993646402017e-6 3.844690677708289e-6 5.092748822473012e-6

Outlying measurements have slight (1.3695987654320781e-2%) effect on estimated standard deviation.

sort . nub/1000(1->50)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.8897471153132278e-4 1.896450748620939e-4 1.9038773593283963e-4
Standard deviation 1.2401682363228752e-6 1.6853324236268766e-6 2.1846570522630237e-6

Outlying measurements have slight (1.1626297577854586e-2%) effect on estimated standard deviation.

sort . nub/1000(1->20)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 8.610048884092083e-5 8.630791133317868e-5 8.659910935733947e-5
Standard deviation 4.783447497176129e-7 6.190963975146361e-7 8.494447522774935e-7

Outlying measurements have no (9.802960494069032e-3%) effect on estimated standard deviation.

sort . nub/1000(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.275609739656748e-5 5.292896950281136e-5 5.308739102784644e-5
Standard deviation 3.555855397333798e-7 4.262519668970709e-7 5.445592214689172e-7

Outlying measurements have no (8.927846765684614e-3%) effect on estimated standard deviation.

nub . sort/1000(1->536870911)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 4.819456360913092e-3 4.83505984917049e-3 4.8472120821764e-3
Standard deviation 2.7037236506337205e-5 3.354947469667422e-5 4.401561498114546e-5

Outlying measurements have slight (3.443877551020408e-2%) effect on estimated standard deviation.

nub . sort/1000(1->100000)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 4.793870997908157e-3 4.806648739335707e-3 4.815766087723638e-3
Standard deviation 2.351700243489309e-5 2.9693617131924283e-5 3.604648826453838e-5

Outlying measurements have slight (3.4438775510204085e-2%) effect on estimated standard deviation.

nub . sort/1000(1->8000)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 4.1662863369722825e-3 4.17582589622093e-3 4.185024618141699e-3
Standard deviation 1.733093233510453e-5 2.3192524533281533e-5 3.3203906058439655e-5

Outlying measurements have slight (3.222222222222222e-2%) effect on estimated standard deviation.

nub . sort/1000(1->5000)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 4.107918965543092e-3 4.117628823379225e-3 4.135429855904125e-3
Standard deviation 1.892529026737733e-5 2.6131519277870328e-5 4.01268132196734e-5

Outlying measurements have slight (3.222222222222201e-2%) effect on estimated standard deviation.

nub . sort/1000(1->2000)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.06373555294145e-3 3.07157258139758e-3 3.0791125777757345e-3
Standard deviation 1.8010045649659483e-5 2.2561776104948963e-5 2.9304153874426543e-5

Outlying measurements have slight (2.854671280276817e-2%) effect on estimated standard deviation.

nub . sort/1000(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.7111093215607084e-3 2.7193081642987805e-3 2.725846069917118e-3
Standard deviation 1.4240403657923868e-5 1.869944443803834e-5 2.5637115752487966e-5

Outlying measurements have slight (2.700617283950604e-2%) effect on estimated standard deviation.

nub . sort/1000(1->500)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.2513713315758694e-3 1.2555565244731191e-3 1.259688578503607e-3
Standard deviation 8.056698423433784e-6 9.318608738919692e-6 1.1187710469498835e-5

Outlying measurements have slight (1.9991670137442616e-2%) effect on estimated standard deviation.

nub . sort/1000(1->100)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.6827139100511683e-4 3.6940243256248004e-4 3.7154320192838374e-4
Standard deviation 3.5289510189415686e-6 4.49713339370202e-6 6.143204538482736e-6

Outlying measurements have slight (1.3695987654320988e-2%) effect on estimated standard deviation.

nub . sort/1000(1->50)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.1575544718031294e-4 3.1724960595480643e-4 3.200966627109548e-4
Standard deviation 3.2413869211082227e-6 4.810891602034594e-6 6.8566750309795896e-6

Outlying measurements have slight (6.26401705550201e-2%) effect on estimated standard deviation.

nub . sort/1000(1->20)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.88224359215638e-4 2.893245080669463e-4 2.9090175882909693e-4
Standard deviation 3.1016992217654365e-6 4.046984957968173e-6 5.398285041985691e-6

Outlying measurements have slight (1.2818350480688337e-2%) effect on estimated standard deviation.

nub . sort/1000(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.683657302019353e-4 2.696549625903623e-4 2.7111621469641875e-4
Standard deviation 2.705135951906428e-6 3.514121864395073e-6 4.795253320566446e-6

Outlying measurements have slight (1.2656147271532109e-2%) effect on estimated standard deviation.

understanding this report

In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.

Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.

We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)

A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.