criterion performance measurements

overview

want to understand this report?

sort . nub/1000(1->536870911)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.8030607079905625e-3 5.847673332757955e-3 5.913985054954675e-3
Standard deviation 8.11496599366019e-5 1.1275551075131968e-4 1.5265216591340346e-4

Outlying measurements have slight (3.84e-2%) effect on estimated standard deviation.

sort . nub/1000(1->100000)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.882844672959634e-3 5.915395829097553e-3 5.955056738708926e-3
Standard deviation 6.88633605930516e-5 8.703878064994451e-5 1.071252045503089e-4

Outlying measurements have slight (3.8400000000000004e-2%) effect on estimated standard deviation.

sort . nub/1000(1->8000)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.606319150164333e-3 5.633493880724761e-3 5.680448988654709e-3
Standard deviation 5.551491611618823e-5 7.785103024631084e-5 1.1625549671053169e-4

Outlying measurements have slight (3.698224852071005e-2%) effect on estimated standard deviation.

sort . nub/1000(1->5000)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.012626655873147e-3 5.041671910100225e-3 5.091472289904934e-3
Standard deviation 6.844983824138137e-5 8.301964824013525e-5 1.0486671227294366e-4

Outlying measurements have slight (3.5665294924554176e-2%) effect on estimated standard deviation.

sort . nub/1000(1->2000)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 4.586916227884006e-3 4.600228755482725e-3 4.612805786741889e-3
Standard deviation 2.598188973102432e-5 3.464893228406241e-5 4.474398181136793e-5

Outlying measurements have slight (3.4438775510204044e-2%) effect on estimated standard deviation.

sort . nub/1000(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 4.21677278146656e-3 4.2412959379400485e-3 4.281532405426029e-3
Standard deviation 7.668916672386469e-5 9.29462911175188e-5 1.1453608695880062e-4

Outlying measurements have slight (3.329369797859706e-2%) effect on estimated standard deviation.

sort . nub/1000(1->500)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.4480666260661456e-3 2.4622613810271333e-3 2.4795101353515822e-3
Standard deviation 3.719457091291283e-5 4.730607193932097e-5 6.160377566837977e-5

Outlying measurements have slight (2.562326869806057e-2%) effect on estimated standard deviation.

sort . nub/1000(1->100)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.994953752808882e-4 6.026314720454706e-4 6.054824526740607e-4
Standard deviation 6.1523122492115535e-6 7.76603645129974e-6 1.0447535945741747e-5

Outlying measurements have slight (1.562106324011115e-2%) effect on estimated standard deviation.

sort . nub/1000(1->50)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.1468710202542887e-4 3.1668549044166214e-4 3.1893104864750036e-4
Standard deviation 4.46156511511019e-6 5.645504742846971e-6 6.783715899783288e-6

Outlying measurements have slight (8.762271553914013e-2%) effect on estimated standard deviation.

sort . nub/1000(1->20)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.388773539614713e-4 1.4028018793387832e-4 1.4225727007920414e-4
Standard deviation 2.315146256488421e-6 3.770911077473782e-6 5.773897292977209e-6

Outlying measurements have moderate (0.19649721325343658%) effect on estimated standard deviation.

sort . nub/1000(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 7.796389922367095e-5 7.835348641473416e-5 7.897151647392277e-5
Standard deviation 1.0264561132625791e-6 1.1796451647784858e-6 1.4903472955480065e-6

Outlying measurements have slight (8.226880152433973e-2%) effect on estimated standard deviation.

nub . sort/1000(1->536870911)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.788381547983991e-3 5.819456519252726e-3 5.85824366300011e-3
Standard deviation 6.65378187078395e-5 1.0123525586767025e-4 1.425197885115305e-4

Outlying measurements have slight (3.8400000000000004e-2%) effect on estimated standard deviation.

nub . sort/1000(1->100000)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.796676555573845e-3 5.827882644661667e-3 5.865854878162755e-3
Standard deviation 5.646583310637199e-5 8.028511031770332e-5 1.0219662332103066e-4

Outlying measurements have slight (3.84e-2%) effect on estimated standard deviation.

nub . sort/1000(1->8000)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.051071799294266e-3 5.079119049849748e-3 5.1038459494403806e-3
Standard deviation 4.838723480257865e-5 5.7062544755513116e-5 7.531451243809564e-5

Outlying measurements have slight (3.566529492455405e-2%) effect on estimated standard deviation.

nub . sort/1000(1->5000)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 4.829818744422628e-3 4.854358870091492e-3 4.874841348497941e-3
Standard deviation 4.541473717719111e-5 5.551500575935908e-5 7.284690797411011e-5

Outlying measurements have slight (3.566529492455418e-2%) effect on estimated standard deviation.

nub . sort/1000(1->2000)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.6349545376499442e-3 3.6461283902815907e-3 3.6654172687313907e-3
Standard deviation 2.4256930987202233e-5 3.640606442237857e-5 5.388224281793376e-5

Outlying measurements have slight (3.02734375e-2%) effect on estimated standard deviation.

nub . sort/1000(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.150477999935822e-3 3.1579529906010117e-3 3.1715073205344523e-3
Standard deviation 1.715128667759864e-5 2.7468511539001738e-5 3.856053626331965e-5

Outlying measurements have slight (2.8546712802768166e-2%) effect on estimated standard deviation.

nub . sort/1000(1->500)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.4891426626195635e-3 1.494063269928757e-3 1.5031624848052282e-3
Standard deviation 1.0796812249969468e-5 1.3652003472657426e-5 2.0749913219070965e-5

Outlying measurements have slight (2.126654064272212e-2%) effect on estimated standard deviation.

nub . sort/1000(1->100)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 4.3150077727662953e-4 4.3473340558199906e-4 4.4073820061236996e-4
Standard deviation 7.94163280396863e-6 1.0028422655039476e-5 1.2758310497716221e-5

Outlying measurements have moderate (0.12319478303389135%) effect on estimated standard deviation.

nub . sort/1000(1->50)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.750344123194462e-4 3.760862091565599e-4 3.7724331680023723e-4
Standard deviation 2.3637568179607485e-6 2.9491211981389184e-6 3.751624153639552e-6

Outlying measurements have slight (1.3695987654320873e-2%) effect on estimated standard deviation.

nub . sort/1000(1->20)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.422960220302701e-4 3.4314966533110315e-4 3.451879937302772e-4
Standard deviation 2.0258858566281917e-6 2.7609850422231975e-6 3.4463661747766217e-6

Outlying measurements have slight (1.3330898466033555e-2%) effect on estimated standard deviation.

nub . sort/1000(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.2648401813008645e-4 3.275173421509213e-4 3.300686584576381e-4
Standard deviation 3.0066146874170707e-6 4.121351295649877e-6 5.920471461173899e-6

Outlying measurements have slight (1.3155555555555793e-2%) effect on estimated standard deviation.

understanding this report

In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.

Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.

We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)

A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.