criterion performance measurements

overview

want to understand this report?

100% dups/sort . nub 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 6.495756522820152e-3 6.530205790166014e-3 6.576393076706263e-3
Standard deviation 8.830836133527619e-5 1.1517586102164174e-4 1.5668837967432948e-4

Outlying measurements have slight (2.62965668371073e-2%) effect on estimated standard deviation.

100% dups/nub . sort 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 6.552997148485894e-3 6.5592695171571135e-3 6.56588198401969e-3
Standard deviation 1.5310387088452472e-5 1.8833509463995673e-5 2.4650562470385175e-5

Outlying measurements have slight (2.629656683710737e-2%) effect on estimated standard deviation.

99% dups/sort . nub 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.11604083524372252 0.11882082062930598 0.12392330559374187
Standard deviation 2.451035690575174e-3 5.831446117118993e-3 9.354306525327058e-3

Outlying measurements have moderate (0.11232982463627844%) effect on estimated standard deviation.

99% dups/nub . sort 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.5253607045761642e-2 2.5341466483710686e-2 2.544714857283164e-2
Standard deviation 1.7255450094617221e-4 2.1081229557355397e-4 2.545460795803395e-4

Outlying measurements have slight (4.986149584487535e-2%) effect on estimated standard deviation.

95% dups/sort . nub 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.15128337655238377 0.15714901469520895 0.17003410302372354
Standard deviation 3.7485191194458357e-3 1.2306456621344318e-2 1.904786393713292e-2

Outlying measurements have moderate (0.1354289498357612%) effect on estimated standard deviation.

95% dups/nub . sort 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 4.751603920332395e-2 4.764844994812058e-2 4.7816558919761824e-2
Standard deviation 2.0218437233974354e-4 2.9063564289127084e-4 4.1717211980407665e-4

Outlying measurements have slight (7.100591715976329e-2%) effect on estimated standard deviation.

90% dups/sort . nub 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.174687781702366 0.18427782701528275 0.20586213962238634
Standard deviation 8.686270005902302e-3 2.0245306035942393e-2 2.9723613457278655e-2

Outlying measurements have moderate (0.2706175389567998%) effect on estimated standard deviation.

90% dups/nub . sort 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 6.88667974159757e-2 6.910618883273953e-2 6.942319426250644e-2
Standard deviation 3.2855869580750244e-4 4.992217293341379e-4 7.649255209697747e-4

Outlying measurements have slight (8.264462809917347e-2%) effect on estimated standard deviation.

80% dups/sort . nub 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.21594766216110328 0.23007977141387084 0.26144257855552017
Standard deviation 8.84621558153241e-3 2.856425384352476e-2 4.068267518815685e-2

Outlying measurements have moderate (0.31456017370952705%) effect on estimated standard deviation.

80% dups/nub . sort 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.10654584860608031 0.107067904249589 0.1078815958190425
Standard deviation 6.172880022874042e-4 1.0072856391568023e-3 1.5192792493753061e-3

Outlying measurements have slight (9.87654320987653e-2%) effect on estimated standard deviation.

75% dups/sort . nub 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.2528648201333256 0.2638832522332958 0.27692488229992024
Standard deviation 7.209585582656658e-3 1.550865701496188e-2 2.2291630009865324e-2

Outlying measurements have moderate (0.16381027473811283%) effect on estimated standard deviation.

75% dups/nub . sort 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.12362367263540837 0.12385751342754002 0.12457172231850948
Standard deviation 1.9100551019964767e-4 5.658876070015247e-4 8.87641428882017e-4

Outlying measurements have moderate (0.109375%) effect on estimated standard deviation.

70% dups/sort . nub 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.24403939532000551 0.25804153828666965 0.28292802179998944
Standard deviation 6.752229949424971e-3 2.3263191258895265e-2 3.18396300846753e-2

Outlying measurements have moderate (0.18462020045919433%) effect on estimated standard deviation.

70% dups/nub . sort 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.14271379806801993 0.14452118090135985 0.14710585175511134
Standard deviation 2.507047574628366e-3 3.1495605425568346e-3 3.809846377023347e-3

Outlying measurements have moderate (0.12244897959183673%) effect on estimated standard deviation.

60% dups/sort . nub 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.28689641539999683 0.2993704454166618 0.3198520100666428
Standard deviation 6.1498908858324715e-3 2.1785893497781428e-2 3.085702381324064e-2

Outlying measurements have moderate (0.17639646188390917%) effect on estimated standard deviation.

60% dups/nub . sort 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.18886427879178946 0.19728303189169574 0.20078499218332505
Standard deviation 1.5689108033020982e-3 7.2687201894642995e-3 1.0531555987435402e-2

Outlying measurements have moderate (0.1388888888888887%) effect on estimated standard deviation.

50% dups/sort . nub 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.29674487196663446 0.30668664618331376 0.3252277476000017
Standard deviation 9.649353772448015e-4 1.8289056095104676e-2 2.3550227434062522e-2

Outlying measurements have moderate (0.16485066105880508%) effect on estimated standard deviation.

50% dups/nub . sort 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.22241111389996981 0.22643166379720203 0.23049438272221323
Standard deviation 4.4866079738961825e-3 5.772930705382345e-3 7.189631905884572e-3

Outlying measurements have moderate (0.13888888888888887%) effect on estimated standard deviation.

40% dups/sort . nub 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.33451752862504236 0.3427336323958154 0.35498783658321525
Standard deviation 1.254030621143893e-3 1.1587728763433145e-2 1.4776951483171848e-2

Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.

40% dups/nub . sort 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.256916516173369 0.25795234677334067 0.25874961239999267
Standard deviation 8.241215711146913e-4 1.1076639746704657e-3 1.3751044752224719e-3

Outlying measurements have moderate (0.16%) effect on estimated standard deviation.

30% dups/sort . nub 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.3768826481249903 0.3803497922083352 0.382686336166671
Standard deviation 2.070929166620772e-3 3.6806384882983024e-3 5.199164651497965e-3

Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.

30% dups/nub . sort 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.2933924408000015 0.2968932218399944 0.3004765207199943
Standard deviation 3.578272936862454e-3 4.802895132884522e-3 6.13022325626411e-3

Outlying measurements have moderate (0.16%) effect on estimated standard deviation.

25% dups/sort . nub 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.36470523254164294 0.374632454666686 0.3905446133751127
Standard deviation 9.954959166880994e-4 1.5028594314976829e-2 1.8948528062873456e-2

Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.

25% dups/nub . sort 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.3145459763832643 0.3163054921766396 0.3193821917799733
Standard deviation 1.019175115194216e-3 3.0329624608057956e-3 4.397653350273793e-3

Outlying measurements have moderate (0.16%) effect on estimated standard deviation.

20% dups/sort . nub 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.37441342620836815 0.3844799637291724 0.40105057762499996
Standard deviation 4.203053458345124e-3 1.6115169906184067e-2 2.156073083238072e-2

Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.

20% dups/nub . sort 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.33547367393748573 0.34664766718748297 0.3549086057499835
Standard deviation 5.911147903590027e-3 1.144948367015708e-2 1.585818997341295e-2

Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.

10% dups/sort . nub 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.40919100206248515 0.4160215574375125 0.4228733930000696
Standard deviation 3.666455750078512e-3 8.496739036642347e-3 1.1589977997382175e-2

Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.

10% dups/nub . sort 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.36987991837500545 0.37115815254168183 0.3733964901250033
Standard deviation 2.9322824997279895e-4 2.145636737891348e-3 2.7070581953145345e-3

Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.

5% dups/sort . nub 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.41072165041666153 0.4204379942083136 0.4350247464999484
Standard deviation 3.7387552499978938e-3 1.3870870671078266e-2 1.812179211455632e-2

Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.

5% dups/nub . sort 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.3875979013750168 0.3878699976458364 0.3882539329374737
Standard deviation 8.392320833650047e-5 4.013225221454599e-4 5.314732469640595e-4

Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.

1% dups/sort . nub 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.4277861582708387 0.4386208597708266 0.4471792392499765
Standard deviation 4.552644041676018e-3 1.0661114530305322e-2 1.3554987329055563e-2

Outlying measurements have moderate (0.18749999999999994%) effect on estimated standard deviation.

1% dups/nub . sort 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.40562339424998584 0.4064671198333514 0.4073284802500581
Standard deviation 2.9858066667998173e-4 9.787815177875047e-4 1.3125758148442284e-3

Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.

0% dups/sort . nub 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.4293955011249911 0.444758853125028 0.45883364250005343
Standard deviation 8.361886499983484e-3 1.775316145170711e-2 2.2567808247451134e-2

Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.

0% dups/nub . sort 10000

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.4096028806875296 0.4167483400208359 0.4237708096666589
Standard deviation 5.3551187916790755e-3 8.494346015959863e-3 1.0034140409730982e-2

Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.

understanding this report

In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.

Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.

We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)

A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.