criterion performance measurements

overview

want to understand this report?

sort . nub/100(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 8.412708403539346e-5 8.428280861474216e-5 8.452334796567459e-5
Standard deviation 5.438169935156804e-7 6.492923594043273e-7 7.89173338721058e-7

Outlying measurements have no (9.707804690503367e-3%) effect on estimated standard deviation.

sort . nub/200(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.7994599638226123e-4 2.80881900232012e-4 2.8208918995837945e-4
Standard deviation 2.1054445165783925e-6 2.7545376029424036e-6 3.7369693558939195e-6

Outlying measurements have slight (1.2656147271532333e-2%) effect on estimated standard deviation.

sort . nub/400(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 9.227254912630544e-4 9.26958402567981e-4 9.371512337375196e-4
Standard deviation 1.057212860119539e-5 1.8122655392299185e-5 2.9527020449792474e-5

Outlying measurements have slight (8.476653960219219e-2%) effect on estimated standard deviation.

sort . nub/600(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.8626159503263137e-3 1.8671998774736898e-3 1.8778925804745984e-3
Standard deviation 1.3872146312489998e-5 2.0190195423272e-5 2.8274258381439847e-5

Outlying measurements have slight (2.3242630385487392e-2%) effect on estimated standard deviation.

sort . nub/800(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.79564513085102e-3 2.8085986001559027e-3 2.8252056247624457e-3
Standard deviation 2.777798187654579e-5 3.569496263915663e-5 4.739987171506165e-5

Outlying measurements have slight (2.7755102040816326e-2%) effect on estimated standard deviation.

sort . nub/1000(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 4.141632403002326e-3 4.157204368671638e-3 4.179978311615923e-3
Standard deviation 3.271282195585513e-5 4.740820789766323e-5 6.403663711314183e-5

Outlying measurements have slight (3.2222222222222034e-2%) effect on estimated standard deviation.

sort . nub/1250(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.789782848648188e-3 5.810931796971934e-3 5.834506492311298e-3
Standard deviation 4.3087738299234705e-5 5.33483311688547e-5 6.978538835027976e-5

Outlying measurements have slight (3.839999999999999e-2%) effect on estimated standard deviation.

sort . nub/1500(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 7.605045873378376e-3 7.63021691624117e-3 7.664515791920662e-3
Standard deviation 5.714815097891909e-5 7.164130229463679e-5 9.931315895344051e-5

Outlying measurements have slight (4.338842975206612e-2%) effect on estimated standard deviation.

sort . nub/2000(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.1315661369180037e-2 1.1366227299099049e-2 1.151224197676315e-2
Standard deviation 1.519884788253717e-4 1.9011722155764564e-4 2.3647680090724938e-4

Outlying measurements have slight (5.2469135802469126e-2%) effect on estimated standard deviation.

sort . nub/3000(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.7375851106096452e-2 1.791241330362621e-2 1.8297633931193043e-2
Standard deviation 7.084108723293579e-4 9.683982564262129e-4 1.182731214779378e-3

Outlying measurements have moderate (0.13506429647037746%) effect on estimated standard deviation.

sort . nub/4000(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.8115921423625907e-2 2.8371922946974194e-2 2.8542889150469365e-2
Standard deviation 2.4429820242778905e-4 3.732883064547752e-4 5.080929940065159e-4

Outlying measurements have slight (8.264462809917351e-2%) effect on estimated standard deviation.

sort . nub/5000(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.663095960338613e-2 3.689112313856714e-2 3.745232803206252e-2
Standard deviation 3.248852539954043e-4 7.089823541368252e-4 1.070605751893903e-3

Outlying measurements have slight (9.87654320987654e-2%) effect on estimated standard deviation.

nub . sort/100(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 8.427698330561578e-5 8.495931191498575e-5 8.55249579435448e-5
Standard deviation 1.1064420396365576e-6 1.3340559082419088e-6 1.7521992582666873e-6

Outlying measurements have slight (8.339826790112466e-2%) effect on estimated standard deviation.

nub . sort/200(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.753253071650909e-4 2.766076716980822e-4 2.78072350586011e-4
Standard deviation 2.8156882253911094e-6 3.7746242632801048e-6 5.149665835910217e-6

Outlying measurements have slight (1.265614727153205e-2%) effect on estimated standard deviation.

nub . sort/400(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 8.729867651943291e-4 8.760550918410735e-4 8.792004564629108e-4
Standard deviation 4.907822881213445e-6 7.742248277634635e-6 1.1056010799347357e-5

Outlying measurements have slight (1.7538265306122448e-2%) effect on estimated standard deviation.

nub . sort/600(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.5968345157558444e-3 1.6052869524335257e-3 1.6145915819292807e-3
Standard deviation 1.9491849759931553e-5 2.3772192074613517e-5 3.0023828272796626e-5

Outlying measurements have slight (2.1728395061728273e-2%) effect on estimated standard deviation.

nub . sort/800(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.307165371369687e-3 2.3147433215015734e-3 2.3320674266703e-3
Standard deviation 2.38848431375833e-5 2.7882264945555643e-5 3.772246728307259e-5

Outlying measurements have slight (2.498356344510191e-2%) effect on estimated standard deviation.

nub . sort/1000(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.012282332844349e-3 3.024704774962962e-3 3.0426091013997392e-3
Standard deviation 2.380509069195574e-5 3.1802312846490546e-5 4.1050731827309136e-5

Outlying measurements have slight (2.8546712802768166e-2%) effect on estimated standard deviation.

nub . sort/1250(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 4.327755158968874e-3 4.430139805295954e-3 4.558208817332696e-3
Standard deviation 2.3752777635830614e-4 2.955268218339374e-4 3.7281178807016204e-4

Outlying measurements have moderate (0.3304307491512877%) effect on estimated standard deviation.

nub . sort/1500(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 4.691393875722979e-3 4.719770551854024e-3 4.74410882366315e-3
Standard deviation 4.8022719822338415e-5 5.847045950151639e-5 7.42093680812756e-5

Outlying measurements have slight (3.443877551020404e-2%) effect on estimated standard deviation.

nub . sort/2000(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.947432964131459e-3 5.965707637338252e-3 5.98687432147252e-3
Standard deviation 4.0852076390746066e-5 4.871205423212286e-5 6.0015921412325386e-5

Outlying measurements have slight (3.84e-2%) effect on estimated standard deviation.

nub . sort/3000(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 8.206151204436386e-3 8.228719127079205e-3 8.253530545751417e-3
Standard deviation 3.534361749969394e-5 5.5157224548335045e-5 7.367562412305112e-5

Outlying measurements have slight (4.535147392290249e-2%) effect on estimated standard deviation.

nub . sort/4000(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 9.678917193516831e-3 9.702687732491328e-3 9.781452464452523e-3
Standard deviation 5.4148338964026366e-5 8.467956471702421e-5 1.2475575488861884e-4

Outlying measurements have slight (4.986149584487534e-2%) effect on estimated standard deviation.

nub . sort/5000(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.0650843372757063e-2 1.0679375863811014e-2 1.0742839891806879e-2
Standard deviation 5.6812200375506224e-5 8.703541035268541e-5 1.4022611976622943e-4

Outlying measurements have slight (5.24691358024691e-2%) effect on estimated standard deviation.

understanding this report

In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.

Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.

We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)

A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.