criterion performance measurements

overview

want to understand this report?

sort . nub/100(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 9.173037631339266e-6 9.263500695171424e-6 9.393906264027345e-6
Standard deviation 1.7267844934161444e-7 2.964032454657826e-7 4.6350227354634946e-7

Outlying measurements have moderate (0.35333688155601706%) effect on estimated standard deviation.

sort . nub/200(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.626138077059697e-5 1.6377406540075786e-5 1.651494254430343e-5
Standard deviation 2.0583096783959642e-7 2.7091865608710256e-7 3.6718922009510705e-7

Outlying measurements have moderate (0.1192189152503338%) effect on estimated standard deviation.

sort . nub/400(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.300880638253835e-5 3.322857196315732e-5 3.374055754674127e-5
Standard deviation 3.7477449612129175e-7 8.538351316513155e-7 1.5514799948844173e-6

Outlying measurements have moderate (0.2284354006469801%) effect on estimated standard deviation.

sort . nub/600(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 4.849760221781421e-5 4.902799413332862e-5 5.015436389552221e-5
Standard deviation 7.318818724282779e-7 1.7415115525807207e-6 2.981387801265773e-6

Outlying measurements have moderate (0.34840446280503207%) effect on estimated standard deviation.

sort . nub/800(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 6.437414999781205e-5 6.472684477308253e-5 6.524848680989611e-5
Standard deviation 6.042081630208192e-7 1.0821678306774127e-6 1.688473756933986e-6

Outlying measurements have slight (9.728253553567343e-2%) effect on estimated standard deviation.

sort . nub/1000(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 7.821479992817453e-5 7.870138286838307e-5 7.931782390982856e-5
Standard deviation 1.0923813315867377e-6 1.3744176704707712e-6 1.8042558711862426e-6

Outlying measurements have moderate (0.10112300932130171%) effect on estimated standard deviation.

sort . nub/1250(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 9.815050989904133e-5 9.863713958217255e-5 9.93987303809119e-5
Standard deviation 1.1839084404007441e-6 1.405475693983285e-6 1.7766739206702041e-6

Outlying measurements have slight (6.682069500324749e-2%) effect on estimated standard deviation.

sort . nub/1500(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.1791351737241545e-4 1.1860788810480439e-4 1.193115734881251e-4
Standard deviation 1.3701964319503003e-6 1.6285801252641432e-6 1.8956111574815516e-6

Outlying measurements have slight (6.894774723969316e-2%) effect on estimated standard deviation.

sort . nub/2000(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.603854631658863e-4 1.615096477188385e-4 1.6296068802038345e-4
Standard deviation 2.706992699221691e-6 3.820293453991605e-6 4.762213660607595e-6

Outlying measurements have moderate (0.15962607733174458%) effect on estimated standard deviation.

sort . nub/3000(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.3568644899271872e-4 2.3681390324191672e-4 2.3824385155384452e-4
Standard deviation 2.2472180181999038e-6 2.8782699252596115e-6 3.893140016043041e-6

Outlying measurements have slight (1.2193263222069817e-2%) effect on estimated standard deviation.

sort . nub/4000(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.145291709130323e-4 3.165342417608908e-4 3.189269588465818e-4
Standard deviation 4.812035186829266e-6 5.9115748747703235e-6 6.973815226105643e-6

Outlying measurements have slight (8.813017164482563e-2%) effect on estimated standard deviation.

sort . nub/5000(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.8009149966559144e-4 3.8250489289561154e-4 3.8440269254658507e-4
Standard deviation 4.173583456744971e-6 5.245938239637845e-6 7.128609853760524e-6

Outlying measurements have slight (1.388613370363018e-2%) effect on estimated standard deviation.

nub . sort/100(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.51632580871722e-5 2.5259311556676857e-5 2.534620171427621e-5
Standard deviation 1.5597080512400342e-7 2.0465063989035898e-7 2.8280310586885845e-7

Outlying measurements have no (7.873519778281516e-3%) effect on estimated standard deviation.

nub . sort/200(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.47315385134815e-5 5.4821898684632735e-5 5.489856019068529e-5
Standard deviation 1.6348971175349367e-7 2.2159050231492887e-7 2.743148412846618e-7

Outlying measurements have no (8.927846765684548e-3%) effect on estimated standard deviation.

nub . sort/400(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.2073701684762699e-4 1.2095540701388764e-4 1.2132268968755539e-4
Standard deviation 4.3721492043970104e-7 7.132515904655487e-7 1.0172141334765108e-6

Outlying measurements have slight (1.0525124490719627e-2%) effect on estimated standard deviation.

nub . sort/600(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.849510779285588e-4 1.8560549590051798e-4 1.8659333015147436e-4
Standard deviation 1.6822389358112764e-6 2.2093921548457633e-6 2.7870727867433147e-6

Outlying measurements have slight (1.149269875608429e-2%) effect on estimated standard deviation.

nub . sort/800(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.5914789372450444e-4 2.5978504627836195e-4 2.6088024055447105e-4
Standard deviation 1.690784052141868e-6 2.2062256929367453e-6 2.748014032565964e-6

Outlying measurements have slight (1.2497997115846619e-2%) effect on estimated standard deviation.

nub . sort/1000(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.262853034359749e-4 3.2781998130246656e-4 3.299091525349773e-4
Standard deviation 3.848017965464503e-6 4.672618217654571e-6 5.955007989692829e-6

Outlying measurements have slight (1.3155555555555969e-2%) effect on estimated standard deviation.

nub . sort/1250(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 4.026974404565907e-4 4.0450338102160604e-4 4.070680802293748e-4
Standard deviation 4.035940422874438e-6 5.6976664418757645e-6 8.373705016920455e-6

Outlying measurements have slight (1.4081632653061362e-2%) effect on estimated standard deviation.

nub . sort/1500(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.020732346487404e-4 5.032080680090447e-4 5.047853124689998e-4
Standard deviation 2.5792447857217724e-6 3.373650000384108e-6 4.387168211758303e-6

Outlying measurements have slight (1.492194674012856e-2%) effect on estimated standard deviation.

nub . sort/2000(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 6.756059118389956e-4 6.795444256135095e-4 6.850317792731881e-4
Standard deviation 7.571691484839963e-6 1.3595376578526537e-5 2.0808952913063772e-5

Outlying measurements have slight (7.881519515524552e-2%) effect on estimated standard deviation.

nub . sort/3000(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.0922605913624816e-3 1.095390295530309e-3 1.1010980070242848e-3
Standard deviation 8.879066467986537e-6 1.248669918563487e-5 1.8791804391352596e-5

Outlying measurements have slight (1.886094674556218e-2%) effect on estimated standard deviation.

nub . sort/4000(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.57559137473375e-3 1.582135580437252e-3 1.5912356033219222e-3
Standard deviation 1.8449417858359638e-5 2.4189196433254277e-5 3.106690660221411e-5

Outlying measurements have slight (2.172839506172859e-2%) effect on estimated standard deviation.

nub . sort/5000(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.103042498484733e-3 2.117745432682725e-3 2.15138977885411e-3
Standard deviation 1.8963415984758468e-5 6.106839540208284e-5 1.0959058071007448e-4

Outlying measurements have moderate (0.11766002551042844%) effect on estimated standard deviation.

understanding this report

In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.

Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.

We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)

A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.