criterion performance measurements
overview
want to understand this report?
sort . nub/100(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.189051739196837e-5 | 5.213009633842205e-5 | 5.2260715586492254e-5 |
| Standard deviation | 3.1335689348233645e-7 | 4.2016444360299073e-7 | 5.345166555679025e-7 |
Outlying measurements have no (8.848852040816587e-3%) effect on estimated standard deviation.
sort . nub/200(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.919642804108845e-4 | 1.9220668842413532e-4 | 1.9253028755024512e-4 |
| Standard deviation | 6.924737804310115e-7 | 8.733471333038741e-7 | 1.1281072764380537e-6 |
Outlying measurements have slight (1.1626297577854607e-2%) effect on estimated standard deviation.
sort . nub/400(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.217720806315301e-4 | 7.224701603192232e-4 | 7.233848393056666e-4 |
| Standard deviation | 1.7886620352966222e-6 | 2.3076747894340128e-6 | 2.824525598556395e-6 |
Outlying measurements have slight (1.6661878770468177e-2%) effect on estimated standard deviation.
sort . nub/600(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.564397167561532e-3 | 1.5663364594617547e-3 | 1.5689013433929015e-3 |
| Standard deviation | 6.002431294824394e-6 | 8.131329212748678e-6 | 1.0683114843636815e-5 |
Outlying measurements have slight (2.1728395061728394e-2%) effect on estimated standard deviation.
sort . nub/800(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.6156623869798704e-3 | 2.618303788960707e-3 | 2.6212629458387646e-3 |
| Standard deviation | 5.78358970128607e-6 | 7.701718537505484e-6 | 9.667011906827076e-6 |
Outlying measurements have slight (2.6296566837107235e-2%) effect on estimated standard deviation.
sort . nub/1000(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.962213567882971e-3 | 3.966456913977613e-3 | 3.973365451782813e-3 |
| Standard deviation | 1.2670536135211392e-5 | 1.4827319572280745e-5 | 1.836527542978984e-5 |
Outlying measurements have slight (3.222222222222222e-2%) effect on estimated standard deviation.
sort . nub/1250(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.0234306351573935e-3 | 6.030594330810306e-3 | 6.0518494412292335e-3 |
| Standard deviation | 1.5998177385833088e-5 | 3.005813584084085e-5 | 5.0533248947659435e-5 |
Outlying measurements have slight (3.839999999999986e-2%) effect on estimated standard deviation.
sort . nub/1500(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.527777864394591e-3 | 8.549353333302916e-3 | 8.578752900253415e-3 |
| Standard deviation | 3.7202138591553815e-5 | 5.923197474603375e-5 | 8.515884583376159e-5 |
Outlying measurements have slight (4.5351473922902494e-2%) effect on estimated standard deviation.
sort . nub/2000(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.4346239780284528e-2 | 1.4363435389400802e-2 | 1.4400387414714078e-2 |
| Standard deviation | 3.082559402024139e-5 | 5.443561683689289e-5 | 8.542640162422553e-5 |
Outlying measurements have slight (5.859375e-2%) effect on estimated standard deviation.
sort . nub/3000(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.0551387973334027e-2 | 3.059314023887596e-2 | 3.0652697084415953e-2 |
| Standard deviation | 3.458453120783851e-5 | 9.02372243517934e-5 | 1.4054705313870563e-4 |
Outlying measurements have slight (9.000000000000001e-2%) effect on estimated standard deviation.
sort . nub/4000(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.02617229112984e-2 | 5.044203476919684e-2 | 5.085220837501936e-2 |
| Standard deviation | 2.0052347811525353e-4 | 3.8402962962817846e-4 | 5.472983090623422e-4 |
Outlying measurements have moderate (0.109375%) effect on estimated standard deviation.
sort . nub/5000(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.538290112219986e-2 | 7.615232524721845e-2 | 7.811174965279595e-2 |
| Standard deviation | 3.226918876376404e-4 | 1.6771504793275604e-3 | 2.3401266738680565e-3 |
Outlying measurements have moderate (0.13888888888888884%) effect on estimated standard deviation.
nub . sort/100(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.612158382602718e-5 | 5.6250744677467936e-5 | 5.659077379037449e-5 |
| Standard deviation | 2.8958736814037615e-7 | 5.16495454025757e-7 | 9.59808959266229e-7 |
Outlying measurements have no (9.00826446280991e-3%) effect on estimated standard deviation.
nub . sort/200(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.025923148447266e-4 | 2.034189563678715e-4 | 2.0444798655587656e-4 |
| Standard deviation | 1.6629697099495956e-6 | 2.4030411908146757e-6 | 3.318255482541725e-6 |
Outlying measurements have slight (1.1763038548752863e-2%) effect on estimated standard deviation.
nub . sort/400(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.766258226246357e-4 | 7.774224557961502e-4 | 7.781893396738849e-4 |
| Standard deviation | 1.6434499011464476e-6 | 2.019840722511961e-6 | 2.5061913422512693e-6 |
Outlying measurements have slight (1.694411414982164e-2%) effect on estimated standard deviation.
nub . sort/600(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.6985949176842223e-3 | 1.7024868050875756e-3 | 1.711488708208363e-3 |
| Standard deviation | 7.268897628742281e-6 | 1.2349406237863693e-5 | 1.8085338883158206e-5 |
Outlying measurements have slight (2.221074380165284e-2%) effect on estimated standard deviation.
nub . sort/800(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.890175929876266e-3 | 2.896875746841001e-3 | 2.9041526449512306e-3 |
| Standard deviation | 1.3246683157753286e-5 | 1.9496754699373722e-5 | 2.551446636136738e-5 |
Outlying measurements have slight (2.7755102040816326e-2%) effect on estimated standard deviation.
nub . sort/1000(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.290233686381259e-3 | 4.297305676564661e-3 | 4.314890655129391e-3 |
| Standard deviation | 1.664431616699726e-5 | 2.7001143234478295e-5 | 3.845975587712351e-5 |
Outlying measurements have slight (3.329369797859691e-2%) effect on estimated standard deviation.
nub . sort/1250(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.321387417966262e-3 | 6.348620416217634e-3 | 6.3866863059482415e-3 |
| Standard deviation | 5.383113883236453e-5 | 7.631870970815167e-5 | 9.537241511140924e-5 |
Outlying measurements have slight (3.993055555555533e-2%) effect on estimated standard deviation.
nub . sort/1500(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.615547222310872e-3 | 8.650457909237949e-3 | 8.679457762997033e-3 |
| Standard deviation | 5.145620885817278e-5 | 6.765109157449832e-5 | 9.086452518526152e-5 |
Outlying measurements have slight (4.75e-2%) effect on estimated standard deviation.
nub . sort/2000(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.4677183088991004e-2 | 1.4721149739973457e-2 | 1.4785967075170619e-2 |
| Standard deviation | 6.327954969946401e-5 | 9.338658822293738e-5 | 1.2898297229723963e-4 |
Outlying measurements have slight (6.222222222222214e-2%) effect on estimated standard deviation.
nub . sort/3000(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.853796948834239e-2 | 2.867025901526434e-2 | 2.8825876823639608e-2 |
| Standard deviation | 1.4960707757574493e-4 | 2.1569815755650683e-4 | 2.996357553037469e-4 |
Outlying measurements have slight (8.264462809917343e-2%) effect on estimated standard deviation.
nub . sort/4000(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.375839540370247e-2 | 4.399164912349904e-2 | 4.444917283331893e-2 |
| Standard deviation | 1.8153020598941032e-4 | 4.5921466742600877e-4 | 7.104143875688181e-4 |
Outlying measurements have slight (9.876543209876543e-2%) effect on estimated standard deviation.
nub . sort/5000(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.223088502719259e-2 | 6.3445092219398e-2 | 6.438775305952807e-2 |
| Standard deviation | 9.050387112497013e-4 | 1.3527085969774414e-3 | 1.7786774115030046e-3 |
Outlying measurements have moderate (0.12244897959183673%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.