criterion performance measurements
overview
want to understand this report?
sort . nub/100(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.453850938638037e-5 | 5.471412776027625e-5 | 5.491529161853362e-5 |
| Standard deviation | 3.7697879722947886e-7 | 6.397775743462969e-7 | 9.345004082372795e-7 |
Outlying measurements have no (9.008264462809999e-3%) effect on estimated standard deviation.
sort . nub/200(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.986990388169876e-4 | 1.9893862236542727e-4 | 1.9949136964973537e-4 |
| Standard deviation | 9.115951843069287e-7 | 1.100605800286499e-6 | 1.4545431593816167e-6 |
Outlying measurements have slight (1.1763038548752698e-2%) effect on estimated standard deviation.
sort . nub/400(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.414636715580551e-4 | 7.427698084659206e-4 | 7.43885625014405e-4 |
| Standard deviation | 2.5024575305449925e-6 | 3.322815556794531e-6 | 4.337696387376704e-6 |
Outlying measurements have slight (1.6661878770468253e-2%) effect on estimated standard deviation.
sort . nub/600(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.6494096933717607e-3 | 1.6517972690098944e-3 | 1.6566223102468516e-3 |
| Standard deviation | 6.564182223480334e-6 | 8.309000909112374e-6 | 1.219031405293991e-5 |
Outlying measurements have slight (2.2210743801652746e-2%) effect on estimated standard deviation.
sort . nub/800(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.8510772774250426e-3 | 2.857573422871714e-3 | 2.864390102895223e-3 |
| Standard deviation | 1.1792448225060534e-5 | 1.7650915483578528e-5 | 2.580523301562686e-5 |
Outlying measurements have slight (2.7755102040816326e-2%) effect on estimated standard deviation.
sort . nub/1000(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.379310997747846e-3 | 4.399213493398461e-3 | 4.420124758815521e-3 |
| Standard deviation | 3.5175638481365103e-5 | 4.8487678212819074e-5 | 6.003832417670799e-5 |
Outlying measurements have slight (3.329369797859691e-2%) effect on estimated standard deviation.
sort . nub/1250(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.686882756046457e-3 | 6.70409519324151e-3 | 6.728038679921771e-3 |
| Standard deviation | 3.791903654905298e-5 | 5.244010187695931e-5 | 7.574583101925404e-5 |
Outlying measurements have slight (4.158790170132312e-2%) effect on estimated standard deviation.
sort . nub/1500(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.745521974270319e-3 | 9.762268140557563e-3 | 9.779747425291372e-3 |
| Standard deviation | 2.9679734900136248e-5 | 3.573388119203744e-5 | 4.652984228011401e-5 |
Outlying measurements have slight (4.986149584487535e-2%) effect on estimated standard deviation.
sort . nub/2000(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.728268893333036e-2 | 1.733569587261118e-2 | 1.7421017955464583e-2 |
| Standard deviation | 8.238709178367203e-5 | 1.4427494873396114e-4 | 2.1805909969071688e-4 |
Outlying measurements have slight (6.632653061224489e-2%) effect on estimated standard deviation.
sort . nub/3000(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.93191138256386e-2 | 3.9453436002884514e-2 | 3.977344160762788e-2 |
| Standard deviation | 1.9712448106069073e-4 | 3.191652420889177e-4 | 4.90559024418983e-4 |
Outlying measurements have slight (9.876543209876541e-2%) effect on estimated standard deviation.
sort . nub/4000(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.919462500236694e-2 | 7.01031441894613e-2 | 7.082037386804867e-2 |
| Standard deviation | 8.132177476423067e-4 | 1.0878608090100369e-3 | 1.3520919552770234e-3 |
Outlying measurements have moderate (0.12244897959183673%) effect on estimated standard deviation.
sort . nub/5000(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.10772653821998346 | 0.10869444035999928 | 0.10961440770000383 |
| Standard deviation | 1.0921661338497296e-3 | 1.272291576062404e-3 | 1.5007757340956254e-3 |
Outlying measurements have moderate (0.16%) effect on estimated standard deviation.
nub . sort/100(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.782080260462084e-5 | 5.801871818512889e-5 | 5.829150454067115e-5 |
| Standard deviation | 3.610371358394735e-7 | 6.305845363754592e-7 | 8.995785105012933e-7 |
Outlying measurements have no (9.090143927278908e-3%) effect on estimated standard deviation.
nub . sort/200(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.0676286569605522e-4 | 2.0744929452215518e-4 | 2.0851782726882289e-4 |
| Standard deviation | 1.7755214999875027e-6 | 2.3265783107110517e-6 | 2.866223371903915e-6 |
Outlying measurements have slight (1.1763038548752802e-2%) effect on estimated standard deviation.
nub . sort/400(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.984157388769116e-4 | 8.001211422436365e-4 | 8.036483998075468e-4 |
| Standard deviation | 4.923649004721395e-6 | 6.941073010214901e-6 | 8.799929766039026e-6 |
Outlying measurements have slight (1.7236072637734613e-2%) effect on estimated standard deviation.
nub . sort/600(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.786612626774282e-3 | 1.7892369885313209e-3 | 1.7924610015618186e-3 |
| Standard deviation | 5.665312783320385e-6 | 8.598010387484495e-6 | 1.2227604128828813e-5 |
Outlying measurements have slight (2.2714981070848923e-2%) effect on estimated standard deviation.
nub . sort/800(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.1076592921578812e-3 | 3.1137830440364517e-3 | 3.124371019525439e-3 |
| Standard deviation | 1.1675437932457002e-5 | 2.0764853400867385e-5 | 3.3315622474381944e-5 |
Outlying measurements have slight (2.8546712802768166e-2%) effect on estimated standard deviation.
nub . sort/1000(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.811401245419065e-3 | 4.831423841666399e-3 | 4.851152603042935e-3 |
| Standard deviation | 3.708778400199269e-5 | 4.706829127055399e-5 | 6.474347707097212e-5 |
Outlying measurements have slight (3.4438775510204085e-2%) effect on estimated standard deviation.
nub . sort/1250(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.352561775574147e-3 | 7.38812984659878e-3 | 7.430993909336748e-3 |
| Standard deviation | 8.389934010196237e-5 | 1.0243771394567226e-4 | 1.446370703493651e-4 |
Outlying measurements have slight (4.338842975206612e-2%) effect on estimated standard deviation.
nub . sort/1500(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.0451049497016387e-2 | 1.048616622610532e-2 | 1.05330847608525e-2 |
| Standard deviation | 6.512161026936831e-5 | 7.423145283043214e-5 | 9.681006542142066e-5 |
Outlying measurements have slight (4.986149584487534e-2%) effect on estimated standard deviation.
nub . sort/2000(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.8106190535114013e-2 | 1.818871288436092e-2 | 1.8265146392153028e-2 |
| Standard deviation | 8.146107424708149e-5 | 1.6272357012631335e-4 | 2.0887544853779569e-4 |
Outlying measurements have slight (6.632653061224476e-2%) effect on estimated standard deviation.
nub . sort/3000(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.950679114939697e-2 | 3.971894308964747e-2 | 3.999238516113379e-2 |
| Standard deviation | 2.311581340160376e-4 | 3.787073661153276e-4 | 5.160695501832153e-4 |
Outlying measurements have slight (9.876543209876536e-2%) effect on estimated standard deviation.
nub . sort/4000(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.881639110713747e-2 | 6.907053715814523e-2 | 6.969499453738177e-2 |
| Standard deviation | 2.4863740114935994e-4 | 5.36914520320153e-4 | 7.546363427730072e-4 |
Outlying measurements have moderate (0.12244897959183675%) effect on estimated standard deviation.
nub . sort/5000(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.10705690508002134 | 0.10864958899666817 | 0.11229230031665338 |
| Standard deviation | 1.377542827549194e-3 | 2.782372863500341e-3 | 3.9910287973872905e-3 |
Outlying measurements have moderate (0.16000000000000003%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.