criterion performance measurements
overview
want to understand this report?
sort . nub/100(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.546841629269781e-5 | 8.584937048096788e-5 | 8.624970747521808e-5 |
| Standard deviation | 7.20174009973552e-7 | 1.0700347883622215e-6 | 1.3952192787962392e-6 |
Outlying measurements have slight (1.3909043397648202e-2%) effect on estimated standard deviation.
sort . nub/200(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.9269619781980823e-4 | 2.942839951956196e-4 | 2.958179225917927e-4 |
| Standard deviation | 3.73693173320572e-6 | 4.448128531775306e-6 | 5.460641977658418e-6 |
Outlying measurements have slight (6.11899384488482e-2%) effect on estimated standard deviation.
sort . nub/400(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.0490775038703086e-3 | 1.0555796109079862e-3 | 1.0614110226542365e-3 |
| Standard deviation | 1.3244502633068148e-5 | 1.6938676525715548e-5 | 2.2514852992271964e-5 |
Outlying measurements have slight (1.886094674556201e-2%) effect on estimated standard deviation.
sort . nub/600(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.1434514179692068e-3 | 2.154907885036662e-3 | 2.16344544312799e-3 |
| Standard deviation | 2.2627662933834434e-5 | 2.9277158891979785e-5 | 3.842668234715998e-5 |
Outlying measurements have slight (2.4375000000000004e-2%) effect on estimated standard deviation.
sort . nub/800(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.768658456339316e-3 | 3.7862631454278167e-3 | 3.8048734581313266e-3 |
| Standard deviation | 3.275055670647576e-5 | 4.0237084014588134e-5 | 5.387195365267709e-5 |
Outlying measurements have slight (3.1217481789802166e-2%) effect on estimated standard deviation.
sort . nub/1000(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.503171782593754e-3 | 5.557585611308571e-3 | 5.600811022644455e-3 |
| Standard deviation | 8.368164662479094e-5 | 1.1652621901952423e-4 | 1.5124494094881225e-4 |
Outlying measurements have slight (3.698224852070996e-2%) effect on estimated standard deviation.
sort . nub/1250(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.229522931324985e-3 | 8.273338679383095e-3 | 8.337513155700037e-3 |
| Standard deviation | 9.739411972780688e-5 | 1.407263018836428e-4 | 1.9917809086695312e-4 |
Outlying measurements have slight (4.5351473922902306e-2%) effect on estimated standard deviation.
sort . nub/1500(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.1994928868800953e-2 | 1.2161831278178483e-2 | 1.2497797554607016e-2 |
| Standard deviation | 3.1191978756822544e-4 | 4.20070532294413e-4 | 5.8192487559138e-4 |
Outlying measurements have moderate (0.10698387816364817%) effect on estimated standard deviation.
sort . nub/2000(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.982015398541929e-2 | 2.010893909686353e-2 | 2.0646271288210616e-2 |
| Standard deviation | 4.103266398413527e-4 | 7.6363598025004e-4 | 1.1182436383265634e-3 |
Outlying measurements have slight (7.199138974637169e-2%) effect on estimated standard deviation.
sort . nub/3000(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.354638064637443e-2 | 4.368810493288845e-2 | 4.3885791942154584e-2 |
| Standard deviation | 2.044560634943097e-4 | 2.7200347462272984e-4 | 3.642692870062572e-4 |
Outlying measurements have slight (9.876543209876537e-2%) effect on estimated standard deviation.
sort . nub/4000(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.25206570833419e-2 | 6.467311350443514e-2 | 6.85288620163532e-2 |
| Standard deviation | 1.6772991643773574e-3 | 3.997945169871697e-3 | 5.8528937282406514e-3 |
Outlying measurements have moderate (0.13093649723795298%) effect on estimated standard deviation.
sort . nub/5000(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.074479716667838e-2 | 9.425673861110227e-2 | 9.945467494439224e-2 |
| Standard deviation | 4.316267845667606e-3 | 6.322347437249021e-3 | 8.182793995115185e-3 |
Outlying measurements have moderate (0.15058857962501387%) effect on estimated standard deviation.
nub . sort/100(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.422455278798626e-5 | 8.474645057829384e-5 | 8.538784783354422e-5 |
| Standard deviation | 1.1598322526093595e-6 | 1.4288616978935427e-6 | 1.7716789831921928e-6 |
Outlying measurements have moderate (0.10157602422804703%) effect on estimated standard deviation.
nub . sort/200(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.890094175266903e-4 | 2.897204846428098e-4 | 2.909722456572082e-4 |
| Standard deviation | 1.4636123591269492e-6 | 2.399934281403193e-6 | 3.624998031502396e-6 |
Outlying measurements have slight (1.2818350480688112e-2%) effect on estimated standard deviation.
nub . sort/400(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.0065755189400732e-3 | 1.009410001568842e-3 | 1.0147694221071207e-3 |
| Standard deviation | 6.651144362656975e-6 | 9.707960849834675e-6 | 1.3470880834600166e-5 |
Outlying measurements have slight (1.851192595229611e-2%) effect on estimated standard deviation.
nub . sort/600(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.113584810754151e-3 | 2.120144321501387e-3 | 2.1264917735033923e-3 |
| Standard deviation | 1.2960324948911842e-5 | 1.7982782643479557e-5 | 2.3177795768776952e-5 |
Outlying measurements have slight (2.4374999999999997e-2%) effect on estimated standard deviation.
nub . sort/800(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.561114110774647e-3 | 3.5706501847382606e-3 | 3.596388431411545e-3 |
| Standard deviation | 3.108874526371017e-5 | 4.482764751992023e-5 | 7.088834406268091e-5 |
Outlying measurements have slight (3.02734375e-2%) effect on estimated standard deviation.
nub . sort/1000(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.183659805923741e-3 | 5.205463949548057e-3 | 5.246485133874872e-3 |
| Standard deviation | 5.063855549501027e-5 | 7.371287932914897e-5 | 1.0684476326803026e-4 |
Outlying measurements have slight (3.566529492455405e-2%) effect on estimated standard deviation.
nub . sort/1250(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.38348896559875e-3 | 8.498221078429992e-3 | 8.597681035040492e-3 |
| Standard deviation | 2.1508383370989867e-4 | 2.6678945658692584e-4 | 3.2527925148666107e-4 |
Outlying measurements have slight (8.839045621637563e-2%) effect on estimated standard deviation.
nub . sort/1500(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.571928628248731e-3 | 9.607996156957899e-3 | 9.707481216872704e-3 |
| Standard deviation | 6.772964730730569e-5 | 1.142632012598363e-4 | 1.5668649372768293e-4 |
Outlying measurements have slight (4.986149584487534e-2%) effect on estimated standard deviation.
nub . sort/2000(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.5194281565111342e-2 | 1.5284024159194689e-2 | 1.5448743075537804e-2 |
| Standard deviation | 1.4146362596824245e-4 | 2.2197391433386566e-4 | 2.9551935722971456e-4 |
Outlying measurements have slight (6.222222222222222e-2%) effect on estimated standard deviation.
nub . sort/3000(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.8630070203724718e-2 | 2.89187092577765e-2 | 2.9369789411828044e-2 |
| Standard deviation | 2.612653806637892e-4 | 6.519421010333519e-4 | 1.0063590666287189e-3 |
Outlying measurements have slight (8.264462809917356e-2%) effect on estimated standard deviation.
nub . sort/4000(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.241673675661926e-2 | 4.261864764417643e-2 | 4.277925535713089e-2 |
| Standard deviation | 2.338933061629494e-4 | 3.337015010484756e-4 | 4.546730097369296e-4 |
Outlying measurements have slight (9.87654320987654e-2%) effect on estimated standard deviation.
nub . sort/5000(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.947612604764107e-2 | 5.990333514695194e-2 | 6.035778493743039e-2 |
| Standard deviation | 3.902139466383181e-4 | 6.380750130338991e-4 | 9.206341287543349e-4 |
Outlying measurements have moderate (0.12244897959183673%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.