criterion performance measurements
overview
want to understand this report?
| sort . nub | |
| nub . sort |
sort . nub/100(1->1400)
84 85 83.5 84.5 85.5
mean |
400 600 800 1000 200 iters 40 60 80 100 0 s 20 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 84.0 μs | 84.2 μs | 84.6 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 84.1 μs | 84.3 μs | 84.5 μs |
| Standard deviation | 544 ns | 649 ns | 789 ns |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
sort . nub/200(1->1400)
278 280 282 284 286 288
mean |
100 150 200 250 300 50 iters 40 60 80 100 120 0 s 20 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 280 μs | 281 μs | 284 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 280 μs | 281 μs | 282 μs |
| Standard deviation | 2.11 μs | 2.75 μs | 3.74 μs |
Outlying measurements have slight (1.3%) effect on estimated standard deviation.
sort . nub/400(1->1400)
900 920 940 960 980 1 ms
mean |
40 60 80 100 20 iters 40 60 80 100 120 0 s 20 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 916 μs | 924 μs | 939 μs |
| R² goodness-of-fit | 0.997 | 0.999 | 1.000 |
| Mean execution time | 923 μs | 927 μs | 937 μs |
| Standard deviation | 10.6 μs | 18.1 μs | 29.5 μs |
Outlying measurements have slight (8.5%) effect on estimated standard deviation.
sort . nub/600(1->1400)
1.82 1.84 1.86 1.88 1.90 1.92 1.94
mean |
20 30 40 50 60 10 iters 50 75 100 125 0 s 25 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 1.85 ms | 1.86 ms | 1.87 ms |
| R² goodness-of-fit | 0.999 | 1.000 | 1.000 |
| Mean execution time | 1.86 ms | 1.87 ms | 1.88 ms |
| Standard deviation | 13.9 μs | 20.2 μs | 28.3 μs |
Outlying measurements have slight (2.3%) effect on estimated standard deviation.
sort . nub/800(1->1400)
2.75 2.77 2.80 2.83 2.85 2.88 2.9
mean |
10 15 20 25 30 35 40 5 iters 50 75 100 125 0 s 25 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 2.78 ms | 2.80 ms | 2.82 ms |
| R² goodness-of-fit | 0.999 | 1.000 | 1.000 |
| Mean execution time | 2.80 ms | 2.81 ms | 2.83 ms |
| Standard deviation | 27.8 μs | 35.7 μs | 47.4 μs |
Outlying measurements have slight (2.8%) effect on estimated standard deviation.
sort . nub/1000(1->1400)
4.1 4.15 4.2 4.25
mean |
10 15 20 25 30 5 iters 50 75 100 125 150 0 s 25 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 4.12 ms | 4.14 ms | 4.16 ms |
| R² goodness-of-fit | 0.999 | 1.000 | 1.000 |
| Mean execution time | 4.14 ms | 4.16 ms | 4.18 ms |
| Standard deviation | 32.7 μs | 47.4 μs | 64.0 μs |
Outlying measurements have slight (3.2%) effect on estimated standard deviation.
sort . nub/1250(1->1400)
5.7 5.75 5.80 5.85 5.9
mean |
10 15 20 25 5 iters 100 150 200 0 s 50 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 5.74 ms | 5.79 ms | 5.84 ms |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 5.79 ms | 5.81 ms | 5.83 ms |
| Standard deviation | 43.1 μs | 53.3 μs | 69.8 μs |
Outlying measurements have slight (3.8%) effect on estimated standard deviation.
sort . nub/1500(1->1400)
7.5 7.55 7.60 7.65 7.7 7.75 7.8
mean |
10 15 20 5 iters 100 150 200 0 s 50 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 7.56 ms | 7.62 ms | 7.66 ms |
| R² goodness-of-fit | 0.999 | 1.000 | 1.000 |
| Mean execution time | 7.61 ms | 7.63 ms | 7.66 ms |
| Standard deviation | 57.1 μs | 71.6 μs | 99.3 μs |
Outlying measurements have slight (4.3%) effect on estimated standard deviation.
sort . nub/2000(1->1400)
11 11.1 11.2 11.3 11.4 11.5 11.6 11.7 11.8
mean |
5 8 10 13 15 18 2.5 iters 100 150 200 250 0 s 50 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 11.0 ms | 11.1 ms | 11.3 ms |
| R² goodness-of-fit | 0.999 | 0.999 | 1.000 |
| Mean execution time | 11.3 ms | 11.4 ms | 11.5 ms |
| Standard deviation | 152 μs | 190 μs | 236 μs |
Outlying measurements have slight (5.2%) effect on estimated standard deviation.
sort . nub/3000(1->1400)
16 17 18 19 16.5 17.5 18.5
mean |
4 6 8 10 12 14 2 iters 100 150 200 250 300 0 s 50 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 18.7 ms | 19.2 ms | 19.9 ms |
| R² goodness-of-fit | 0.997 | 0.998 | 1.000 |
| Mean execution time | 17.4 ms | 17.9 ms | 18.3 ms |
| Standard deviation | 708 μs | 968 μs | 1.18 ms |
Outlying measurements have moderate (13.5%) effect on estimated standard deviation.
sort . nub/4000(1->1400)
28 29 27.8 28.2 28.4 28.6 28.8
mean |
4 6 8 10 2 iters 100 150 200 250 300 350 0 s 50 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 27.6 ms | 28.0 ms | 28.3 ms |
| R² goodness-of-fit | 0.999 | 1.000 | 1.000 |
| Mean execution time | 28.1 ms | 28.4 ms | 28.5 ms |
| Standard deviation | 244 μs | 373 μs | 508 μs |
Outlying measurements have slight (8.3%) effect on estimated standard deviation.
sort . nub/5000(1->1400)
37 38 36.5 37.5 38.5
mean |
2 3 4 5 6 7 8 9 1 iters 100 150 200 250 300 350 0 s 50 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 35.8 ms | 36.1 ms | 36.4 ms |
| R² goodness-of-fit | 0.999 | 1.000 | 1.000 |
| Mean execution time | 36.6 ms | 36.9 ms | 37.5 ms |
| Standard deviation | 325 μs | 709 μs | 1.07 ms |
Outlying measurements have slight (9.9%) effect on estimated standard deviation.
nub . sort/100(1->1400)
82 83 84 85 86 87 88
mean |
400 600 800 1000 200 iters 40 60 80 100 0 s 20 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 84.5 μs | 85.1 μs | 85.8 μs |
| R² goodness-of-fit | 0.999 | 1.000 | 1.000 |
| Mean execution time | 84.3 μs | 85.0 μs | 85.5 μs |
| Standard deviation | 1.11 μs | 1.33 μs | 1.75 μs |
Outlying measurements have slight (8.3%) effect on estimated standard deviation.
nub . sort/200(1->1400)
270 273 275 278 280 283 285 288
mean |
100 150 200 250 300 50 iters 40 60 80 100 0 s 20 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 275 μs | 277 μs | 278 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 275 μs | 277 μs | 278 μs |
| Standard deviation | 2.82 μs | 3.77 μs | 5.15 μs |
Outlying measurements have slight (1.3%) effect on estimated standard deviation.
nub . sort/400(1->1400)
865 870 875 880 885 890 895 900
mean |
40 60 80 100 20 iters 40 60 80 100 120 0 s 20 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 869 μs | 876 μs | 883 μs |
| R² goodness-of-fit | 0.999 | 1.000 | 1.000 |
| Mean execution time | 873 μs | 876 μs | 879 μs |
| Standard deviation | 4.91 μs | 7.74 μs | 11.1 μs |
Outlying measurements have slight (1.8%) effect on estimated standard deviation.
nub . sort/600(1->1400)
1.56 1.58 1.6 1.62 1.64
mean |
20 30 40 50 60 10 iters 50 75 100 125 0 s 25 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 1.61 ms | 1.62 ms | 1.63 ms |
| R² goodness-of-fit | 0.999 | 0.999 | 1.000 |
| Mean execution time | 1.60 ms | 1.61 ms | 1.61 ms |
| Standard deviation | 19.5 μs | 23.8 μs | 30.0 μs |
Outlying measurements have slight (2.2%) effect on estimated standard deviation.
nub . sort/800(1->1400)
2.28 2.30 2.32 2.34 2.36 2.38
mean |
20 30 40 50 10 iters 50 75 100 125 0 s 25 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 2.30 ms | 2.32 ms | 2.34 ms |
| R² goodness-of-fit | 0.999 | 1.000 | 1.000 |
| Mean execution time | 2.31 ms | 2.31 ms | 2.33 ms |
| Standard deviation | 23.9 μs | 27.9 μs | 37.7 μs |
Outlying measurements have slight (2.5%) effect on estimated standard deviation.
nub . sort/1000(1->1400)
2.95 2.98 3.00 3.03 3.05 3.08 3.1
mean |
10 15 20 25 30 35 40 5 iters 50 75 100 125 150 0 s 25 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 3.01 ms | 3.04 ms | 3.06 ms |
| R² goodness-of-fit | 0.999 | 1.000 | 1.000 |
| Mean execution time | 3.01 ms | 3.02 ms | 3.04 ms |
| Standard deviation | 23.8 μs | 31.8 μs | 41.1 μs |
Outlying measurements have slight (2.9%) effect on estimated standard deviation.
nub . sort/1250(1->1400)
4 5 4.2 4.4 4.60 4.80
mean |
10 15 20 25 30 5 iters 50 75 100 125 150 0 s 25 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 3.85 ms | 3.96 ms | 4.06 ms |
| R² goodness-of-fit | 0.996 | 0.996 | 0.998 |
| Mean execution time | 4.33 ms | 4.43 ms | 4.56 ms |
| Standard deviation | 238 μs | 296 μs | 373 μs |
Outlying measurements have moderate (33.0%) effect on estimated standard deviation.
nub . sort/1500(1->1400)
4.65 4.7 4.75 4.80 4.85
mean |
10 15 20 25 30 5 iters 50 75 100 125 150 0 s 25 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 4.62 ms | 4.66 ms | 4.69 ms |
| R² goodness-of-fit | 0.999 | 1.000 | 1.000 |
| Mean execution time | 4.69 ms | 4.72 ms | 4.74 ms |
| Standard deviation | 48.0 μs | 58.5 μs | 74.2 μs |
Outlying measurements have slight (3.4%) effect on estimated standard deviation.
nub . sort/2000(1->1400)
5.9 5.95 6.00 6.05
mean |
10 15 20 25 5 iters 100 150 200 0 s 50 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 5.97 ms | 6.01 ms | 6.06 ms |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 5.95 ms | 5.97 ms | 5.99 ms |
| Standard deviation | 40.9 μs | 48.7 μs | 60.0 μs |
Outlying measurements have slight (3.8%) effect on estimated standard deviation.
nub . sort/3000(1->1400)
8.15 8.2 8.25 8.30 8.35
mean |
10 15 20 5 iters 100 150 200 0 s 50 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 8.21 ms | 8.24 ms | 8.27 ms |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 8.21 ms | 8.23 ms | 8.25 ms |
| Standard deviation | 35.3 μs | 55.2 μs | 73.7 μs |
Outlying measurements have slight (4.5%) effect on estimated standard deviation.
nub . sort/4000(1->1400)
9.60 9.65 9.70 9.75 9.8 9.85 9.9 9.95
mean |
5 8 10 13 15 18 2.5 iters 100 150 200 0 s 50 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 9.66 ms | 9.69 ms | 9.75 ms |
| R² goodness-of-fit | 0.999 | 1.000 | 1.000 |
| Mean execution time | 9.68 ms | 9.70 ms | 9.78 ms |
| Standard deviation | 54.1 μs | 84.7 μs | 125 μs |
Outlying measurements have slight (5.0%) effect on estimated standard deviation.
nub . sort/5000(1->1400)
10.6 10.6 10.7 10.7 10.8 10.8 10.8 10.9 10.9
mean |
5 8 10 13 15 18 2.5 iters 100 150 200 0 s 50 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 10.6 ms | 10.7 ms | 10.7 ms |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 10.7 ms | 10.7 ms | 10.7 ms |
| Standard deviation | 56.8 μs | 87.0 μs | 140 μs |
Outlying measurements have slight (5.2%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.