criterion performance measurements
overview
want to understand this report?
100% dups | |
99% dups | |
95% dups | |
90% dups | |
80% dups | |
75% dups | |
70% dups | |
60% dups | |
50% dups | |
40% dups | |
30% dups | |
25% dups | |
20% dups | |
10% dups | |
5% dups | |
1% dups | |
0% dups |
100% dups/sort . nub 10000
3.55 3.58 3.60 3.63 3.65 3.68 3.7
mean |
20 30 40 50 60 70 10 iters 100 150 200 250 300 0 s 50 ms
|
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | 3.55 ms | 3.57 ms | 3.58 ms |
R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
Mean execution time | 3.58 ms | 3.59 ms | 3.60 ms |
Standard deviation | 24.9 μs | 33.8 μs | 45.7 μs |
Outlying measurements have slight (2.1%) effect on estimated standard deviation.
100% dups/nub . sort 10000
6.9 6.95 7.00 7.05 7.10 7.15 7.2
mean |
10 15 20 25 30 35 40 5 iters 100 150 200 250 300 350 0 s 50 ms
|
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | 6.92 ms | 6.94 ms | 6.96 ms |
R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
Mean execution time | 6.94 ms | 6.95 ms | 6.98 ms |
Standard deviation | 27.9 μs | 53.5 μs | 103 μs |
Outlying measurements have slight (2.7%) effect on estimated standard deviation.
99% dups/sort . nub 10000
83 84 83.3 83.5 83.8 84.3
mean |
4 6 8 10 2 iters 400 600 800 0 s 200 ms 1 s
|
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | 83.5 ms | 83.9 ms | 84.1 ms |
R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
Mean execution time | 83.4 ms | 83.7 ms | 83.9 ms |
Standard deviation | 286 μs | 399 μs | 574 μs |
Outlying measurements have slight (9.0%) effect on estimated standard deviation.
99% dups/nub . sort 10000
28 27.4 27.5 27.6 27.7 27.8 27.9 28.1 28.2
mean |
5 8 10 13 15 18 2.5 iters 200 300 400 500 600 0 s 100 ms
|
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | 27.9 ms | 28.0 ms | 28.3 ms |
R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
Mean execution time | 27.7 ms | 27.8 ms | 27.9 ms |
Standard deviation | 170 μs | 227 μs | 293 μs |
Outlying measurements have slight (5.2%) effect on estimated standard deviation.
95% dups/sort . nub 10000
120 120 121 121 122 122
mean |
2 3 4 5 6 7 8 1 iters 400 600 800 0 s 200 ms 1 s
|
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | 120 ms | 121 ms | 122 ms |
R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
Mean execution time | 120 ms | 121 ms | 122 ms |
Standard deviation | 602 μs | 924 μs | 1.26 ms |
Outlying measurements have moderate (10.9%) effect on estimated standard deviation.
95% dups/nub . sort 10000
51.6 51.8 52.0 52.2 52.4 52.6
mean |
4 6 8 10 12 2 iters 200 300 400 500 600 700 0 s 100 ms
|
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | 52.1 ms | 52.2 ms | 52.3 ms |
R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
Mean execution time | 52.0 ms | 52.1 ms | 52.2 ms |
Standard deviation | 164 μs | 265 μs | 394 μs |
Outlying measurements have slight (7.1%) effect on estimated standard deviation.
90% dups/sort . nub 10000
150 153 155 158 160 163
mean |
2 3 4 5 6 7 1 iters 400 600 800 0 s 200 ms 1 s 1.2
|
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | 146 ms | 149 ms | 153 ms |
R² goodness-of-fit | 0.998 | 0.999 | 1.000 |
Mean execution time | 151 ms | 153 ms | 159 ms |
Standard deviation | 731 μs | 4.28 ms | 6.32 ms |
Outlying measurements have moderate (12.2%) effect on estimated standard deviation.
90% dups/nub . sort 10000
75 76 77 78 79 75.5 76.5 77.5 78.5
mean |
4 6 8 10 2 iters 400 600 800 0 s 200 ms
|
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | 74.4 ms | 76.2 ms | 77.6 ms |
R² goodness-of-fit | 0.999 | 0.999 | 1.000 |
Mean execution time | 75.6 ms | 76.1 ms | 77.2 ms |
Standard deviation | 549 μs | 1.17 ms | 1.89 ms |
Outlying measurements have slight (9.0%) effect on estimated standard deviation.
80% dups/sort . nub 10000
189 190 191 192 193 194
mean |
2 3 4 5 6 1 iters 500 750 0 s 250 ms 1 s 1.25
|
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | 188 ms | 190 ms | 193 ms |
R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
Mean execution time | 190 ms | 191 ms | 192 ms |
Standard deviation | 639 μs | 1.63 ms | 2.35 ms |
Outlying measurements have moderate (13.9%) effect on estimated standard deviation.
80% dups/nub . sort 10000
116 117 117 118 118 119 119 120
mean |
2 3 4 5 6 7 8 1 iters 400 600 800 0 s 200 ms 1 s
|
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | 117 ms | 118 ms | 120 ms |
R² goodness-of-fit | 0.999 | 1.000 | 1.000 |
Mean execution time | 117 ms | 118 ms | 119 ms |
Standard deviation | 846 μs | 1.20 ms | 1.78 ms |
Outlying measurements have moderate (10.9%) effect on estimated standard deviation.
75% dups/sort . nub 10000
203 204 205 206 207 208 209
mean |
2 3 4 5 6 1 iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | 202 ms | 207 ms | 214 ms |
R² goodness-of-fit | 0.998 | 0.999 | 1.000 |
Mean execution time | 204 ms | 205 ms | 207 ms |
Standard deviation | 850 μs | 2.22 ms | 3.16 ms |
Outlying measurements have moderate (13.9%) effect on estimated standard deviation.
75% dups/nub . sort 10000
139 140 141 142 143 144 145
mean |
2 3 4 5 6 7 1 iters 400 600 800 0 s 200 ms 1 s 1.2
|
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | 139 ms | 145 ms | 150 ms |
R² goodness-of-fit | 0.997 | 0.999 | 1.000 |
Mean execution time | 139 ms | 141 ms | 143 ms |
Standard deviation | 1.01 ms | 2.34 ms | 3.40 ms |
Outlying measurements have moderate (12.2%) effect on estimated standard deviation.
70% dups/sort . nub 10000
217 218 218 219 219
mean |
2 3 4 5 6 1 iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | 218 ms | 219 ms | 220 ms |
R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
Mean execution time | 217 ms | 218 ms | 219 ms |
Standard deviation | 418 μs | 817 μs | 1.12 ms |
Outlying measurements have moderate (13.9%) effect on estimated standard deviation.
70% dups/nub . sort 10000
158 159 159 160 160
mean |
2 3 4 5 6 7 1 iters 500 750 0 s 250 ms 1 s 1.25
|
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | 159 ms | 160 ms | 161 ms |
R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
Mean execution time | 159 ms | 159 ms | 159 ms |
Standard deviation | 514 μs | 768 μs | 1.03 ms |
Outlying measurements have moderate (12.2%) effect on estimated standard deviation.
60% dups/sort . nub 10000
244 244 245 245 246 246
mean |
2 3 4 5 1 iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | 245 ms | 246 ms | 248 ms |
R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
Mean execution time | 244 ms | 244 ms | 245 ms |
Standard deviation | 355 μs | 914 μs | 1.30 ms |
Outlying measurements have moderate (16.0%) effect on estimated standard deviation.
60% dups/nub . sort 10000
197 198 198 199 199
mean |
2 3 4 5 6 1 iters 500 750 0 s 250 ms 1 s 1.25
|
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | 196 ms | 198 ms | 199 ms |
R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
Mean execution time | 197 ms | 198 ms | 198 ms |
Standard deviation | 516 μs | 873 μs | 1.16 ms |
Outlying measurements have moderate (13.9%) effect on estimated standard deviation.
50% dups/sort . nub 10000
274 274 275 275
mean |
2 3 4 5 1 iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | 274 ms | 275 ms | 276 ms |
R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
Mean execution time | 274 ms | 274 ms | 275 ms |
Standard deviation | 566 μs | 727 μs | 891 μs |
Outlying measurements have moderate (16.0%) effect on estimated standard deviation.
50% dups/nub . sort 10000
237 238 238 239 239 240 240 241 241
mean |
2 3 4 5 1 iters 500 750 0 s 250 ms 1 s 1.25
|
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | 240 ms | 241 ms | 243 ms |
R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
Mean execution time | 238 ms | 240 ms | 241 ms |
Standard deviation | 191 μs | 1.58 ms | 2.04 ms |
Outlying measurements have moderate (16.0%) effect on estimated standard deviation.
40% dups/sort . nub 10000
298 299 300 301 302 303 304
mean |
2 3 4 5 1 iters 2 0 s 500 ms 1 s 1.5
|
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | 302 ms | 303 ms | 307 ms |
R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
Mean execution time | 299 ms | 301 ms | 302 ms |
Standard deviation | 1.04 ms | 2.03 ms | 3.09 ms |
Outlying measurements have moderate (16.0%) effect on estimated standard deviation.
40% dups/nub . sort 10000
277 278 279 280 281 282 283
mean |
2 3 4 5 1 iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | 281 ms | 283 ms | 286 ms |
R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
Mean execution time | 279 ms | 280 ms | 282 ms |
Standard deviation | 1.11 ms | 1.94 ms | 2.88 ms |
Outlying measurements have moderate (16.0%) effect on estimated standard deviation.
30% dups/sort . nub 10000
327 327 328 328
mean |
2 3 4 5 1 iters 2 0 s 500 ms 1 s 1.5
|
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | 324 ms | 326 ms | 327 ms |
R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
Mean execution time | 326 ms | 327 ms | 328 ms |
Standard deviation | 44.1 μs | 753 μs | 966 μs |
Outlying measurements have moderate (16.0%) effect on estimated standard deviation.
30% dups/nub . sort 10000
326 327 328 329 330
mean |
2 3 4 5 1 iters 2 0 s 500 ms 1 s 1.5
|
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | 326 ms | 329 ms | 333 ms |
R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
Mean execution time | 327 ms | 328 ms | 329 ms |
Standard deviation | 902 μs | 1.48 ms | 2.21 ms |
Outlying measurements have moderate (16.0%) effect on estimated standard deviation.
25% dups/sort . nub 10000
336 337 338 339 340 341 342
mean |
1 2 2 3 3 4 4 0.5 iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | 335 ms | 342 ms | 348 ms |
R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
Mean execution time | 337 ms | 339 ms | 341 ms |
Standard deviation | 1.19 ms | 2.47 ms | 3.37 ms |
Outlying measurements have moderate (18.7%) effect on estimated standard deviation.
25% dups/nub . sort 10000
345 346 347 348 349 350 351
mean |
1 2 2 3 3 4 4 0.5 iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | 348 ms | 352 ms | 357 ms |
R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
Mean execution time | 346 ms | 349 ms | 351 ms |
Standard deviation | 268 μs | 2.94 ms | 3.64 ms |
Outlying measurements have moderate (18.8%) effect on estimated standard deviation.
20% dups/sort . nub 10000
349 350 351 352 353 354 355
mean |
1 2 2 3 3 4 4 0.5 iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | 349 ms | 357 ms | 365 ms |
R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
Mean execution time | 349 ms | 352 ms | 354 ms |
Standard deviation | 2.68 ms | 3.10 ms | 3.19 ms |
Outlying measurements have moderate (18.8%) effect on estimated standard deviation.
20% dups/nub . sort 10000
368 370 372 374 376 378 380
mean |
1 2 2 3 3 4 4 0.5 iters 2 0 s 500 ms 1 s 1.5
|
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | 332 ms | 368 ms | 391 ms |
R² goodness-of-fit | 0.997 | 0.999 | 1.000 |
Mean execution time | 369 ms | 374 ms | 377 ms |
Standard deviation | 2.73 ms | 4.82 ms | 6.80 ms |
Outlying measurements have moderate (18.8%) effect on estimated standard deviation.
10% dups/sort . nub 10000
373 374 375 376 377 378
mean |
1 2 2 3 3 4 4 0.5 iters 2 0 s 500 ms 1 s 1.5
|
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | 374 ms | 379 ms | 386 ms |
R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
Mean execution time | 374 ms | 376 ms | 377 ms |
Standard deviation | 968 μs | 2.24 ms | 3.09 ms |
Outlying measurements have moderate (18.8%) effect on estimated standard deviation.
10% dups/nub . sort 10000
413 414 415 416 417 418 419
mean |
1 2 2 3 3 4 4 0.5 iters 2 0 s 500 ms 1 s 1.5
|
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | 411 ms | 421 ms | 426 ms |
R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
Mean execution time | 414 ms | 416 ms | 418 ms |
Standard deviation | 1.49 ms | 2.55 ms | 3.42 ms |
Outlying measurements have moderate (18.8%) effect on estimated standard deviation.
5% dups/sort . nub 10000
384 385 385 386 386 387 387 388 388
mean |
1 2 2 3 3 4 4 0.5 iters 2 0 s 500 ms 1 s 1.5
|
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | 380 ms | 384 ms | 388 ms |
R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
Mean execution time | 385 ms | 386 ms | 387 ms |
Standard deviation | 606 μs | 1.53 ms | 2.08 ms |
Outlying measurements have moderate (18.8%) effect on estimated standard deviation.
5% dups/nub . sort 10000
430 432 434 436 438
mean |
1 2 2 3 3 4 4 0.5 iters 2 0 s 500 ms 1 s 1.5
|
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | 437 ms | 441 ms | 445 ms |
R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
Mean execution time | 432 ms | 436 ms | 438 ms |
Standard deviation | 1.11 ms | 3.46 ms | 4.67 ms |
Outlying measurements have moderate (18.8%) effect on estimated standard deviation.
1% dups/sort . nub 10000
397 398 398 399 399 400 400
mean |
1 2 2 3 3 4 4 0.5 iters 2 0 s 500 ms 1 s 1.5
|
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | 388 ms | 397 ms | 403 ms |
R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
Mean execution time | 397 ms | 399 ms | 400 ms |
Standard deviation | 1.28 ms | 1.57 ms | 1.77 ms |
Outlying measurements have moderate (18.8%) effect on estimated standard deviation.
1% dups/nub . sort 10000
448 450 452 454 456
mean |
1 2 2 3 3 4 4 0.5 iters 2 0 s 500 ms 1 s 1.5
|
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | 444 ms | 456 ms | 463 ms |
R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
Mean execution time | 450 ms | 453 ms | 455 ms |
Standard deviation | 1.52 ms | 3.56 ms | 4.93 ms |
Outlying measurements have moderate (18.8%) effect on estimated standard deviation.
0% dups/sort . nub 10000
395 398 400 403 405 408
mean |
1 2 2 3 3 4 4 0.5 iters 2 0 s 500 ms 1 s 1.5
|
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | 409 ms | 411 ms | 414 ms |
R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
Mean execution time | 396 ms | 401 ms | 405 ms |
Standard deviation | 1.56 ms | 5.47 ms | 7.33 ms |
Outlying measurements have moderate (18.7%) effect on estimated standard deviation.
0% dups/nub . sort 10000
453 454 455 456 457 458
mean |
1 2 2 3 3 4 4 0.5 iters 2 0 s 500 ms 1 s 1.5
|
lower bound | estimate | upper bound | |
---|---|---|---|
OLS regression | 446 ms | 457 ms | 469 ms |
R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
Mean execution time | 453 ms | 455 ms | 457 ms |
Standard deviation | 915 μs | 2.45 ms | 3.11 ms |
Outlying measurements have moderate (18.7%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.