criterion performance measurements

overview

want to understand this report?

100% dups/sort . nub 10000

3.55
3.58
3.60
3.63
3.65
3.68
3.7
100% dups/sort . nub 10000 time densities
mean
20
30
40
50
60
70
10 iters
100
150
200
250
300
0 s
50 ms
regression
100% dups/sort . nub 10000 times
lower bound estimate upper bound
OLS regression 3.55 ms 3.57 ms 3.58 ms
R² goodness-of-fit 1.000 1.000 1.000
Mean execution time 3.58 ms 3.59 ms 3.60 ms
Standard deviation 24.9 μs 33.8 μs 45.7 μs

Outlying measurements have slight (2.1%) effect on estimated standard deviation.

100% dups/nub . sort 10000

6.9
6.95
7.00
7.05
7.10
7.15
7.2
100% dups/nub . sort 10000 time densities
mean
10
15
20
25
30
35
40
5 iters
100
150
200
250
300
350
0 s
50 ms
regression
100% dups/nub . sort 10000 times
lower bound estimate upper bound
OLS regression 6.92 ms 6.94 ms 6.96 ms
R² goodness-of-fit 1.000 1.000 1.000
Mean execution time 6.94 ms 6.95 ms 6.98 ms
Standard deviation 27.9 μs 53.5 μs 103 μs

Outlying measurements have slight (2.7%) effect on estimated standard deviation.

99% dups/sort . nub 10000

83
84
83.3
83.5
83.8
84.3
99% dups/sort . nub 10000 time densities
mean
4
6
8
10
2 iters
400
600
800
0 s
200 ms
1 s
regression
99% dups/sort . nub 10000 times
lower bound estimate upper bound
OLS regression 83.5 ms 83.9 ms 84.1 ms
R² goodness-of-fit 1.000 1.000 1.000
Mean execution time 83.4 ms 83.7 ms 83.9 ms
Standard deviation 286 μs 399 μs 574 μs

Outlying measurements have slight (9.0%) effect on estimated standard deviation.

99% dups/nub . sort 10000

28
27.4
27.5
27.6
27.7
27.8
27.9
28.1
28.2
99% dups/nub . sort 10000 time densities
mean
5
8
10
13
15
18
2.5 iters
200
300
400
500
600
0 s
100 ms
regression
99% dups/nub . sort 10000 times
lower bound estimate upper bound
OLS regression 27.9 ms 28.0 ms 28.3 ms
R² goodness-of-fit 1.000 1.000 1.000
Mean execution time 27.7 ms 27.8 ms 27.9 ms
Standard deviation 170 μs 227 μs 293 μs

Outlying measurements have slight (5.2%) effect on estimated standard deviation.

95% dups/sort . nub 10000

120
120
121
121
122
122
95% dups/sort . nub 10000 time densities
mean
2
3
4
5
6
7
8
1 iters
400
600
800
0 s
200 ms
1 s
regression
95% dups/sort . nub 10000 times
lower bound estimate upper bound
OLS regression 120 ms 121 ms 122 ms
R² goodness-of-fit 1.000 1.000 1.000
Mean execution time 120 ms 121 ms 122 ms
Standard deviation 602 μs 924 μs 1.26 ms

Outlying measurements have moderate (10.9%) effect on estimated standard deviation.

95% dups/nub . sort 10000

51.6
51.8
52.0
52.2
52.4
52.6
95% dups/nub . sort 10000 time densities
mean
4
6
8
10
12
2 iters
200
300
400
500
600
700
0 s
100 ms
regression
95% dups/nub . sort 10000 times
lower bound estimate upper bound
OLS regression 52.1 ms 52.2 ms 52.3 ms
R² goodness-of-fit 1.000 1.000 1.000
Mean execution time 52.0 ms 52.1 ms 52.2 ms
Standard deviation 164 μs 265 μs 394 μs

Outlying measurements have slight (7.1%) effect on estimated standard deviation.

90% dups/sort . nub 10000

150
153
155
158
160
163
90% dups/sort . nub 10000 time densities
mean
2
3
4
5
6
7
1 iters
400
600
800
0 s
200 ms
1 s
1.2
regression
90% dups/sort . nub 10000 times
lower bound estimate upper bound
OLS regression 146 ms 149 ms 153 ms
R² goodness-of-fit 0.998 0.999 1.000
Mean execution time 151 ms 153 ms 159 ms
Standard deviation 731 μs 4.28 ms 6.32 ms

Outlying measurements have moderate (12.2%) effect on estimated standard deviation.

90% dups/nub . sort 10000

75
76
77
78
79
75.5
76.5
77.5
78.5
90% dups/nub . sort 10000 time densities
mean
4
6
8
10
2 iters
400
600
800
0 s
200 ms
regression
90% dups/nub . sort 10000 times
lower bound estimate upper bound
OLS regression 74.4 ms 76.2 ms 77.6 ms
R² goodness-of-fit 0.999 0.999 1.000
Mean execution time 75.6 ms 76.1 ms 77.2 ms
Standard deviation 549 μs 1.17 ms 1.89 ms

Outlying measurements have slight (9.0%) effect on estimated standard deviation.

80% dups/sort . nub 10000

189
190
191
192
193
194
80% dups/sort . nub 10000 time densities
mean
2
3
4
5
6
1 iters
500
750
0 s
250 ms
1 s
1.25
regression
80% dups/sort . nub 10000 times
lower bound estimate upper bound
OLS regression 188 ms 190 ms 193 ms
R² goodness-of-fit 1.000 1.000 1.000
Mean execution time 190 ms 191 ms 192 ms
Standard deviation 639 μs 1.63 ms 2.35 ms

Outlying measurements have moderate (13.9%) effect on estimated standard deviation.

80% dups/nub . sort 10000

116
117
117
118
118
119
119
120
80% dups/nub . sort 10000 time densities
mean
2
3
4
5
6
7
8
1 iters
400
600
800
0 s
200 ms
1 s
regression
80% dups/nub . sort 10000 times
lower bound estimate upper bound
OLS regression 117 ms 118 ms 120 ms
R² goodness-of-fit 0.999 1.000 1.000
Mean execution time 117 ms 118 ms 119 ms
Standard deviation 846 μs 1.20 ms 1.78 ms

Outlying measurements have moderate (10.9%) effect on estimated standard deviation.

75% dups/sort . nub 10000

203
204
205
206
207
208
209
75% dups/sort . nub 10000 time densities
mean
2
3
4
5
6
1 iters
500
750
0 s
250 ms
1 s
1.25
1.5
regression
75% dups/sort . nub 10000 times
lower bound estimate upper bound
OLS regression 202 ms 207 ms 214 ms
R² goodness-of-fit 0.998 0.999 1.000
Mean execution time 204 ms 205 ms 207 ms
Standard deviation 850 μs 2.22 ms 3.16 ms

Outlying measurements have moderate (13.9%) effect on estimated standard deviation.

75% dups/nub . sort 10000

139
140
141
142
143
144
145
75% dups/nub . sort 10000 time densities
mean
2
3
4
5
6
7
1 iters
400
600
800
0 s
200 ms
1 s
1.2
regression
75% dups/nub . sort 10000 times
lower bound estimate upper bound
OLS regression 139 ms 145 ms 150 ms
R² goodness-of-fit 0.997 0.999 1.000
Mean execution time 139 ms 141 ms 143 ms
Standard deviation 1.01 ms 2.34 ms 3.40 ms

Outlying measurements have moderate (12.2%) effect on estimated standard deviation.

70% dups/sort . nub 10000

217
218
218
219
219
70% dups/sort . nub 10000 time densities
mean
2
3
4
5
6
1 iters
500
750
0 s
250 ms
1 s
1.25
1.5
regression
70% dups/sort . nub 10000 times
lower bound estimate upper bound
OLS regression 218 ms 219 ms 220 ms
R² goodness-of-fit 1.000 1.000 1.000
Mean execution time 217 ms 218 ms 219 ms
Standard deviation 418 μs 817 μs 1.12 ms

Outlying measurements have moderate (13.9%) effect on estimated standard deviation.

70% dups/nub . sort 10000

158
159
159
160
160
70% dups/nub . sort 10000 time densities
mean
2
3
4
5
6
7
1 iters
500
750
0 s
250 ms
1 s
1.25
regression
70% dups/nub . sort 10000 times
lower bound estimate upper bound
OLS regression 159 ms 160 ms 161 ms
R² goodness-of-fit 1.000 1.000 1.000
Mean execution time 159 ms 159 ms 159 ms
Standard deviation 514 μs 768 μs 1.03 ms

Outlying measurements have moderate (12.2%) effect on estimated standard deviation.

60% dups/sort . nub 10000

244
244
245
245
246
246
60% dups/sort . nub 10000 time densities
mean
2
3
4
5
1 iters
500
750
0 s
250 ms
1 s
1.25
1.5
regression
60% dups/sort . nub 10000 times
lower bound estimate upper bound
OLS regression 245 ms 246 ms 248 ms
R² goodness-of-fit 1.000 1.000 1.000
Mean execution time 244 ms 244 ms 245 ms
Standard deviation 355 μs 914 μs 1.30 ms

Outlying measurements have moderate (16.0%) effect on estimated standard deviation.

60% dups/nub . sort 10000

197
198
198
199
199
60% dups/nub . sort 10000 time densities
mean
2
3
4
5
6
1 iters
500
750
0 s
250 ms
1 s
1.25
regression
60% dups/nub . sort 10000 times
lower bound estimate upper bound
OLS regression 196 ms 198 ms 199 ms
R² goodness-of-fit 1.000 1.000 1.000
Mean execution time 197 ms 198 ms 198 ms
Standard deviation 516 μs 873 μs 1.16 ms

Outlying measurements have moderate (13.9%) effect on estimated standard deviation.

50% dups/sort . nub 10000

274
274
275
275
50% dups/sort . nub 10000 time densities
mean
2
3
4
5
1 iters
500
750
0 s
250 ms
1 s
1.25
1.5
regression
50% dups/sort . nub 10000 times
lower bound estimate upper bound
OLS regression 274 ms 275 ms 276 ms
R² goodness-of-fit 1.000 1.000 1.000
Mean execution time 274 ms 274 ms 275 ms
Standard deviation 566 μs 727 μs 891 μs

Outlying measurements have moderate (16.0%) effect on estimated standard deviation.

50% dups/nub . sort 10000

237
238
238
239
239
240
240
241
241
50% dups/nub . sort 10000 time densities
mean
2
3
4
5
1 iters
500
750
0 s
250 ms
1 s
1.25
regression
50% dups/nub . sort 10000 times
lower bound estimate upper bound
OLS regression 240 ms 241 ms 243 ms
R² goodness-of-fit 1.000 1.000 1.000
Mean execution time 238 ms 240 ms 241 ms
Standard deviation 191 μs 1.58 ms 2.04 ms

Outlying measurements have moderate (16.0%) effect on estimated standard deviation.

40% dups/sort . nub 10000

298
299
300
301
302
303
304
40% dups/sort . nub 10000 time densities
mean
2
3
4
5
1 iters
2
0 s
500 ms
1 s
1.5
regression
40% dups/sort . nub 10000 times
lower bound estimate upper bound
OLS regression 302 ms 303 ms 307 ms
R² goodness-of-fit 1.000 1.000 1.000
Mean execution time 299 ms 301 ms 302 ms
Standard deviation 1.04 ms 2.03 ms 3.09 ms

Outlying measurements have moderate (16.0%) effect on estimated standard deviation.

40% dups/nub . sort 10000

277
278
279
280
281
282
283
40% dups/nub . sort 10000 time densities
mean
2
3
4
5
1 iters
500
750
0 s
250 ms
1 s
1.25
1.5
regression
40% dups/nub . sort 10000 times
lower bound estimate upper bound
OLS regression 281 ms 283 ms 286 ms
R² goodness-of-fit 1.000 1.000 1.000
Mean execution time 279 ms 280 ms 282 ms
Standard deviation 1.11 ms 1.94 ms 2.88 ms

Outlying measurements have moderate (16.0%) effect on estimated standard deviation.

30% dups/sort . nub 10000

327
327
328
328
30% dups/sort . nub 10000 time densities
mean
2
3
4
5
1 iters
2
0 s
500 ms
1 s
1.5
regression
30% dups/sort . nub 10000 times
lower bound estimate upper bound
OLS regression 324 ms 326 ms 327 ms
R² goodness-of-fit 1.000 1.000 1.000
Mean execution time 326 ms 327 ms 328 ms
Standard deviation 44.1 μs 753 μs 966 μs

Outlying measurements have moderate (16.0%) effect on estimated standard deviation.

30% dups/nub . sort 10000

326
327
328
329
330
30% dups/nub . sort 10000 time densities
mean
2
3
4
5
1 iters
2
0 s
500 ms
1 s
1.5
regression
30% dups/nub . sort 10000 times
lower bound estimate upper bound
OLS regression 326 ms 329 ms 333 ms
R² goodness-of-fit 1.000 1.000 1.000
Mean execution time 327 ms 328 ms 329 ms
Standard deviation 902 μs 1.48 ms 2.21 ms

Outlying measurements have moderate (16.0%) effect on estimated standard deviation.

25% dups/sort . nub 10000

336
337
338
339
340
341
342
25% dups/sort . nub 10000 time densities
mean
1
2
2
3
3
4
4
0.5 iters
500
750
0 s
250 ms
1 s
1.25
1.5
regression
25% dups/sort . nub 10000 times
lower bound estimate upper bound
OLS regression 335 ms 342 ms 348 ms
R² goodness-of-fit 1.000 1.000 1.000
Mean execution time 337 ms 339 ms 341 ms
Standard deviation 1.19 ms 2.47 ms 3.37 ms

Outlying measurements have moderate (18.7%) effect on estimated standard deviation.

25% dups/nub . sort 10000

345
346
347
348
349
350
351
25% dups/nub . sort 10000 time densities
mean
1
2
2
3
3
4
4
0.5 iters
500
750
0 s
250 ms
1 s
1.25
1.5
regression
25% dups/nub . sort 10000 times
lower bound estimate upper bound
OLS regression 348 ms 352 ms 357 ms
R² goodness-of-fit 1.000 1.000 1.000
Mean execution time 346 ms 349 ms 351 ms
Standard deviation 268 μs 2.94 ms 3.64 ms

Outlying measurements have moderate (18.8%) effect on estimated standard deviation.

20% dups/sort . nub 10000

349
350
351
352
353
354
355
20% dups/sort . nub 10000 time densities
mean
1
2
2
3
3
4
4
0.5 iters
500
750
0 s
250 ms
1 s
1.25
1.5
regression
20% dups/sort . nub 10000 times
lower bound estimate upper bound
OLS regression 349 ms 357 ms 365 ms
R² goodness-of-fit 1.000 1.000 1.000
Mean execution time 349 ms 352 ms 354 ms
Standard deviation 2.68 ms 3.10 ms 3.19 ms

Outlying measurements have moderate (18.8%) effect on estimated standard deviation.

20% dups/nub . sort 10000

368
370
372
374
376
378
380
20% dups/nub . sort 10000 time densities
mean
1
2
2
3
3
4
4
0.5 iters
2
0 s
500 ms
1 s
1.5
regression
20% dups/nub . sort 10000 times
lower bound estimate upper bound
OLS regression 332 ms 368 ms 391 ms
R² goodness-of-fit 0.997 0.999 1.000
Mean execution time 369 ms 374 ms 377 ms
Standard deviation 2.73 ms 4.82 ms 6.80 ms

Outlying measurements have moderate (18.8%) effect on estimated standard deviation.

10% dups/sort . nub 10000

373
374
375
376
377
378
10% dups/sort . nub 10000 time densities
mean
1
2
2
3
3
4
4
0.5 iters
2
0 s
500 ms
1 s
1.5
regression
10% dups/sort . nub 10000 times
lower bound estimate upper bound
OLS regression 374 ms 379 ms 386 ms
R² goodness-of-fit 1.000 1.000 1.000
Mean execution time 374 ms 376 ms 377 ms
Standard deviation 968 μs 2.24 ms 3.09 ms

Outlying measurements have moderate (18.8%) effect on estimated standard deviation.

10% dups/nub . sort 10000

413
414
415
416
417
418
419
10% dups/nub . sort 10000 time densities
mean
1
2
2
3
3
4
4
0.5 iters
2
0 s
500 ms
1 s
1.5
regression
10% dups/nub . sort 10000 times
lower bound estimate upper bound
OLS regression 411 ms 421 ms 426 ms
R² goodness-of-fit 1.000 1.000 1.000
Mean execution time 414 ms 416 ms 418 ms
Standard deviation 1.49 ms 2.55 ms 3.42 ms

Outlying measurements have moderate (18.8%) effect on estimated standard deviation.

5% dups/sort . nub 10000

384
385
385
386
386
387
387
388
388
5% dups/sort . nub 10000 time densities
mean
1
2
2
3
3
4
4
0.5 iters
2
0 s
500 ms
1 s
1.5
regression
5% dups/sort . nub 10000 times
lower bound estimate upper bound
OLS regression 380 ms 384 ms 388 ms
R² goodness-of-fit 1.000 1.000 1.000
Mean execution time 385 ms 386 ms 387 ms
Standard deviation 606 μs 1.53 ms 2.08 ms

Outlying measurements have moderate (18.8%) effect on estimated standard deviation.

5% dups/nub . sort 10000

430
432
434
436
438
5% dups/nub . sort 10000 time densities
mean
1
2
2
3
3
4
4
0.5 iters
2
0 s
500 ms
1 s
1.5
regression
5% dups/nub . sort 10000 times
lower bound estimate upper bound
OLS regression 437 ms 441 ms 445 ms
R² goodness-of-fit 1.000 1.000 1.000
Mean execution time 432 ms 436 ms 438 ms
Standard deviation 1.11 ms 3.46 ms 4.67 ms

Outlying measurements have moderate (18.8%) effect on estimated standard deviation.

1% dups/sort . nub 10000

397
398
398
399
399
400
400
1% dups/sort . nub 10000 time densities
mean
1
2
2
3
3
4
4
0.5 iters
2
0 s
500 ms
1 s
1.5
regression
1% dups/sort . nub 10000 times
lower bound estimate upper bound
OLS regression 388 ms 397 ms 403 ms
R² goodness-of-fit 1.000 1.000 1.000
Mean execution time 397 ms 399 ms 400 ms
Standard deviation 1.28 ms 1.57 ms 1.77 ms

Outlying measurements have moderate (18.8%) effect on estimated standard deviation.

1% dups/nub . sort 10000

448
450
452
454
456
1% dups/nub . sort 10000 time densities
mean
1
2
2
3
3
4
4
0.5 iters
2
0 s
500 ms
1 s
1.5
regression
1% dups/nub . sort 10000 times
lower bound estimate upper bound
OLS regression 444 ms 456 ms 463 ms
R² goodness-of-fit 1.000 1.000 1.000
Mean execution time 450 ms 453 ms 455 ms
Standard deviation 1.52 ms 3.56 ms 4.93 ms

Outlying measurements have moderate (18.8%) effect on estimated standard deviation.

0% dups/sort . nub 10000

395
398
400
403
405
408
0% dups/sort . nub 10000 time densities
mean
1
2
2
3
3
4
4
0.5 iters
2
0 s
500 ms
1 s
1.5
regression
0% dups/sort . nub 10000 times
lower bound estimate upper bound
OLS regression 409 ms 411 ms 414 ms
R² goodness-of-fit 1.000 1.000 1.000
Mean execution time 396 ms 401 ms 405 ms
Standard deviation 1.56 ms 5.47 ms 7.33 ms

Outlying measurements have moderate (18.7%) effect on estimated standard deviation.

0% dups/nub . sort 10000

453
454
455
456
457
458
0% dups/nub . sort 10000 time densities
mean
1
2
2
3
3
4
4
0.5 iters
2
0 s
500 ms
1 s
1.5
regression
0% dups/nub . sort 10000 times
lower bound estimate upper bound
OLS regression 446 ms 457 ms 469 ms
R² goodness-of-fit 1.000 1.000 1.000
Mean execution time 453 ms 455 ms 457 ms
Standard deviation 915 μs 2.45 ms 3.11 ms

Outlying measurements have moderate (18.7%) effect on estimated standard deviation.

understanding this report

In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.

Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.

We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)

A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.