criterion performance measurements
overview
want to understand this report?
100% dups/sort . nub 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.709139516281723e-5 | 8.729125230127311e-5 | 8.751431291191765e-5 |
| Standard deviation | 6.191549929632812e-7 | 7.159766004562372e-7 | 8.589027844424205e-7 |
Outlying measurements have no (8.263888888888918e-3%) effect on estimated standard deviation.
100% dups/nub . sort 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.263574015783523e-4 | 3.267099464089787e-4 | 3.2752596820612606e-4 |
| Standard deviation | 8.716668340132161e-7 | 1.697343770018259e-6 | 3.407946085219074e-6 |
Outlying measurements have slight (1.0637067869117743e-2%) effect on estimated standard deviation.
99% dups/sort . nub 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.3705507189064083e-3 | 1.3726924776604778e-3 | 1.3745178825551702e-3 |
| Standard deviation | 5.191575924082461e-6 | 6.680036930299542e-6 | 8.85715910224543e-6 |
Outlying measurements have slight (1.514792899408277e-2%) effect on estimated standard deviation.
99% dups/nub . sort 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.802636417650225e-4 | 6.809330684748058e-4 | 6.816657668295868e-4 |
| Standard deviation | 1.9003928784104976e-6 | 2.3649162388213397e-6 | 3.105098999311087e-6 |
Outlying measurements have slight (1.2656147271531887e-2%) effect on estimated standard deviation.
95% dups/sort . nub 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.993573700632426e-3 | 1.9976638486346415e-3 | 2.003494900874798e-3 |
| Standard deviation | 1.3908732116922468e-5 | 1.6633438099974404e-5 | 2.1106067280788563e-5 |
Outlying measurements have slight (1.7236072637734686e-2%) effect on estimated standard deviation.
95% dups/nub . sort 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.00600928935038e-3 | 1.007987820391971e-3 | 1.0104075527514996e-3 |
| Standard deviation | 5.883162938995957e-6 | 7.652647779656247e-6 | 1.155766071430393e-5 |
Outlying measurements have slight (1.388613370363026e-2%) effect on estimated standard deviation.
90% dups/sort . nub 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.4893080949112355e-3 | 2.495645307301316e-3 | 2.5034699112563654e-3 |
| Standard deviation | 1.7355593284829975e-5 | 2.2299769202917542e-5 | 2.9924230828005707e-5 |
Outlying measurements have slight (1.851192595229619e-2%) effect on estimated standard deviation.
90% dups/nub . sort 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.3655017603591798e-3 | 1.3678778930299265e-3 | 1.3727914266647133e-3 |
| Standard deviation | 6.991679686691142e-6 | 1.077612190540742e-5 | 1.924305310326381e-5 |
Outlying measurements have slight (1.5147928994082762e-2%) effect on estimated standard deviation.
80% dups/sort . nub 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.1468249524360024e-3 | 3.1626973369541954e-3 | 3.184557404474011e-3 |
| Standard deviation | 4.777268347332688e-5 | 6.024101532493979e-5 | 7.28867826720394e-5 |
Outlying measurements have slight (7.577555536185404e-2%) effect on estimated standard deviation.
80% dups/nub . sort 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.9406746515519466e-3 | 1.944258117180299e-3 | 1.9494064654815059e-3 |
| Standard deviation | 1.1225835747214432e-5 | 1.4383816950362008e-5 | 2.0197076566044144e-5 |
Outlying measurements have slight (1.694411414982164e-2%) effect on estimated standard deviation.
75% dups/sort . nub 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.399403947868971e-3 | 3.40713043364518e-3 | 3.4142406534391203e-3 |
| Standard deviation | 2.1351947434558788e-5 | 2.5558035553542293e-5 | 3.236145419028371e-5 |
Outlying measurements have slight (2.039930555555542e-2%) effect on estimated standard deviation.
75% dups/nub . sort 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.1671755169907687e-3 | 2.1728454346559532e-3 | 2.1806955526681945e-3 |
| Standard deviation | 1.867061554890569e-5 | 2.3496975869824836e-5 | 3.07629351163998e-5 |
Outlying measurements have slight (1.753826530612234e-2%) effect on estimated standard deviation.
70% dups/sort . nub 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.643154262667756e-3 | 3.657026035925171e-3 | 3.671900341103555e-3 |
| Standard deviation | 3.890602283062321e-5 | 4.546136938645117e-5 | 5.527311104774123e-5 |
Outlying measurements have slight (2.1266540642721984e-2%) effect on estimated standard deviation.
70% dups/nub . sort 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.373877198542995e-3 | 2.3789333147411227e-3 | 2.3838507816065163e-3 |
| Standard deviation | 1.3423157329520634e-5 | 1.6513805120842472e-5 | 2.0683859571818507e-5 |
Outlying measurements have slight (1.8175582990397805e-2%) effect on estimated standard deviation.
60% dups/sort . nub 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.92876622221414e-3 | 3.948350416100217e-3 | 3.965031941491767e-3 |
| Standard deviation | 4.028329031708148e-5 | 5.53772325408171e-5 | 8.42144257288484e-5 |
Outlying measurements have slight (2.172839506172867e-2%) effect on estimated standard deviation.
60% dups/nub . sort 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.804571926284837e-3 | 2.812220594300142e-3 | 2.819423502883415e-3 |
| Standard deviation | 2.1264469243237123e-5 | 2.5274756242123484e-5 | 3.09167919216234e-5 |
Outlying measurements have slight (1.9223375624759707e-2%) effect on estimated standard deviation.
50% dups/sort . nub 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.5744068097482975e-3 | 4.593672220439813e-3 | 4.621053696647679e-3 |
| Standard deviation | 5.853029304975827e-5 | 7.340667275041105e-5 | 9.256448940588686e-5 |
Outlying measurements have slight (2.3242630385487555e-2%) effect on estimated standard deviation.
50% dups/nub . sort 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.25054749444719e-3 | 3.261807403550279e-3 | 3.2732683903103047e-3 |
| Standard deviation | 2.929742630169291e-5 | 3.685184486259769e-5 | 4.8469500991866813e-5 |
Outlying measurements have slight (2.039930555555548e-2%) effect on estimated standard deviation.
40% dups/sort . nub 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.995636169936953e-3 | 5.033094375689797e-3 | 5.089904633406109e-3 |
| Standard deviation | 1.0516204410242017e-4 | 1.3874987560804996e-4 | 2.0965893211065406e-4 |
Outlying measurements have moderate (0.11449337357524733%) effect on estimated standard deviation.
40% dups/nub . sort 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.0041484175140675e-3 | 4.022406510063196e-3 | 4.044168458789761e-3 |
| Standard deviation | 4.9883363859874885e-5 | 6.289999724504343e-5 | 8.380443752898207e-5 |
Outlying measurements have slight (2.1728395061728568e-2%) effect on estimated standard deviation.
30% dups/sort . nub 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.3117552210513825e-3 | 5.335187758340923e-3 | 5.3638959477850575e-3 |
| Standard deviation | 7.019360782240822e-5 | 8.280935610574041e-5 | 1.0418406980504766e-4 |
Outlying measurements have slight (2.4375000000000185e-2%) effect on estimated standard deviation.
30% dups/nub . sort 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.588190735207057e-3 | 4.612134105505917e-3 | 4.632909674284537e-3 |
| Standard deviation | 5.566936461390218e-5 | 6.721211071414107e-5 | 8.085673413803723e-5 |
Outlying measurements have slight (2.324263038548764e-2%) effect on estimated standard deviation.
25% dups/sort . nub 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.589697219721926e-3 | 5.612125092782699e-3 | 5.631035085581516e-3 |
| Standard deviation | 5.166328951069497e-5 | 6.360985470631922e-5 | 9.050896257493368e-5 |
Outlying measurements have slight (2.498356344510188e-2%) effect on estimated standard deviation.
25% dups/nub . sort 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.803047841585502e-3 | 4.8181216421449595e-3 | 4.837416434096096e-3 |
| Standard deviation | 3.7905212994883074e-5 | 5.423952389139938e-5 | 7.477541404298587e-5 |
Outlying measurements have slight (2.3242630385487382e-2%) effect on estimated standard deviation.
20% dups/sort . nub 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.675610454672006e-3 | 5.715542399953325e-3 | 5.756187947545737e-3 |
| Standard deviation | 1.0420237339004716e-4 | 1.2406854872505432e-4 | 1.4992214537078998e-4 |
Outlying measurements have slight (7.189200863962177e-2%) effect on estimated standard deviation.
20% dups/nub . sort 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.015593606304137e-3 | 5.037534953023713e-3 | 5.069989141156842e-3 |
| Standard deviation | 6.0493280840393474e-5 | 7.928520709023255e-5 | 1.1763562757029197e-4 |
Outlying measurements have slight (2.3795359904818607e-2%) effect on estimated standard deviation.
10% dups/sort . nub 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.1396399058308414e-3 | 6.191311439561789e-3 | 6.2685711497890454e-3 |
| Standard deviation | 1.3602329582507655e-4 | 1.844123706880661e-4 | 2.5523997585085925e-4 |
Outlying measurements have moderate (0.12355488224925985%) effect on estimated standard deviation.
10% dups/nub . sort 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.772645274941049e-3 | 5.801073458931043e-3 | 5.839448900667828e-3 |
| Standard deviation | 7.878261893706728e-5 | 9.870981253725712e-5 | 1.224250609594203e-4 |
Outlying measurements have slight (2.4983563445101952e-2%) effect on estimated standard deviation.
5% dups/sort . nub 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.559219365940528e-3 | 6.59591642472227e-3 | 6.623842805485694e-3 |
| Standard deviation | 6.344937169201397e-5 | 8.94225864284955e-5 | 1.2753620284150253e-4 |
Outlying measurements have slight (2.700617283950606e-2%) effect on estimated standard deviation.
5% dups/nub . sort 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.956689120446498e-3 | 6.014319430605044e-3 | 6.108256416775385e-3 |
| Standard deviation | 1.3867984281991407e-4 | 2.1454638977711442e-4 | 3.2915895754668373e-4 |
Outlying measurements have moderate (0.14983905517610793%) effect on estimated standard deviation.
1% dups/sort . nub 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.5756405903126544e-3 | 6.6054883708679315e-3 | 6.641706870405213e-3 |
| Standard deviation | 7.049949998263882e-5 | 9.595726092631596e-5 | 1.44383946371016e-4 |
Outlying measurements have slight (2.7006172839506293e-2%) effect on estimated standard deviation.
1% dups/nub . sort 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.16338010301321e-3 | 6.188515540625973e-3 | 6.225016901723535e-3 |
| Standard deviation | 7.141527042532965e-5 | 9.356189002082396e-5 | 1.423280686541034e-4 |
Outlying measurements have slight (2.5623268698060857e-2%) effect on estimated standard deviation.
0% dups/sort . nub 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.4801950030019305e-3 | 6.514590416431889e-3 | 6.5561556282223935e-3 |
| Standard deviation | 8.446987681995031e-5 | 1.1546760484648343e-4 | 1.5253300957243431e-4 |
Outlying measurements have slight (2.6296566837107162e-2%) effect on estimated standard deviation.
0% dups/nub . sort 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.171925962116612e-3 | 6.208315484090736e-3 | 6.269910154603805e-3 |
| Standard deviation | 8.364973804137252e-5 | 1.3873332998967416e-4 | 2.1651249123221276e-4 |
Outlying measurements have slight (7.381523804758039e-2%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.