criterion performance measurements
overview
want to understand this report?
100% dups/sort . nub 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.506199407855747e-6 | 9.533344927306132e-6 | 9.579662770118266e-6 |
| Standard deviation | 7.788578317146581e-8 | 1.2059202684695953e-7 | 2.0507713264482737e-7 |
Outlying measurements have slight (8.601801801458439e-2%) effect on estimated standard deviation.
100% dups/nub . sort 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.489032158001608e-5 | 2.4916219073725594e-5 | 2.4951720773729413e-5 |
| Standard deviation | 7.587516904680986e-8 | 1.0300925156501195e-7 | 1.6546693802588553e-7 |
Outlying measurements have no (6.848989298454085e-3%) effect on estimated standard deviation.
99% dups/sort . nub 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.8030887209426587e-5 | 1.8093309668051653e-5 | 1.824506250568549e-5 |
| Standard deviation | 1.6590477388198947e-7 | 3.135905065293543e-7 | 5.878479506939903e-7 |
Outlying measurements have moderate (0.14348049806236976%) effect on estimated standard deviation.
99% dups/nub . sort 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.9188650463944022e-5 | 2.921066961318587e-5 | 2.9234115268857368e-5 |
| Standard deviation | 6.657638656176988e-8 | 8.052982846999279e-8 | 1.0148643623972312e-7 |
Outlying measurements have no (6.992660186470683e-3%) effect on estimated standard deviation.
95% dups/sort . nub 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.6523588416947566e-5 | 2.661845969896582e-5 | 2.6857863326220924e-5 |
| Standard deviation | 1.847498934435794e-7 | 4.5623261876976276e-7 | 7.620444041444078e-7 |
Outlying measurements have moderate (0.13176387494772002%) effect on estimated standard deviation.
95% dups/nub . sort 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.2482030912000734e-5 | 3.252075021777542e-5 | 3.2553216524572636e-5 |
| Standard deviation | 1.0197092368589854e-7 | 1.263674786949519e-7 | 1.7076716153301654e-7 |
Outlying measurements have no (7.091836734694001e-3%) effect on estimated standard deviation.
90% dups/sort . nub 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.17413891502177e-5 | 3.180371664955553e-5 | 3.1883810691853e-5 |
| Standard deviation | 1.8046666880290424e-7 | 2.205986430316174e-7 | 2.649908024298204e-7 |
Outlying measurements have no (7.091836734693916e-3%) effect on estimated standard deviation.
90% dups/nub . sort 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.696470893863702e-5 | 3.699213178455968e-5 | 3.7022622068490034e-5 |
| Standard deviation | 8.054635091499016e-8 | 9.688689536575464e-8 | 1.2001494638270888e-7 |
Outlying measurements have no (7.245990729394186e-3%) effect on estimated standard deviation.
80% dups/sort . nub 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.04828536843137e-5 | 4.05428582932666e-5 | 4.0626355139019674e-5 |
| Standard deviation | 1.5449011933043693e-7 | 2.4604286762337334e-7 | 3.6999780004998465e-7 |
Outlying measurements have no (7.352537722908018e-3%) effect on estimated standard deviation.
80% dups/nub . sort 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.2138898615746195e-5 | 4.2181899570731864e-5 | 4.2221564770040875e-5 |
| Standard deviation | 1.0623481050259668e-7 | 1.3271660040538678e-7 | 1.7470315739074468e-7 |
Outlying measurements have no (7.352537722908069e-3%) effect on estimated standard deviation.
75% dups/sort . nub 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.2662472772506926e-5 | 4.279025028777853e-5 | 4.3208661152095884e-5 |
| Standard deviation | 2.1543727969557192e-7 | 7.14044204897398e-7 | 1.4525828501667695e-6 |
Outlying measurements have moderate (0.12012593495442134%) effect on estimated standard deviation.
75% dups/nub . sort 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.3648739398882026e-5 | 4.370298311611201e-5 | 4.374972399096055e-5 |
| Standard deviation | 1.354961196215962e-7 | 1.7141509231679834e-7 | 2.3512646533195465e-7 |
Outlying measurements have no (7.406994876364477e-3%) effect on estimated standard deviation.
70% dups/sort . nub 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.830476751462465e-5 | 4.839869859308024e-5 | 4.8561998597691935e-5 |
| Standard deviation | 2.904605704423535e-7 | 3.981759045812392e-7 | 6.851629580560995e-7 |
Outlying measurements have no (7.518365472911122e-3%) effect on estimated standard deviation.
70% dups/nub . sort 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.725361470117372e-5 | 4.7339047572711336e-5 | 4.742183913969977e-5 |
| Standard deviation | 2.2859119024568983e-7 | 2.6834866483978077e-7 | 3.2452656767781524e-7 |
Outlying measurements have no (7.518365472910889e-3%) effect on estimated standard deviation.
60% dups/sort . nub 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.8090846405942324e-5 | 5.815362460072407e-5 | 5.822590793633837e-5 |
| Standard deviation | 1.711317721497855e-7 | 2.1542310875598075e-7 | 3.070014314628949e-7 |
Outlying measurements have no (7.75146484375e-3%) effect on estimated standard deviation.
60% dups/nub . sort 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.152795357322076e-5 | 5.1621792785660984e-5 | 5.1948603868191946e-5 |
| Standard deviation | 1.8961772652867517e-7 | 5.167777026785364e-7 | 1.006210351931962e-6 |
Outlying measurements have no (7.63313609467465e-3%) effect on estimated standard deviation.
50% dups/sort . nub 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.102448687777393e-5 | 7.114465719956556e-5 | 7.125803404750195e-5 |
| Standard deviation | 3.285187279481033e-7 | 3.8301121854728875e-7 | 4.4430261092728743e-7 |
Outlying measurements have no (7.99947970863684e-3%) effect on estimated standard deviation.
50% dups/nub . sort 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.216760532214396e-5 | 6.222325942067162e-5 | 6.227940080693357e-5 |
| Standard deviation | 1.5532010539987443e-7 | 1.861137446302061e-7 | 2.2477592752806935e-7 |
Outlying measurements have no (7.812015624031048e-3%) effect on estimated standard deviation.
40% dups/sort . nub 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.592734401081782e-5 | 7.602981963307924e-5 | 7.615531284586813e-5 |
| Standard deviation | 3.0319654179788726e-7 | 3.8880101573241405e-7 | 5.257668067557484e-7 |
Outlying measurements have no (8.063983078855362e-3%) effect on estimated standard deviation.
40% dups/nub . sort 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.402396858796955e-5 | 7.412936131880487e-5 | 7.42238806559826e-5 |
| Standard deviation | 2.8492387935633747e-7 | 3.527993078924831e-7 | 4.825494112981022e-7 |
Outlying measurements have no (8.063983078855032e-3%) effect on estimated standard deviation.
30% dups/sort . nub 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.464283171539283e-5 | 7.471294180123418e-5 | 7.477418090674481e-5 |
| Standard deviation | 1.9353250683471706e-7 | 2.3535952609191135e-7 | 2.89721754358187e-7 |
Outlying measurements have no (8.063983078855178e-3%) effect on estimated standard deviation.
30% dups/nub . sort 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.515535919779273e-5 | 8.527077872374879e-5 | 8.540465620056536e-5 |
| Standard deviation | 3.373041264566037e-7 | 4.093647003270449e-7 | 5.133163486578405e-7 |
Outlying measurements have no (8.263888888888913e-3%) effect on estimated standard deviation.
25% dups/sort . nub 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.027766528215185e-5 | 8.036210535860705e-5 | 8.043970269509145e-5 |
| Standard deviation | 2.2258559227889254e-7 | 2.7437252567356445e-7 | 3.436531281062275e-7 |
Outlying measurements have no (8.19616146438085e-3%) effect on estimated standard deviation.
25% dups/nub . sort 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.15107162445355e-5 | 7.158709341169248e-5 | 7.170859335625039e-5 |
| Standard deviation | 2.2082241758392982e-7 | 3.1584459149077336e-7 | 5.375995160796542e-7 |
Outlying measurements have no (7.999479708636681e-3%) effect on estimated standard deviation.
20% dups/sort . nub 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.005803322618797e-5 | 8.018410929162212e-5 | 8.032176307374042e-5 |
| Standard deviation | 3.6953886815020264e-7 | 4.348129365845977e-7 | 5.163452428872806e-7 |
Outlying measurements have no (8.196161464380807e-3%) effect on estimated standard deviation.
20% dups/nub . sort 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.96542530100877e-5 | 7.979800819092779e-5 | 7.999467312694976e-5 |
| Standard deviation | 4.218147579538677e-7 | 5.309057089146472e-7 | 7.09259625242031e-7 |
Outlying measurements have no (8.129535071217402e-3%) effect on estimated standard deviation.
10% dups/sort . nub 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.930276302800664e-5 | 8.940982089528513e-5 | 8.954014738404084e-5 |
| Standard deviation | 3.2576288901769385e-7 | 3.846152155617525e-7 | 4.5407114777300705e-7 |
Outlying measurements have no (8.332744862650959e-3%) effect on estimated standard deviation.
10% dups/nub . sort 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.841800433250586e-5 | 8.853217618245628e-5 | 8.866277553397142e-5 |
| Standard deviation | 3.37486749554895e-7 | 4.246153835034374e-7 | 5.613945730376566e-7 |
Outlying measurements have no (8.332744862650916e-3%) effect on estimated standard deviation.
5% dups/sort . nub 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.239877106365282e-5 | 9.254938810549572e-5 | 9.276962709483772e-5 |
| Standard deviation | 4.0862532527717486e-7 | 6.005970959930519e-7 | 1.0106876604174564e-6 |
Outlying measurements have no (8.332744862650893e-3%) effect on estimated standard deviation.
5% dups/nub . sort 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.678598069816114e-5 | 8.689954684575846e-5 | 8.704669962043269e-5 |
| Standard deviation | 3.3825842407494205e-7 | 4.5059541241966857e-7 | 6.849354552326556e-7 |
Outlying measurements have no (8.263888888888762e-3%) effect on estimated standard deviation.
1% dups/sort . nub 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.28580796851969e-5 | 9.297222816319116e-5 | 9.315508486314834e-5 |
| Standard deviation | 3.3088066141055705e-7 | 4.455945401568461e-7 | 6.935709845378347e-7 |
Outlying measurements have no (8.402757828210329e-3%) effect on estimated standard deviation.
1% dups/nub . sort 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.487278289563909e-5 | 9.497009693101598e-5 | 9.508428379178751e-5 |
| Standard deviation | 3.0172321892364983e-7 | 3.5873172750208836e-7 | 4.5173705619257877e-7 |
Outlying measurements have no (8.402757828210285e-3%) effect on estimated standard deviation.
0% dups/sort . nub 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.441329405973054e-5 | 9.449906674280637e-5 | 9.458642232920148e-5 |
| Standard deviation | 2.5007274657162566e-7 | 3.003239994561301e-7 | 3.7626946803274343e-7 |
Outlying measurements have no (8.402757828210263e-3%) effect on estimated standard deviation.
0% dups/nub . sort 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.40447789179181e-5 | 9.414654288530164e-5 | 9.425830218293692e-5 |
| Standard deviation | 2.9557388571175623e-7 | 3.588265393830428e-7 | 4.463905671485504e-7 |
Outlying measurements have no (8.402757828210284e-3%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.