criterion performance measurements
overview
want to understand this report?
sort . nub/1000(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.8030607079905625e-3 | 5.847673332757955e-3 | 5.913985054954675e-3 |
| Standard deviation | 8.11496599366019e-5 | 1.1275551075131968e-4 | 1.5265216591340346e-4 |
Outlying measurements have slight (3.84e-2%) effect on estimated standard deviation.
sort . nub/1000(1->100000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.882844672959634e-3 | 5.915395829097553e-3 | 5.955056738708926e-3 |
| Standard deviation | 6.88633605930516e-5 | 8.703878064994451e-5 | 1.071252045503089e-4 |
Outlying measurements have slight (3.8400000000000004e-2%) effect on estimated standard deviation.
sort . nub/1000(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.606319150164333e-3 | 5.633493880724761e-3 | 5.680448988654709e-3 |
| Standard deviation | 5.551491611618823e-5 | 7.785103024631084e-5 | 1.1625549671053169e-4 |
Outlying measurements have slight (3.698224852071005e-2%) effect on estimated standard deviation.
sort . nub/1000(1->5000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.012626655873147e-3 | 5.041671910100225e-3 | 5.091472289904934e-3 |
| Standard deviation | 6.844983824138137e-5 | 8.301964824013525e-5 | 1.0486671227294366e-4 |
Outlying measurements have slight (3.5665294924554176e-2%) effect on estimated standard deviation.
sort . nub/1000(1->2000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.586916227884006e-3 | 4.600228755482725e-3 | 4.612805786741889e-3 |
| Standard deviation | 2.598188973102432e-5 | 3.464893228406241e-5 | 4.474398181136793e-5 |
Outlying measurements have slight (3.4438775510204044e-2%) effect on estimated standard deviation.
sort . nub/1000(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.21677278146656e-3 | 4.2412959379400485e-3 | 4.281532405426029e-3 |
| Standard deviation | 7.668916672386469e-5 | 9.29462911175188e-5 | 1.1453608695880062e-4 |
Outlying measurements have slight (3.329369797859706e-2%) effect on estimated standard deviation.
sort . nub/1000(1->500)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.4480666260661456e-3 | 2.4622613810271333e-3 | 2.4795101353515822e-3 |
| Standard deviation | 3.719457091291283e-5 | 4.730607193932097e-5 | 6.160377566837977e-5 |
Outlying measurements have slight (2.562326869806057e-2%) effect on estimated standard deviation.
sort . nub/1000(1->100)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.994953752808882e-4 | 6.026314720454706e-4 | 6.054824526740607e-4 |
| Standard deviation | 6.1523122492115535e-6 | 7.76603645129974e-6 | 1.0447535945741747e-5 |
Outlying measurements have slight (1.562106324011115e-2%) effect on estimated standard deviation.
sort . nub/1000(1->50)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.1468710202542887e-4 | 3.1668549044166214e-4 | 3.1893104864750036e-4 |
| Standard deviation | 4.46156511511019e-6 | 5.645504742846971e-6 | 6.783715899783288e-6 |
Outlying measurements have slight (8.762271553914013e-2%) effect on estimated standard deviation.
sort . nub/1000(1->20)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.388773539614713e-4 | 1.4028018793387832e-4 | 1.4225727007920414e-4 |
| Standard deviation | 2.315146256488421e-6 | 3.770911077473782e-6 | 5.773897292977209e-6 |
Outlying measurements have moderate (0.19649721325343658%) effect on estimated standard deviation.
sort . nub/1000(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.796389922367095e-5 | 7.835348641473416e-5 | 7.897151647392277e-5 |
| Standard deviation | 1.0264561132625791e-6 | 1.1796451647784858e-6 | 1.4903472955480065e-6 |
Outlying measurements have slight (8.226880152433973e-2%) effect on estimated standard deviation.
nub . sort/1000(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.788381547983991e-3 | 5.819456519252726e-3 | 5.85824366300011e-3 |
| Standard deviation | 6.65378187078395e-5 | 1.0123525586767025e-4 | 1.425197885115305e-4 |
Outlying measurements have slight (3.8400000000000004e-2%) effect on estimated standard deviation.
nub . sort/1000(1->100000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.796676555573845e-3 | 5.827882644661667e-3 | 5.865854878162755e-3 |
| Standard deviation | 5.646583310637199e-5 | 8.028511031770332e-5 | 1.0219662332103066e-4 |
Outlying measurements have slight (3.84e-2%) effect on estimated standard deviation.
nub . sort/1000(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.051071799294266e-3 | 5.079119049849748e-3 | 5.1038459494403806e-3 |
| Standard deviation | 4.838723480257865e-5 | 5.7062544755513116e-5 | 7.531451243809564e-5 |
Outlying measurements have slight (3.566529492455405e-2%) effect on estimated standard deviation.
nub . sort/1000(1->5000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.829818744422628e-3 | 4.854358870091492e-3 | 4.874841348497941e-3 |
| Standard deviation | 4.541473717719111e-5 | 5.551500575935908e-5 | 7.284690797411011e-5 |
Outlying measurements have slight (3.566529492455418e-2%) effect on estimated standard deviation.
nub . sort/1000(1->2000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.6349545376499442e-3 | 3.6461283902815907e-3 | 3.6654172687313907e-3 |
| Standard deviation | 2.4256930987202233e-5 | 3.640606442237857e-5 | 5.388224281793376e-5 |
Outlying measurements have slight (3.02734375e-2%) effect on estimated standard deviation.
nub . sort/1000(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.150477999935822e-3 | 3.1579529906010117e-3 | 3.1715073205344523e-3 |
| Standard deviation | 1.715128667759864e-5 | 2.7468511539001738e-5 | 3.856053626331965e-5 |
Outlying measurements have slight (2.8546712802768166e-2%) effect on estimated standard deviation.
nub . sort/1000(1->500)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.4891426626195635e-3 | 1.494063269928757e-3 | 1.5031624848052282e-3 |
| Standard deviation | 1.0796812249969468e-5 | 1.3652003472657426e-5 | 2.0749913219070965e-5 |
Outlying measurements have slight (2.126654064272212e-2%) effect on estimated standard deviation.
nub . sort/1000(1->100)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.3150077727662953e-4 | 4.3473340558199906e-4 | 4.4073820061236996e-4 |
| Standard deviation | 7.94163280396863e-6 | 1.0028422655039476e-5 | 1.2758310497716221e-5 |
Outlying measurements have moderate (0.12319478303389135%) effect on estimated standard deviation.
nub . sort/1000(1->50)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.750344123194462e-4 | 3.760862091565599e-4 | 3.7724331680023723e-4 |
| Standard deviation | 2.3637568179607485e-6 | 2.9491211981389184e-6 | 3.751624153639552e-6 |
Outlying measurements have slight (1.3695987654320873e-2%) effect on estimated standard deviation.
nub . sort/1000(1->20)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.422960220302701e-4 | 3.4314966533110315e-4 | 3.451879937302772e-4 |
| Standard deviation | 2.0258858566281917e-6 | 2.7609850422231975e-6 | 3.4463661747766217e-6 |
Outlying measurements have slight (1.3330898466033555e-2%) effect on estimated standard deviation.
nub . sort/1000(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.2648401813008645e-4 | 3.275173421509213e-4 | 3.300686584576381e-4 |
| Standard deviation | 3.0066146874170707e-6 | 4.121351295649877e-6 | 5.920471461173899e-6 |
Outlying measurements have slight (1.3155555555555793e-2%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.