criterion performance measurements
overview
want to understand this report?
100% dups/sort . nub 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.495756522820152e-3 | 6.530205790166014e-3 | 6.576393076706263e-3 |
| Standard deviation | 8.830836133527619e-5 | 1.1517586102164174e-4 | 1.5668837967432948e-4 |
Outlying measurements have slight (2.62965668371073e-2%) effect on estimated standard deviation.
100% dups/nub . sort 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.552997148485894e-3 | 6.5592695171571135e-3 | 6.56588198401969e-3 |
| Standard deviation | 1.5310387088452472e-5 | 1.8833509463995673e-5 | 2.4650562470385175e-5 |
Outlying measurements have slight (2.629656683710737e-2%) effect on estimated standard deviation.
99% dups/sort . nub 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.11604083524372252 | 0.11882082062930598 | 0.12392330559374187 |
| Standard deviation | 2.451035690575174e-3 | 5.831446117118993e-3 | 9.354306525327058e-3 |
Outlying measurements have moderate (0.11232982463627844%) effect on estimated standard deviation.
99% dups/nub . sort 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.5253607045761642e-2 | 2.5341466483710686e-2 | 2.544714857283164e-2 |
| Standard deviation | 1.7255450094617221e-4 | 2.1081229557355397e-4 | 2.545460795803395e-4 |
Outlying measurements have slight (4.986149584487535e-2%) effect on estimated standard deviation.
95% dups/sort . nub 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.15128337655238377 | 0.15714901469520895 | 0.17003410302372354 |
| Standard deviation | 3.7485191194458357e-3 | 1.2306456621344318e-2 | 1.904786393713292e-2 |
Outlying measurements have moderate (0.1354289498357612%) effect on estimated standard deviation.
95% dups/nub . sort 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.751603920332395e-2 | 4.764844994812058e-2 | 4.7816558919761824e-2 |
| Standard deviation | 2.0218437233974354e-4 | 2.9063564289127084e-4 | 4.1717211980407665e-4 |
Outlying measurements have slight (7.100591715976329e-2%) effect on estimated standard deviation.
90% dups/sort . nub 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.174687781702366 | 0.18427782701528275 | 0.20586213962238634 |
| Standard deviation | 8.686270005902302e-3 | 2.0245306035942393e-2 | 2.9723613457278655e-2 |
Outlying measurements have moderate (0.2706175389567998%) effect on estimated standard deviation.
90% dups/nub . sort 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.88667974159757e-2 | 6.910618883273953e-2 | 6.942319426250644e-2 |
| Standard deviation | 3.2855869580750244e-4 | 4.992217293341379e-4 | 7.649255209697747e-4 |
Outlying measurements have slight (8.264462809917347e-2%) effect on estimated standard deviation.
80% dups/sort . nub 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.21594766216110328 | 0.23007977141387084 | 0.26144257855552017 |
| Standard deviation | 8.84621558153241e-3 | 2.856425384352476e-2 | 4.068267518815685e-2 |
Outlying measurements have moderate (0.31456017370952705%) effect on estimated standard deviation.
80% dups/nub . sort 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.10654584860608031 | 0.107067904249589 | 0.1078815958190425 |
| Standard deviation | 6.172880022874042e-4 | 1.0072856391568023e-3 | 1.5192792493753061e-3 |
Outlying measurements have slight (9.87654320987653e-2%) effect on estimated standard deviation.
75% dups/sort . nub 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.2528648201333256 | 0.2638832522332958 | 0.27692488229992024 |
| Standard deviation | 7.209585582656658e-3 | 1.550865701496188e-2 | 2.2291630009865324e-2 |
Outlying measurements have moderate (0.16381027473811283%) effect on estimated standard deviation.
75% dups/nub . sort 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.12362367263540837 | 0.12385751342754002 | 0.12457172231850948 |
| Standard deviation | 1.9100551019964767e-4 | 5.658876070015247e-4 | 8.87641428882017e-4 |
Outlying measurements have moderate (0.109375%) effect on estimated standard deviation.
70% dups/sort . nub 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.24403939532000551 | 0.25804153828666965 | 0.28292802179998944 |
| Standard deviation | 6.752229949424971e-3 | 2.3263191258895265e-2 | 3.18396300846753e-2 |
Outlying measurements have moderate (0.18462020045919433%) effect on estimated standard deviation.
70% dups/nub . sort 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.14271379806801993 | 0.14452118090135985 | 0.14710585175511134 |
| Standard deviation | 2.507047574628366e-3 | 3.1495605425568346e-3 | 3.809846377023347e-3 |
Outlying measurements have moderate (0.12244897959183673%) effect on estimated standard deviation.
60% dups/sort . nub 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.28689641539999683 | 0.2993704454166618 | 0.3198520100666428 |
| Standard deviation | 6.1498908858324715e-3 | 2.1785893497781428e-2 | 3.085702381324064e-2 |
Outlying measurements have moderate (0.17639646188390917%) effect on estimated standard deviation.
60% dups/nub . sort 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.18886427879178946 | 0.19728303189169574 | 0.20078499218332505 |
| Standard deviation | 1.5689108033020982e-3 | 7.2687201894642995e-3 | 1.0531555987435402e-2 |
Outlying measurements have moderate (0.1388888888888887%) effect on estimated standard deviation.
50% dups/sort . nub 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.29674487196663446 | 0.30668664618331376 | 0.3252277476000017 |
| Standard deviation | 9.649353772448015e-4 | 1.8289056095104676e-2 | 2.3550227434062522e-2 |
Outlying measurements have moderate (0.16485066105880508%) effect on estimated standard deviation.
50% dups/nub . sort 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.22241111389996981 | 0.22643166379720203 | 0.23049438272221323 |
| Standard deviation | 4.4866079738961825e-3 | 5.772930705382345e-3 | 7.189631905884572e-3 |
Outlying measurements have moderate (0.13888888888888887%) effect on estimated standard deviation.
40% dups/sort . nub 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.33451752862504236 | 0.3427336323958154 | 0.35498783658321525 |
| Standard deviation | 1.254030621143893e-3 | 1.1587728763433145e-2 | 1.4776951483171848e-2 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
40% dups/nub . sort 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.256916516173369 | 0.25795234677334067 | 0.25874961239999267 |
| Standard deviation | 8.241215711146913e-4 | 1.1076639746704657e-3 | 1.3751044752224719e-3 |
Outlying measurements have moderate (0.16%) effect on estimated standard deviation.
30% dups/sort . nub 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.3768826481249903 | 0.3803497922083352 | 0.382686336166671 |
| Standard deviation | 2.070929166620772e-3 | 3.6806384882983024e-3 | 5.199164651497965e-3 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
30% dups/nub . sort 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.2933924408000015 | 0.2968932218399944 | 0.3004765207199943 |
| Standard deviation | 3.578272936862454e-3 | 4.802895132884522e-3 | 6.13022325626411e-3 |
Outlying measurements have moderate (0.16%) effect on estimated standard deviation.
25% dups/sort . nub 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.36470523254164294 | 0.374632454666686 | 0.3905446133751127 |
| Standard deviation | 9.954959166880994e-4 | 1.5028594314976829e-2 | 1.8948528062873456e-2 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
25% dups/nub . sort 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.3145459763832643 | 0.3163054921766396 | 0.3193821917799733 |
| Standard deviation | 1.019175115194216e-3 | 3.0329624608057956e-3 | 4.397653350273793e-3 |
Outlying measurements have moderate (0.16%) effect on estimated standard deviation.
20% dups/sort . nub 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.37441342620836815 | 0.3844799637291724 | 0.40105057762499996 |
| Standard deviation | 4.203053458345124e-3 | 1.6115169906184067e-2 | 2.156073083238072e-2 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
20% dups/nub . sort 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.33547367393748573 | 0.34664766718748297 | 0.3549086057499835 |
| Standard deviation | 5.911147903590027e-3 | 1.144948367015708e-2 | 1.585818997341295e-2 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
10% dups/sort . nub 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.40919100206248515 | 0.4160215574375125 | 0.4228733930000696 |
| Standard deviation | 3.666455750078512e-3 | 8.496739036642347e-3 | 1.1589977997382175e-2 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
10% dups/nub . sort 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.36987991837500545 | 0.37115815254168183 | 0.3733964901250033 |
| Standard deviation | 2.9322824997279895e-4 | 2.145636737891348e-3 | 2.7070581953145345e-3 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
5% dups/sort . nub 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.41072165041666153 | 0.4204379942083136 | 0.4350247464999484 |
| Standard deviation | 3.7387552499978938e-3 | 1.3870870671078266e-2 | 1.812179211455632e-2 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
5% dups/nub . sort 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.3875979013750168 | 0.3878699976458364 | 0.3882539329374737 |
| Standard deviation | 8.392320833650047e-5 | 4.013225221454599e-4 | 5.314732469640595e-4 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
1% dups/sort . nub 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.4277861582708387 | 0.4386208597708266 | 0.4471792392499765 |
| Standard deviation | 4.552644041676018e-3 | 1.0661114530305322e-2 | 1.3554987329055563e-2 |
Outlying measurements have moderate (0.18749999999999994%) effect on estimated standard deviation.
1% dups/nub . sort 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.40562339424998584 | 0.4064671198333514 | 0.4073284802500581 |
| Standard deviation | 2.9858066667998173e-4 | 9.787815177875047e-4 | 1.3125758148442284e-3 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
0% dups/sort . nub 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.4293955011249911 | 0.444758853125028 | 0.45883364250005343 |
| Standard deviation | 8.361886499983484e-3 | 1.775316145170711e-2 | 2.2567808247451134e-2 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
0% dups/nub . sort 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.4096028806875296 | 0.4167483400208359 | 0.4237708096666589 |
| Standard deviation | 5.3551187916790755e-3 | 8.494346015959863e-3 | 1.0034140409730982e-2 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.