criterion performance measurements
overview
want to understand this report?
100% dups/sort . nub 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.1846673488369543e-5 | 5.1955115406381845e-5 | 5.2061557747401875e-5 |
| Standard deviation | 2.9991646178475403e-7 | 3.635531429170844e-7 | 4.5156132657263837e-7 |
Outlying measurements have no (7.633136094674724e-3%) effect on estimated standard deviation.
100% dups/nub . sort 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.735847659301642e-4 | 2.741124947548288e-4 | 2.7485082556186213e-4 |
| Standard deviation | 1.5630867788484867e-6 | 2.05575317071832e-6 | 2.809239498459923e-6 |
Outlying measurements have slight (1.0308159722222137e-2%) effect on estimated standard deviation.
99% dups/sort . nub 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.221053157340888e-4 | 8.229932032071981e-4 | 8.241047823357662e-4 |
| Standard deviation | 2.709666115196165e-6 | 3.3934741781392894e-6 | 4.948032906156776e-6 |
Outlying measurements have slight (1.3330898466033587e-2%) effect on estimated standard deviation.
99% dups/nub . sort 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.483815404350628e-4 | 5.492934302618322e-4 | 5.511655277403761e-4 |
| Standard deviation | 2.576192772261684e-6 | 4.3826023710605345e-6 | 8.775043996370364e-6 |
Outlying measurements have slight (1.190303382203518e-2%) effect on estimated standard deviation.
95% dups/sort . nub 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.265097962399076e-3 | 1.2666050328703552e-3 | 1.2712154905793547e-3 |
| Standard deviation | 3.2499741088292884e-6 | 8.004516947554153e-6 | 1.69030967128391e-5 |
Outlying measurements have slight (1.492194674012856e-2%) effect on estimated standard deviation.
95% dups/nub . sort 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.076586785526663e-4 | 8.086150926952698e-4 | 8.097737258425246e-4 |
| Standard deviation | 2.776971546148492e-6 | 3.6257027519365476e-6 | 5.267778685133948e-6 |
Outlying measurements have slight (1.3155555555555556e-2%) effect on estimated standard deviation.
90% dups/sort . nub 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.4682554683186561e-3 | 1.4700822797642436e-3 | 1.4723261466917836e-3 |
| Standard deviation | 5.77990536001612e-6 | 6.8783873394752736e-6 | 8.311105257778285e-6 |
Outlying measurements have slight (1.5621063240110858e-2%) effect on estimated standard deviation.
90% dups/nub . sort 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.0333705097576409e-3 | 1.0351571692264105e-3 | 1.037046561454947e-3 |
| Standard deviation | 5.138623238063956e-6 | 6.27518557306157e-6 | 8.388027836503652e-6 |
Outlying measurements have slight (1.4081632653061225e-2%) effect on estimated standard deviation.
80% dups/sort . nub 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.9544593511227446e-3 | 1.9595253810031866e-3 | 1.986685125954167e-3 |
| Standard deviation | 7.109095469721591e-6 | 2.983272952259414e-5 | 7.062377613972369e-5 |
Outlying measurements have slight (1.69441141498216e-2%) effect on estimated standard deviation.
80% dups/nub . sort 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.4790030960061978e-3 | 1.480393513537747e-3 | 1.4819395021824098e-3 |
| Standard deviation | 4.459559095072825e-6 | 5.261108102306972e-6 | 6.973737405607548e-6 |
Outlying measurements have slight (1.562106324011086e-2%) effect on estimated standard deviation.
75% dups/sort . nub 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.1063843706450197e-3 | 2.1083250051752386e-3 | 2.1110608138761484e-3 |
| Standard deviation | 5.696363254675608e-6 | 7.6060600368838986e-6 | 1.0427657823120928e-5 |
Outlying measurements have slight (1.7538265306122326e-2%) effect on estimated standard deviation.
75% dups/nub . sort 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.6511121935169843e-3 | 1.65411403820713e-3 | 1.6570184445031228e-3 |
| Standard deviation | 8.625934653609475e-6 | 1.0570432209394402e-5 | 1.373408102989234e-5 |
Outlying measurements have slight (1.6124697661918742e-2%) effect on estimated standard deviation.
70% dups/sort . nub 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.2438410798641803e-3 | 2.2460240746936264e-3 | 2.248264935319031e-3 |
| Standard deviation | 6.273409242398431e-6 | 7.419349457822353e-6 | 9.099219592297674e-6 |
Outlying measurements have slight (1.7851239669421204e-2%) effect on estimated standard deviation.
70% dups/nub . sort 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.916391455640179e-3 | 1.9209731590028095e-3 | 1.936004191896445e-3 |
| Standard deviation | 9.113359011517176e-6 | 2.6450358886811063e-5 | 5.692768776700677e-5 |
Outlying measurements have slight (1.694411414982156e-2%) effect on estimated standard deviation.
60% dups/sort . nub 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.6496771949495687e-3 | 2.654350724027507e-3 | 2.6593615679438008e-3 |
| Standard deviation | 1.2582704020098308e-5 | 1.5466980100738843e-5 | 1.9726492666175793e-5 |
Outlying measurements have slight (1.8860946745561893e-2%) effect on estimated standard deviation.
60% dups/nub . sort 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.404851774771879e-3 | 2.4086140132885438e-3 | 2.4141733236255703e-3 |
| Standard deviation | 1.1276294217368505e-5 | 1.5594148431212945e-5 | 2.5129508969452253e-5 |
Outlying measurements have slight (1.8175582990397805e-2%) effect on estimated standard deviation.
50% dups/sort . nub 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.8863710327238293e-3 | 2.8928746951275078e-3 | 2.9058772958997186e-3 |
| Standard deviation | 1.699685920877944e-5 | 2.8789145783205682e-5 | 5.586070589376644e-5 |
Outlying measurements have slight (1.9223375624759672e-2%) effect on estimated standard deviation.
50% dups/nub . sort 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.659323261328539e-3 | 2.6625385686578377e-3 | 2.66594393437302e-3 |
| Standard deviation | 9.463685116360663e-6 | 1.0759834287911765e-5 | 1.2479412664345792e-5 |
Outlying measurements have slight (1.88609467455621e-2%) effect on estimated standard deviation.
40% dups/sort . nub 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.1817050730097675e-3 | 3.1854407333431435e-3 | 3.1903494160690957e-3 |
| Standard deviation | 1.034678073701202e-5 | 1.4112316108874426e-5 | 2.2589230427616685e-5 |
Outlying measurements have slight (1.999167013744273e-2%) effect on estimated standard deviation.
40% dups/nub . sort 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.0637214724737053e-3 | 3.0675555143490035e-3 | 3.0719003220668873e-3 |
| Standard deviation | 1.0939933526637702e-5 | 1.3306846078209808e-5 | 1.726837798554499e-5 |
Outlying measurements have slight (1.96e-2%) effect on estimated standard deviation.
30% dups/sort . nub 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.457674540300125e-3 | 3.4608865313036554e-3 | 3.4653749279933677e-3 |
| Standard deviation | 8.985612512429599e-6 | 1.1796571781776954e-5 | 1.6926604468035688e-5 |
Outlying measurements have slight (2.082390221819815e-2%) effect on estimated standard deviation.
30% dups/nub . sort 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.3838091246934064e-3 | 3.388678579219499e-3 | 3.394343382581759e-3 |
| Standard deviation | 1.425664590269864e-5 | 1.7061860599475802e-5 | 2.1000617200617626e-5 |
Outlying measurements have slight (2.039930555555545e-2%) effect on estimated standard deviation.
25% dups/sort . nub 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.549652870446891e-3 | 3.5543325426216133e-3 | 3.5599672847065524e-3 |
| Standard deviation | 1.4546726061048817e-5 | 1.6934148030916654e-5 | 1.9738096968267915e-5 |
Outlying measurements have slight (2.082390221819828e-2%) effect on estimated standard deviation.
25% dups/nub . sort 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.5906538095023225e-3 | 3.595388325778948e-3 | 3.600954460164912e-3 |
| Standard deviation | 1.3467713736532694e-5 | 1.655312342769315e-5 | 2.1830656822502982e-5 |
Outlying measurements have slight (2.082390221819821e-2%) effect on estimated standard deviation.
20% dups/sort . nub 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.781480557741861e-3 | 3.7875125865391287e-3 | 3.794857417447447e-3 |
| Standard deviation | 1.7320335852762688e-5 | 2.130075322494026e-5 | 2.733279404292193e-5 |
Outlying measurements have slight (2.126654064272212e-2%) effect on estimated standard deviation.
20% dups/nub . sort 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.838468399138971e-3 | 3.849484691029673e-3 | 3.889281972377442e-3 |
| Standard deviation | 9.941648543071004e-6 | 6.145465075381796e-5 | 1.287694427306962e-4 |
Outlying measurements have slight (2.1728395061728495e-2%) effect on estimated standard deviation.
10% dups/sort . nub 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.005359222565637e-3 | 4.00933476115889e-3 | 4.0129778631927794e-3 |
| Standard deviation | 9.886946042525709e-6 | 1.2233188768643487e-5 | 1.5369839719679826e-5 |
Outlying measurements have slight (2.1728395061728394e-2%) effect on estimated standard deviation.
10% dups/nub . sort 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.341311098785385e-3 | 4.347202324816069e-3 | 4.356769118538992e-3 |
| Standard deviation | 1.3633725972350999e-5 | 2.222168478250711e-5 | 4.236232346337532e-5 |
Outlying measurements have slight (2.271498107084893e-2%) effect on estimated standard deviation.
5% dups/sort . nub 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.146066768347288e-3 | 4.155627971191964e-3 | 4.165566397896197e-3 |
| Standard deviation | 2.333558400115933e-5 | 2.9079549308113702e-5 | 3.837910171238348e-5 |
Outlying measurements have slight (2.221074380165287e-2%) effect on estimated standard deviation.
5% dups/nub . sort 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.578229207676874e-3 | 4.585055320851664e-3 | 4.590635252015487e-3 |
| Standard deviation | 1.6285736114734832e-5 | 2.0195932478197822e-5 | 2.536945097063175e-5 |
Outlying measurements have slight (2.324263038548753e-2%) effect on estimated standard deviation.
1% dups/sort . nub 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.277758566303519e-3 | 4.283669312447701e-3 | 4.294588333199768e-3 |
| Standard deviation | 1.4989398678068132e-5 | 2.569594450217295e-5 | 4.642911343402163e-5 |
Outlying measurements have slight (2.2210743801652895e-2%) effect on estimated standard deviation.
1% dups/nub . sort 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.7353511544705506e-3 | 4.740556652365754e-3 | 4.7462434209791845e-3 |
| Standard deviation | 1.3909762874366092e-5 | 1.6896987906393625e-5 | 2.0988614633287635e-5 |
Outlying measurements have slight (2.3242630385487476e-2%) effect on estimated standard deviation.
0% dups/sort . nub 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.2809165409568245e-3 | 4.287199949121727e-3 | 4.293040353936015e-3 |
| Standard deviation | 1.6908425317947705e-5 | 2.0016405346159845e-5 | 2.423586485475622e-5 |
Outlying measurements have slight (2.221074380165289e-2%) effect on estimated standard deviation.
0% dups/nub . sort 1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.771318508254847e-3 | 4.779051052224051e-3 | 4.786624511413614e-3 |
| Standard deviation | 1.985040139737209e-5 | 2.3982163982577597e-5 | 3.134066794356606e-5 |
Outlying measurements have slight (2.3242630385487476e-2%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.