criterion performance measurements
overview
want to understand this report?
sort . nub/100(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.412708403539346e-5 | 8.428280861474216e-5 | 8.452334796567459e-5 |
| Standard deviation | 5.438169935156804e-7 | 6.492923594043273e-7 | 7.89173338721058e-7 |
Outlying measurements have no (9.707804690503367e-3%) effect on estimated standard deviation.
sort . nub/200(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.7994599638226123e-4 | 2.80881900232012e-4 | 2.8208918995837945e-4 |
| Standard deviation | 2.1054445165783925e-6 | 2.7545376029424036e-6 | 3.7369693558939195e-6 |
Outlying measurements have slight (1.2656147271532333e-2%) effect on estimated standard deviation.
sort . nub/400(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.227254912630544e-4 | 9.26958402567981e-4 | 9.371512337375196e-4 |
| Standard deviation | 1.057212860119539e-5 | 1.8122655392299185e-5 | 2.9527020449792474e-5 |
Outlying measurements have slight (8.476653960219219e-2%) effect on estimated standard deviation.
sort . nub/600(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.8626159503263137e-3 | 1.8671998774736898e-3 | 1.8778925804745984e-3 |
| Standard deviation | 1.3872146312489998e-5 | 2.0190195423272e-5 | 2.8274258381439847e-5 |
Outlying measurements have slight (2.3242630385487392e-2%) effect on estimated standard deviation.
sort . nub/800(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.79564513085102e-3 | 2.8085986001559027e-3 | 2.8252056247624457e-3 |
| Standard deviation | 2.777798187654579e-5 | 3.569496263915663e-5 | 4.739987171506165e-5 |
Outlying measurements have slight (2.7755102040816326e-2%) effect on estimated standard deviation.
sort . nub/1000(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.141632403002326e-3 | 4.157204368671638e-3 | 4.179978311615923e-3 |
| Standard deviation | 3.271282195585513e-5 | 4.740820789766323e-5 | 6.403663711314183e-5 |
Outlying measurements have slight (3.2222222222222034e-2%) effect on estimated standard deviation.
sort . nub/1250(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.789782848648188e-3 | 5.810931796971934e-3 | 5.834506492311298e-3 |
| Standard deviation | 4.3087738299234705e-5 | 5.33483311688547e-5 | 6.978538835027976e-5 |
Outlying measurements have slight (3.839999999999999e-2%) effect on estimated standard deviation.
sort . nub/1500(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.605045873378376e-3 | 7.63021691624117e-3 | 7.664515791920662e-3 |
| Standard deviation | 5.714815097891909e-5 | 7.164130229463679e-5 | 9.931315895344051e-5 |
Outlying measurements have slight (4.338842975206612e-2%) effect on estimated standard deviation.
sort . nub/2000(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.1315661369180037e-2 | 1.1366227299099049e-2 | 1.151224197676315e-2 |
| Standard deviation | 1.519884788253717e-4 | 1.9011722155764564e-4 | 2.3647680090724938e-4 |
Outlying measurements have slight (5.2469135802469126e-2%) effect on estimated standard deviation.
sort . nub/3000(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.7375851106096452e-2 | 1.791241330362621e-2 | 1.8297633931193043e-2 |
| Standard deviation | 7.084108723293579e-4 | 9.683982564262129e-4 | 1.182731214779378e-3 |
Outlying measurements have moderate (0.13506429647037746%) effect on estimated standard deviation.
sort . nub/4000(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.8115921423625907e-2 | 2.8371922946974194e-2 | 2.8542889150469365e-2 |
| Standard deviation | 2.4429820242778905e-4 | 3.732883064547752e-4 | 5.080929940065159e-4 |
Outlying measurements have slight (8.264462809917351e-2%) effect on estimated standard deviation.
sort . nub/5000(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.663095960338613e-2 | 3.689112313856714e-2 | 3.745232803206252e-2 |
| Standard deviation | 3.248852539954043e-4 | 7.089823541368252e-4 | 1.070605751893903e-3 |
Outlying measurements have slight (9.87654320987654e-2%) effect on estimated standard deviation.
nub . sort/100(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.427698330561578e-5 | 8.495931191498575e-5 | 8.55249579435448e-5 |
| Standard deviation | 1.1064420396365576e-6 | 1.3340559082419088e-6 | 1.7521992582666873e-6 |
Outlying measurements have slight (8.339826790112466e-2%) effect on estimated standard deviation.
nub . sort/200(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.753253071650909e-4 | 2.766076716980822e-4 | 2.78072350586011e-4 |
| Standard deviation | 2.8156882253911094e-6 | 3.7746242632801048e-6 | 5.149665835910217e-6 |
Outlying measurements have slight (1.265614727153205e-2%) effect on estimated standard deviation.
nub . sort/400(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.729867651943291e-4 | 8.760550918410735e-4 | 8.792004564629108e-4 |
| Standard deviation | 4.907822881213445e-6 | 7.742248277634635e-6 | 1.1056010799347357e-5 |
Outlying measurements have slight (1.7538265306122448e-2%) effect on estimated standard deviation.
nub . sort/600(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.5968345157558444e-3 | 1.6052869524335257e-3 | 1.6145915819292807e-3 |
| Standard deviation | 1.9491849759931553e-5 | 2.3772192074613517e-5 | 3.0023828272796626e-5 |
Outlying measurements have slight (2.1728395061728273e-2%) effect on estimated standard deviation.
nub . sort/800(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.307165371369687e-3 | 2.3147433215015734e-3 | 2.3320674266703e-3 |
| Standard deviation | 2.38848431375833e-5 | 2.7882264945555643e-5 | 3.772246728307259e-5 |
Outlying measurements have slight (2.498356344510191e-2%) effect on estimated standard deviation.
nub . sort/1000(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.012282332844349e-3 | 3.024704774962962e-3 | 3.0426091013997392e-3 |
| Standard deviation | 2.380509069195574e-5 | 3.1802312846490546e-5 | 4.1050731827309136e-5 |
Outlying measurements have slight (2.8546712802768166e-2%) effect on estimated standard deviation.
nub . sort/1250(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.327755158968874e-3 | 4.430139805295954e-3 | 4.558208817332696e-3 |
| Standard deviation | 2.3752777635830614e-4 | 2.955268218339374e-4 | 3.7281178807016204e-4 |
Outlying measurements have moderate (0.3304307491512877%) effect on estimated standard deviation.
nub . sort/1500(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.691393875722979e-3 | 4.719770551854024e-3 | 4.74410882366315e-3 |
| Standard deviation | 4.8022719822338415e-5 | 5.847045950151639e-5 | 7.42093680812756e-5 |
Outlying measurements have slight (3.443877551020404e-2%) effect on estimated standard deviation.
nub . sort/2000(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.947432964131459e-3 | 5.965707637338252e-3 | 5.98687432147252e-3 |
| Standard deviation | 4.0852076390746066e-5 | 4.871205423212286e-5 | 6.0015921412325386e-5 |
Outlying measurements have slight (3.84e-2%) effect on estimated standard deviation.
nub . sort/3000(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.206151204436386e-3 | 8.228719127079205e-3 | 8.253530545751417e-3 |
| Standard deviation | 3.534361749969394e-5 | 5.5157224548335045e-5 | 7.367562412305112e-5 |
Outlying measurements have slight (4.535147392290249e-2%) effect on estimated standard deviation.
nub . sort/4000(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.678917193516831e-3 | 9.702687732491328e-3 | 9.781452464452523e-3 |
| Standard deviation | 5.4148338964026366e-5 | 8.467956471702421e-5 | 1.2475575488861884e-4 |
Outlying measurements have slight (4.986149584487534e-2%) effect on estimated standard deviation.
nub . sort/5000(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.0650843372757063e-2 | 1.0679375863811014e-2 | 1.0742839891806879e-2 |
| Standard deviation | 5.6812200375506224e-5 | 8.703541035268541e-5 | 1.4022611976622943e-4 |
Outlying measurements have slight (5.24691358024691e-2%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.