criterion performance measurements
overview
want to understand this report?
sort . nub/100(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.803016898317925e-6 | 5.8427581176719815e-6 | 5.96242641585806e-6 |
| Standard deviation | 6.064463735953833e-8 | 1.8165213657997392e-7 | 3.1967893919400536e-7 |
Outlying measurements have moderate (0.35747111256485814%) effect on estimated standard deviation.
sort . nub/200(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.0616921492212371e-5 | 1.0637277548724143e-5 | 1.0671494246423912e-5 |
| Standard deviation | 5.69118184841315e-8 | 7.118890344143704e-8 | 9.313921242446776e-8 |
Outlying measurements have no (6.8962191358024685e-3%) effect on estimated standard deviation.
sort . nub/400(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.167285639504089e-5 | 2.1789497473419878e-5 | 2.198955664415223e-5 |
| Standard deviation | 2.7747886078607825e-7 | 3.65060201833674e-7 | 4.5850805043775924e-7 |
Outlying measurements have moderate (0.11739271639359747%) effect on estimated standard deviation.
sort . nub/600(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.127896995414949e-5 | 3.139884539397163e-5 | 3.1503455682373745e-5 |
| Standard deviation | 1.8942877274331762e-7 | 2.619049280644739e-7 | 3.6995798168662687e-7 |
Outlying measurements have no (8.12953507121768e-3%) effect on estimated standard deviation.
sort . nub/800(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.1949089224115354e-5 | 4.2204705494077584e-5 | 4.247967711040388e-5 |
| Standard deviation | 5.126467914586057e-7 | 6.96978880952439e-7 | 9.259082059833913e-7 |
Outlying measurements have moderate (0.10586577280381143%) effect on estimated standard deviation.
sort . nub/1000(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.142714401411551e-5 | 5.160488992454314e-5 | 5.1802439613984636e-5 |
| Standard deviation | 4.5772945378397795e-7 | 5.508480317680219e-7 | 7.040972387735487e-7 |
Outlying measurements have no (8.848852040816457e-3%) effect on estimated standard deviation.
sort . nub/1250(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.566610572877019e-5 | 6.591565785481325e-5 | 6.627752900899735e-5 |
| Standard deviation | 5.275816330413659e-7 | 8.094254294384727e-7 | 1.1259302062540059e-6 |
Outlying measurements have slight (4.02597011273025e-2%) effect on estimated standard deviation.
sort . nub/1500(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.799596255779397e-5 | 7.841190871146876e-5 | 7.891515284186254e-5 |
| Standard deviation | 1.0592914300072955e-6 | 1.2716907894269749e-6 | 1.5880563548117364e-6 |
Outlying measurements have slight (9.165100433021929e-2%) effect on estimated standard deviation.
sort . nub/2000(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.0351633746412174e-4 | 1.0396880978678948e-4 | 1.0442834242361642e-4 |
| Standard deviation | 9.873549827837924e-7 | 1.2301613606544101e-6 | 1.6656762297607875e-6 |
Outlying measurements have slight (1.009995835068729e-2%) effect on estimated standard deviation.
sort . nub/3000(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.5289291542882453e-4 | 1.537715740700973e-4 | 1.5501513842563726e-4 |
| Standard deviation | 1.9497654302636497e-6 | 2.3305452326334025e-6 | 3.022380966382574e-6 |
Outlying measurements have slight (7.32055735074276e-2%) effect on estimated standard deviation.
sort . nub/4000(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.0281718885753616e-4 | 2.0363027873923276e-4 | 2.0513571312063186e-4 |
| Standard deviation | 2.353524256061563e-6 | 3.1652486317674437e-6 | 4.325325837623405e-6 |
Outlying measurements have slight (6.750073204285821e-2%) effect on estimated standard deviation.
sort . nub/5000(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.5663059378257827e-4 | 2.5784606898289383e-4 | 2.6093249094415476e-4 |
| Standard deviation | 2.4334466761006336e-6 | 4.155917928477889e-6 | 6.325677279259126e-6 |
Outlying measurements have slight (7.155771065316555e-2%) effect on estimated standard deviation.
nub . sort/100(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.8789441390388496e-5 | 1.8897559774152387e-5 | 1.9017746084731045e-5 |
| Standard deviation | 2.4134381448032934e-7 | 3.1508904316484904e-7 | 4.289416889935044e-7 |
Outlying measurements have moderate (0.12172566390385064%) effect on estimated standard deviation.
nub . sort/200(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.216475057444917e-5 | 4.2317400734772936e-5 | 4.246978092192471e-5 |
| Standard deviation | 3.7062944224911167e-7 | 4.3351137076525807e-7 | 5.089678674216461e-7 |
Outlying measurements have no (8.546373365041474e-3%) effect on estimated standard deviation.
nub . sort/400(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.282245277015957e-5 | 9.320961990839635e-5 | 9.355723657418552e-5 |
| Standard deviation | 8.201438284319131e-7 | 1.0506020037703956e-6 | 1.4108922473546788e-6 |
Outlying measurements have no (9.899999999999907e-3%) effect on estimated standard deviation.
nub . sort/600(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.4832539255063426e-4 | 1.492731058294397e-4 | 1.5042820345400224e-4 |
| Standard deviation | 2.29003584960499e-6 | 2.6726065105320724e-6 | 3.191635743251117e-6 |
Outlying measurements have slight (9.458410032858765e-2%) effect on estimated standard deviation.
nub . sort/800(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.9772263832346315e-4 | 1.9884930315183615e-4 | 2.0170872280043264e-4 |
| Standard deviation | 3.3832443230180613e-6 | 4.739789145417122e-6 | 6.484894287969775e-6 |
Outlying measurements have moderate (0.14692040786180915%) effect on estimated standard deviation.
nub . sort/1000(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.674649207399116e-4 | 2.6904871211779223e-4 | 2.701282698317177e-4 |
| Standard deviation | 3.0907013294925793e-6 | 3.7025180941590646e-6 | 4.8972765880329105e-6 |
Outlying measurements have slight (1.265614727153205e-2%) effect on estimated standard deviation.
nub . sort/1250(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.666235836384875e-4 | 3.686917554876405e-4 | 3.741375068870112e-4 |
| Standard deviation | 3.597770609980744e-6 | 1.023396860661694e-5 | 1.8299030310450773e-5 |
Outlying measurements have moderate (0.17137333050247522%) effect on estimated standard deviation.
nub . sort/1500(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.346983162821604e-4 | 4.367096435763704e-4 | 4.41128227664025e-4 |
| Standard deviation | 4.592231019632417e-6 | 6.3263607061376705e-6 | 8.329612715083202e-6 |
Outlying measurements have slight (1.4282713715605806e-2%) effect on estimated standard deviation.
nub . sort/2000(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.006059545172231e-4 | 6.047946163595983e-4 | 6.076365673088377e-4 |
| Standard deviation | 7.499819691258609e-6 | 9.695987404426651e-6 | 1.2441309132311332e-5 |
Outlying measurements have slight (4.9553393377783576e-2%) effect on estimated standard deviation.
nub . sort/3000(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.012643199449107e-3 | 1.0182876870939695e-3 | 1.0252284007625304e-3 |
| Standard deviation | 1.4490086932630799e-5 | 1.6553110208887063e-5 | 1.9431924437269065e-5 |
Outlying measurements have slight (1.8511925952296015e-2%) effect on estimated standard deviation.
nub . sort/4000(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.4569972433267314e-3 | 1.463655787261681e-3 | 1.4686469020670464e-3 |
| Standard deviation | 1.220352386340164e-5 | 1.6516676004065828e-5 | 2.1868215093035805e-5 |
Outlying measurements have slight (2.1266540642721862e-2%) effect on estimated standard deviation.
nub . sort/5000(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.0887440677281066e-3 | 2.1014172460069587e-3 | 2.120051430518218e-3 |
| Standard deviation | 2.4420399549278726e-5 | 3.953901637408655e-5 | 6.16988966374302e-5 |
Outlying measurements have slight (2.4374999999999973e-2%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.