criterion performance measurements
overview
want to understand this report?
sort . nub/1000(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.449127422239928e-3 | 4.467718588530757e-3 | 4.4822556672707925e-3 |
| Standard deviation | 3.56005662342857e-5 | 4.5177301915570863e-5 | 5.282499624756689e-5 |
Outlying measurements have slight (3.329369797859691e-2%) effect on estimated standard deviation.
sort . nub/1000(1->100000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.3645285547580384e-3 | 4.376037876267016e-3 | 4.390950427259955e-3 |
| Standard deviation | 2.361689658244095e-5 | 3.561800964698775e-5 | 4.994363526778487e-5 |
Outlying measurements have slight (3.329369797859691e-2%) effect on estimated standard deviation.
sort . nub/1000(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.086077134228165e-3 | 4.092736261052948e-3 | 4.104667363093675e-3 |
| Standard deviation | 1.9571057737632654e-5 | 2.5343915979275506e-5 | 3.1131945951610504e-5 |
Outlying measurements have slight (3.222222222222222e-2%) effect on estimated standard deviation.
sort . nub/1000(1->5000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.921240317674046e-3 | 3.940286288556364e-3 | 3.960616295651621e-3 |
| Standard deviation | 3.590058528350664e-5 | 4.631608669192931e-5 | 5.656542134628966e-5 |
Outlying measurements have slight (3.222222222222222e-2%) effect on estimated standard deviation.
sort . nub/1000(1->2000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.267946590972865e-3 | 3.2771502972043843e-3 | 3.298839796799492e-3 |
| Standard deviation | 2.079801461826313e-5 | 2.6488968477933393e-5 | 3.2370136513007594e-5 |
Outlying measurements have slight (2.9384756657483812e-2%) effect on estimated standard deviation.
sort . nub/1000(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.926539427193961e-3 | 2.933046151912885e-3 | 2.9441473140883693e-3 |
| Standard deviation | 1.8326185492795542e-5 | 2.4553549702156607e-5 | 3.130913718262209e-5 |
Outlying measurements have slight (2.7755102040816323e-2%) effect on estimated standard deviation.
sort . nub/1000(1->500)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.690933635869525e-3 | 1.6934282770664038e-3 | 1.6988756351646855e-3 |
| Standard deviation | 6.982194540448961e-6 | 9.7113793068901e-6 | 1.3741010059837093e-5 |
Outlying measurements have slight (2.221074380165289e-2%) effect on estimated standard deviation.
sort . nub/1000(1->100)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.792513322906108e-4 | 3.803298194830379e-4 | 3.828698608649089e-4 |
| Standard deviation | 3.0862993646402017e-6 | 3.844690677708289e-6 | 5.092748822473012e-6 |
Outlying measurements have slight (1.3695987654320781e-2%) effect on estimated standard deviation.
sort . nub/1000(1->50)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.8897471153132278e-4 | 1.896450748620939e-4 | 1.9038773593283963e-4 |
| Standard deviation | 1.2401682363228752e-6 | 1.6853324236268766e-6 | 2.1846570522630237e-6 |
Outlying measurements have slight (1.1626297577854586e-2%) effect on estimated standard deviation.
sort . nub/1000(1->20)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.610048884092083e-5 | 8.630791133317868e-5 | 8.659910935733947e-5 |
| Standard deviation | 4.783447497176129e-7 | 6.190963975146361e-7 | 8.494447522774935e-7 |
Outlying measurements have no (9.802960494069032e-3%) effect on estimated standard deviation.
sort . nub/1000(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.275609739656748e-5 | 5.292896950281136e-5 | 5.308739102784644e-5 |
| Standard deviation | 3.555855397333798e-7 | 4.262519668970709e-7 | 5.445592214689172e-7 |
Outlying measurements have no (8.927846765684614e-3%) effect on estimated standard deviation.
nub . sort/1000(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.819456360913092e-3 | 4.83505984917049e-3 | 4.8472120821764e-3 |
| Standard deviation | 2.7037236506337205e-5 | 3.354947469667422e-5 | 4.401561498114546e-5 |
Outlying measurements have slight (3.443877551020408e-2%) effect on estimated standard deviation.
nub . sort/1000(1->100000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.793870997908157e-3 | 4.806648739335707e-3 | 4.815766087723638e-3 |
| Standard deviation | 2.351700243489309e-5 | 2.9693617131924283e-5 | 3.604648826453838e-5 |
Outlying measurements have slight (3.4438775510204085e-2%) effect on estimated standard deviation.
nub . sort/1000(1->8000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.1662863369722825e-3 | 4.17582589622093e-3 | 4.185024618141699e-3 |
| Standard deviation | 1.733093233510453e-5 | 2.3192524533281533e-5 | 3.3203906058439655e-5 |
Outlying measurements have slight (3.222222222222222e-2%) effect on estimated standard deviation.
nub . sort/1000(1->5000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.107918965543092e-3 | 4.117628823379225e-3 | 4.135429855904125e-3 |
| Standard deviation | 1.892529026737733e-5 | 2.6131519277870328e-5 | 4.01268132196734e-5 |
Outlying measurements have slight (3.222222222222201e-2%) effect on estimated standard deviation.
nub . sort/1000(1->2000)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.06373555294145e-3 | 3.07157258139758e-3 | 3.0791125777757345e-3 |
| Standard deviation | 1.8010045649659483e-5 | 2.2561776104948963e-5 | 2.9304153874426543e-5 |
Outlying measurements have slight (2.854671280276817e-2%) effect on estimated standard deviation.
nub . sort/1000(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.7111093215607084e-3 | 2.7193081642987805e-3 | 2.725846069917118e-3 |
| Standard deviation | 1.4240403657923868e-5 | 1.869944443803834e-5 | 2.5637115752487966e-5 |
Outlying measurements have slight (2.700617283950604e-2%) effect on estimated standard deviation.
nub . sort/1000(1->500)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.2513713315758694e-3 | 1.2555565244731191e-3 | 1.259688578503607e-3 |
| Standard deviation | 8.056698423433784e-6 | 9.318608738919692e-6 | 1.1187710469498835e-5 |
Outlying measurements have slight (1.9991670137442616e-2%) effect on estimated standard deviation.
nub . sort/1000(1->100)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.6827139100511683e-4 | 3.6940243256248004e-4 | 3.7154320192838374e-4 |
| Standard deviation | 3.5289510189415686e-6 | 4.49713339370202e-6 | 6.143204538482736e-6 |
Outlying measurements have slight (1.3695987654320988e-2%) effect on estimated standard deviation.
nub . sort/1000(1->50)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.1575544718031294e-4 | 3.1724960595480643e-4 | 3.200966627109548e-4 |
| Standard deviation | 3.2413869211082227e-6 | 4.810891602034594e-6 | 6.8566750309795896e-6 |
Outlying measurements have slight (6.26401705550201e-2%) effect on estimated standard deviation.
nub . sort/1000(1->20)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.88224359215638e-4 | 2.893245080669463e-4 | 2.9090175882909693e-4 |
| Standard deviation | 3.1016992217654365e-6 | 4.046984957968173e-6 | 5.398285041985691e-6 |
Outlying measurements have slight (1.2818350480688337e-2%) effect on estimated standard deviation.
nub . sort/1000(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.683657302019353e-4 | 2.696549625903623e-4 | 2.7111621469641875e-4 |
| Standard deviation | 2.705135951906428e-6 | 3.514121864395073e-6 | 4.795253320566446e-6 |
Outlying measurements have slight (1.2656147271532109e-2%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.