criterion performance measurements
overview
want to understand this report?
sort . nub/100(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.9890056929707976e-5 | 4.9979490027449354e-5 | 5.007660122773075e-5 |
| Standard deviation | 1.92146822909029e-7 | 2.480548047712669e-7 | 2.925134379121018e-7 |
Outlying measurements have no (8.848852040816277e-3%) effect on estimated standard deviation.
sort . nub/200(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.774573058842562e-4 | 1.7774412061217614e-4 | 1.7827654148536544e-4 |
| Standard deviation | 7.84277115272655e-7 | 1.277278365245899e-6 | 1.834047558259831e-6 |
Outlying measurements have slight (1.1362135024442036e-2%) effect on estimated standard deviation.
sort . nub/400(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.363998986549905e-4 | 6.392523068558972e-4 | 6.427235244901397e-4 |
| Standard deviation | 6.146796688892394e-6 | 7.877905823774261e-6 | 1.057424999053296e-5 |
Outlying measurements have slight (1.586888657648289e-2%) effect on estimated standard deviation.
sort . nub/600(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.3015176707254841e-3 | 1.3057868178207727e-3 | 1.3112017823320482e-3 |
| Standard deviation | 1.0751754877774367e-5 | 1.396035927209839e-5 | 1.8337945946467235e-5 |
Outlying measurements have slight (2.0399305555555452e-2%) effect on estimated standard deviation.
sort . nub/800(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.053731065198546e-3 | 2.0586154035162818e-3 | 2.0640378013787697e-3 |
| Standard deviation | 9.282312985330934e-6 | 1.4211522167369824e-5 | 1.9371484465654823e-5 |
Outlying measurements have slight (2.3795359904818465e-2%) effect on estimated standard deviation.
sort . nub/1000(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.877682553234292e-3 | 2.882623491184915e-3 | 2.8923031770271713e-3 |
| Standard deviation | 1.3050844142931901e-5 | 2.218605685588604e-5 | 3.1168413523922906e-5 |
Outlying measurements have slight (2.7755102040816267e-2%) effect on estimated standard deviation.
sort . nub/1250(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.98752590500976e-3 | 3.995297317442978e-3 | 4.005253782858267e-3 |
| Standard deviation | 1.3041111022993212e-5 | 2.2410081626275807e-5 | 2.9871519550842418e-5 |
Outlying measurements have slight (3.222222222222209e-2%) effect on estimated standard deviation.
sort . nub/1500(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.288364872967205e-3 | 5.29493121357799e-3 | 5.306462418427028e-3 |
| Standard deviation | 1.3854640084822788e-5 | 2.075517788594891e-5 | 2.7886371939557197e-5 |
Outlying measurements have slight (3.698224852070992e-2%) effect on estimated standard deviation.
sort . nub/2000(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.099849215402785e-3 | 8.134179290395723e-3 | 8.193092123664893e-3 |
| Standard deviation | 5.425376132504736e-5 | 8.71866426788114e-5 | 1.4303445037847342e-4 |
Outlying measurements have slight (4.535147392290227e-2%) effect on estimated standard deviation.
sort . nub/3000(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.3802131424208637e-2 | 1.3866087134532685e-2 | 1.391453226793713e-2 |
| Standard deviation | 7.974300580571309e-5 | 1.0996136478146354e-4 | 1.4790569874233712e-4 |
Outlying measurements have slight (5.859375e-2%) effect on estimated standard deviation.
sort . nub/4000(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.0450751970954622e-2 | 2.050668744763688e-2 | 2.064067211457241e-2 |
| Standard deviation | 6.303875092385447e-5 | 1.9387075822774848e-4 | 2.496448231087852e-4 |
Outlying measurements have slight (7.100591715976319e-2%) effect on estimated standard deviation.
sort . nub/5000(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.596486569921882e-2 | 2.6050706814547908e-2 | 2.6198448710965477e-2 |
| Standard deviation | 1.606869881729483e-4 | 2.0867163850032407e-4 | 2.695705960475899e-4 |
Outlying measurements have slight (8.264462809917353e-2%) effect on estimated standard deviation.
nub . sort/100(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.252482346368002e-5 | 5.2634138226322096e-5 | 5.29140273311206e-5 |
| Standard deviation | 3.5800971293832767e-7 | 4.814900172810157e-7 | 6.461100353737091e-7 |
Outlying measurements have no (8.927846765684318e-3%) effect on estimated standard deviation.
nub . sort/200(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.8867566211567245e-4 | 1.8914061373470202e-4 | 1.8958516100844153e-4 |
| Standard deviation | 9.373885262695321e-7 | 1.244509361726705e-6 | 1.6698632444061457e-6 |
Outlying measurements have slight (1.1626297577854841e-2%) effect on estimated standard deviation.
nub . sort/400(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.691975352736602e-4 | 6.712392876586298e-4 | 6.733395637983126e-4 |
| Standard deviation | 4.754519935504887e-6 | 6.145931828188291e-6 | 7.34287733993525e-6 |
Outlying measurements have slight (1.6124697661918895e-2%) effect on estimated standard deviation.
nub . sort/600(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.2182799320168672e-3 | 1.224868891510548e-3 | 1.2369129753949037e-3 |
| Standard deviation | 1.7970079505161785e-5 | 2.4842587124872056e-5 | 3.797246392452889e-5 |
Outlying measurements have slight (7.513799319759219e-2%) effect on estimated standard deviation.
nub . sort/800(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.8715961515452877e-3 | 1.8784255110054567e-3 | 1.885056385977592e-3 |
| Standard deviation | 1.5826285745489875e-5 | 1.7962130783047734e-5 | 2.0498091164475396e-5 |
Outlying measurements have slight (2.3242630385487528e-2%) effect on estimated standard deviation.
nub . sort/1000(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.6543167520671988e-3 | 2.658903648444294e-3 | 2.6634689230478404e-3 |
| Standard deviation | 1.0531373485208126e-5 | 1.2205088769725036e-5 | 1.4638266594973236e-5 |
Outlying measurements have slight (2.700617283950602e-2%) effect on estimated standard deviation.
nub . sort/1250(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.658654728449784e-3 | 3.685673394791171e-3 | 3.7006878684978915e-3 |
| Standard deviation | 4.202864735405513e-5 | 5.098493883039992e-5 | 5.996567091751282e-5 |
Outlying measurements have slight (3.1217481789802187e-2%) effect on estimated standard deviation.
nub . sort/1500(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.360948524094164e-3 | 4.367836455957483e-3 | 4.375031176260478e-3 |
| Standard deviation | 1.5734142118918266e-5 | 1.921849653914224e-5 | 2.3676313522929544e-5 |
Outlying measurements have slight (3.329369797859687e-2%) effect on estimated standard deviation.
nub . sort/2000(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.885489159079811e-3 | 5.892153007146299e-3 | 5.899997582895967e-3 |
| Standard deviation | 1.6051879758707154e-5 | 1.9977402655491634e-5 | 2.7695484473264988e-5 |
Outlying measurements have slight (3.839999999999995e-2%) effect on estimated standard deviation.
nub . sort/3000(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.086584848944194e-3 | 8.09794260887604e-3 | 8.117769925867853e-3 |
| Standard deviation | 3.1140970681087047e-5 | 3.5733146199633344e-5 | 4.650131487726725e-5 |
Outlying measurements have slight (4.535147392290228e-2%) effect on estimated standard deviation.
nub . sort/4000(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.448140238077623e-3 | 9.460618951190538e-3 | 9.473848974736339e-3 |
| Standard deviation | 2.0104058994367147e-5 | 2.6714645652664253e-5 | 3.9506918581402794e-5 |
Outlying measurements have slight (4.75e-2%) effect on estimated standard deviation.
nub . sort/5000(1->1400)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.0706639107201267e-2 | 1.072723038470658e-2 | 1.0772809103532042e-2 |
| Standard deviation | 4.0178951174718254e-5 | 6.0624281580913094e-5 | 8.627761904047203e-5 |
Outlying measurements have slight (5.246913580246878e-2%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.