criterion performance measurements
overview
want to understand this report?
sort . nub/100(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.173037631339266e-6 | 9.263500695171424e-6 | 9.393906264027345e-6 |
| Standard deviation | 1.7267844934161444e-7 | 2.964032454657826e-7 | 4.6350227354634946e-7 |
Outlying measurements have moderate (0.35333688155601706%) effect on estimated standard deviation.
sort . nub/200(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.626138077059697e-5 | 1.6377406540075786e-5 | 1.651494254430343e-5 |
| Standard deviation | 2.0583096783959642e-7 | 2.7091865608710256e-7 | 3.6718922009510705e-7 |
Outlying measurements have moderate (0.1192189152503338%) effect on estimated standard deviation.
sort . nub/400(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.300880638253835e-5 | 3.322857196315732e-5 | 3.374055754674127e-5 |
| Standard deviation | 3.7477449612129175e-7 | 8.538351316513155e-7 | 1.5514799948844173e-6 |
Outlying measurements have moderate (0.2284354006469801%) effect on estimated standard deviation.
sort . nub/600(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.849760221781421e-5 | 4.902799413332862e-5 | 5.015436389552221e-5 |
| Standard deviation | 7.318818724282779e-7 | 1.7415115525807207e-6 | 2.981387801265773e-6 |
Outlying measurements have moderate (0.34840446280503207%) effect on estimated standard deviation.
sort . nub/800(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.437414999781205e-5 | 6.472684477308253e-5 | 6.524848680989611e-5 |
| Standard deviation | 6.042081630208192e-7 | 1.0821678306774127e-6 | 1.688473756933986e-6 |
Outlying measurements have slight (9.728253553567343e-2%) effect on estimated standard deviation.
sort . nub/1000(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.821479992817453e-5 | 7.870138286838307e-5 | 7.931782390982856e-5 |
| Standard deviation | 1.0923813315867377e-6 | 1.3744176704707712e-6 | 1.8042558711862426e-6 |
Outlying measurements have moderate (0.10112300932130171%) effect on estimated standard deviation.
sort . nub/1250(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.815050989904133e-5 | 9.863713958217255e-5 | 9.93987303809119e-5 |
| Standard deviation | 1.1839084404007441e-6 | 1.405475693983285e-6 | 1.7766739206702041e-6 |
Outlying measurements have slight (6.682069500324749e-2%) effect on estimated standard deviation.
sort . nub/1500(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.1791351737241545e-4 | 1.1860788810480439e-4 | 1.193115734881251e-4 |
| Standard deviation | 1.3701964319503003e-6 | 1.6285801252641432e-6 | 1.8956111574815516e-6 |
Outlying measurements have slight (6.894774723969316e-2%) effect on estimated standard deviation.
sort . nub/2000(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.603854631658863e-4 | 1.615096477188385e-4 | 1.6296068802038345e-4 |
| Standard deviation | 2.706992699221691e-6 | 3.820293453991605e-6 | 4.762213660607595e-6 |
Outlying measurements have moderate (0.15962607733174458%) effect on estimated standard deviation.
sort . nub/3000(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.3568644899271872e-4 | 2.3681390324191672e-4 | 2.3824385155384452e-4 |
| Standard deviation | 2.2472180181999038e-6 | 2.8782699252596115e-6 | 3.893140016043041e-6 |
Outlying measurements have slight (1.2193263222069817e-2%) effect on estimated standard deviation.
sort . nub/4000(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.145291709130323e-4 | 3.165342417608908e-4 | 3.189269588465818e-4 |
| Standard deviation | 4.812035186829266e-6 | 5.9115748747703235e-6 | 6.973815226105643e-6 |
Outlying measurements have slight (8.813017164482563e-2%) effect on estimated standard deviation.
sort . nub/5000(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.8009149966559144e-4 | 3.8250489289561154e-4 | 3.8440269254658507e-4 |
| Standard deviation | 4.173583456744971e-6 | 5.245938239637845e-6 | 7.128609853760524e-6 |
Outlying measurements have slight (1.388613370363018e-2%) effect on estimated standard deviation.
nub . sort/100(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.51632580871722e-5 | 2.5259311556676857e-5 | 2.534620171427621e-5 |
| Standard deviation | 1.5597080512400342e-7 | 2.0465063989035898e-7 | 2.8280310586885845e-7 |
Outlying measurements have no (7.873519778281516e-3%) effect on estimated standard deviation.
nub . sort/200(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.47315385134815e-5 | 5.4821898684632735e-5 | 5.489856019068529e-5 |
| Standard deviation | 1.6348971175349367e-7 | 2.2159050231492887e-7 | 2.743148412846618e-7 |
Outlying measurements have no (8.927846765684548e-3%) effect on estimated standard deviation.
nub . sort/400(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.2073701684762699e-4 | 1.2095540701388764e-4 | 1.2132268968755539e-4 |
| Standard deviation | 4.3721492043970104e-7 | 7.132515904655487e-7 | 1.0172141334765108e-6 |
Outlying measurements have slight (1.0525124490719627e-2%) effect on estimated standard deviation.
nub . sort/600(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.849510779285588e-4 | 1.8560549590051798e-4 | 1.8659333015147436e-4 |
| Standard deviation | 1.6822389358112764e-6 | 2.2093921548457633e-6 | 2.7870727867433147e-6 |
Outlying measurements have slight (1.149269875608429e-2%) effect on estimated standard deviation.
nub . sort/800(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.5914789372450444e-4 | 2.5978504627836195e-4 | 2.6088024055447105e-4 |
| Standard deviation | 1.690784052141868e-6 | 2.2062256929367453e-6 | 2.748014032565964e-6 |
Outlying measurements have slight (1.2497997115846619e-2%) effect on estimated standard deviation.
nub . sort/1000(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.262853034359749e-4 | 3.2781998130246656e-4 | 3.299091525349773e-4 |
| Standard deviation | 3.848017965464503e-6 | 4.672618217654571e-6 | 5.955007989692829e-6 |
Outlying measurements have slight (1.3155555555555969e-2%) effect on estimated standard deviation.
nub . sort/1250(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.026974404565907e-4 | 4.0450338102160604e-4 | 4.070680802293748e-4 |
| Standard deviation | 4.035940422874438e-6 | 5.6976664418757645e-6 | 8.373705016920455e-6 |
Outlying measurements have slight (1.4081632653061362e-2%) effect on estimated standard deviation.
nub . sort/1500(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.020732346487404e-4 | 5.032080680090447e-4 | 5.047853124689998e-4 |
| Standard deviation | 2.5792447857217724e-6 | 3.373650000384108e-6 | 4.387168211758303e-6 |
Outlying measurements have slight (1.492194674012856e-2%) effect on estimated standard deviation.
nub . sort/2000(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.756059118389956e-4 | 6.795444256135095e-4 | 6.850317792731881e-4 |
| Standard deviation | 7.571691484839963e-6 | 1.3595376578526537e-5 | 2.0808952913063772e-5 |
Outlying measurements have slight (7.881519515524552e-2%) effect on estimated standard deviation.
nub . sort/3000(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.0922605913624816e-3 | 1.095390295530309e-3 | 1.1010980070242848e-3 |
| Standard deviation | 8.879066467986537e-6 | 1.248669918563487e-5 | 1.8791804391352596e-5 |
Outlying measurements have slight (1.886094674556218e-2%) effect on estimated standard deviation.
nub . sort/4000(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.57559137473375e-3 | 1.582135580437252e-3 | 1.5912356033219222e-3 |
| Standard deviation | 1.8449417858359638e-5 | 2.4189196433254277e-5 | 3.106690660221411e-5 |
Outlying measurements have slight (2.172839506172859e-2%) effect on estimated standard deviation.
nub . sort/5000(1->10)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.103042498484733e-3 | 2.117745432682725e-3 | 2.15138977885411e-3 |
| Standard deviation | 1.8963415984758468e-5 | 6.106839540208284e-5 | 1.0959058071007448e-4 |
Outlying measurements have moderate (0.11766002551042844%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.