criterion performance measurements
overview
want to understand this report?
fonction A/1
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.5090403810440585e-7 | 2.526577774135903e-7 | 2.551385516770397e-7 |
| Standard deviation | 6.613203358528657e-9 | 8.132635033952196e-9 | 1.0097704702724325e-8 |
Outlying measurements have moderate (0.49255957741064577%) effect on estimated standard deviation.
fonction A/10
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.5014802992483456e-7 | 2.5119366139217637e-7 | 2.5237666596320136e-7 |
| Standard deviation | 3.7382110863330196e-9 | 4.443888695218533e-9 | 5.614405729170982e-9 |
Outlying measurements have moderate (0.22525285271780227%) effect on estimated standard deviation.
fonction A/100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.0504313260094405e-7 | 4.0598319032531493e-7 | 4.0708320380837933e-7 |
| Standard deviation | 3.4323707243877726e-9 | 4.164387906052383e-9 | 5.8478200246729996e-9 |
Outlying measurements have slight (8.521616264170062e-2%) effect on estimated standard deviation.
fonction A/1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.461194784475403e-7 | 5.479895116519376e-7 | 5.49815687120218e-7 |
| Standard deviation | 5.876997465412385e-9 | 7.29123632224786e-9 | 9.363663964249797e-9 |
Outlying measurements have moderate (0.13152596047132253%) effect on estimated standard deviation.
fonction A/5000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.481349002051773e-7 | 5.504887119309528e-7 | 5.53616176349442e-7 |
| Standard deviation | 7.46488508079917e-9 | 1.0002342936741274e-8 | 1.3737464075835397e-8 |
Outlying measurements have moderate (0.22047284358056357%) effect on estimated standard deviation.
fonction A/10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.488411458801314e-7 | 5.51922338316818e-7 | 5.563308194741325e-7 |
| Standard deviation | 9.294032352018033e-9 | 1.3579266806887674e-8 | 2.0359243218814142e-8 |
Outlying measurements have moderate (0.3433094134352012%) effect on estimated standard deviation.
fonction A/100000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.646481000938392e-7 | 6.674390121376153e-7 | 6.723882274571351e-7 |
| Standard deviation | 9.750668357710627e-9 | 1.351465208339671e-8 | 2.0848024886813992e-8 |
Outlying measurements have moderate (0.25742574212614244%) effect on estimated standard deviation.
fonction B/1
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.4397609402030816e-7 | 1.4508620393046988e-7 | 1.4653776718914943e-7 |
| Standard deviation | 3.6756941900813516e-9 | 4.9234356947598195e-9 | 7.1113381750189265e-9 |
Outlying measurements have severe (0.5282772258162961%) effect on estimated standard deviation.
fonction B/10
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.4472280022997636e-7 | 1.4581713272596337e-7 | 1.4707278706608933e-7 |
| Standard deviation | 3.78778449889403e-9 | 4.575043908826982e-9 | 6.000784889467741e-9 |
Outlying measurements have moderate (0.4903453218114755%) effect on estimated standard deviation.
fonction B/100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.1783696407739387e-7 | 2.1881102969410328e-7 | 2.199739631232619e-7 |
| Standard deviation | 3.260833541223716e-9 | 3.99972768730033e-9 | 5.133097623838213e-9 |
Outlying measurements have moderate (0.2385833772870107%) effect on estimated standard deviation.
fonction B/1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.800749364891874e-7 | 2.826472508832256e-7 | 2.872319785093576e-7 |
| Standard deviation | 9.132125840260201e-9 | 1.3356901713075108e-8 | 2.094706047134005e-8 |
Outlying measurements have severe (0.6772390258139132%) effect on estimated standard deviation.
fonction B/5000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.7464855865734796e-7 | 2.7610277487660106e-7 | 2.7770884618036223e-7 |
| Standard deviation | 4.749088571049623e-9 | 6.093523454313329e-9 | 7.79388688946002e-9 |
Outlying measurements have moderate (0.30827653479073935%) effect on estimated standard deviation.
fonction B/10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.7561066112335233e-7 | 2.771696338273134e-7 | 2.795708667025864e-7 |
| Standard deviation | 6.183735078670894e-9 | 7.91691532079479e-9 | 1.0340824426176567e-8 |
Outlying measurements have moderate (0.4294326107110721%) effect on estimated standard deviation.
fonction B/100000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.4539602284565173e-7 | 3.471027508871912e-7 | 3.4971728385065626e-7 |
| Standard deviation | 6.22185066852356e-9 | 8.195150937190338e-9 | 1.2047688909857561e-8 |
Outlying measurements have moderate (0.3344209941839077%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.