criterion performance measurements
overview
want to understand this report?
sort . nub/100(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.601688284979452e-5 | 8.641354152638848e-5 | 8.674575659324542e-5 |
| Standard deviation | 8.206977748603663e-7 | 1.0486329466683942e-6 | 1.3527723669743315e-6 |
Outlying measurements have no (9.80296049406898e-3%) effect on estimated standard deviation.
sort . nub/200(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.996945728891985e-4 | 3.0049653917762597e-4 | 3.02252485247663e-4 |
| Standard deviation | 2.2034531296305867e-6 | 3.053190484715889e-6 | 4.0695697819759416e-6 |
Outlying measurements have slight (1.2984764542936476e-2%) effect on estimated standard deviation.
sort . nub/400(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.0751216164153724e-3 | 1.0800424140249764e-3 | 1.0879509470909283e-3 |
| Standard deviation | 1.4218770651263746e-5 | 1.7609023507056518e-5 | 2.261540069357466e-5 |
Outlying measurements have slight (1.886094674556218e-2%) effect on estimated standard deviation.
sort . nub/600(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.3105757370767968e-3 | 2.330059541277886e-3 | 2.383008558758825e-3 |
| Standard deviation | 3.176560823210562e-5 | 8.586226644176986e-5 | 1.4992395874831164e-4 |
Outlying measurements have moderate (0.16998219030884812%) effect on estimated standard deviation.
sort . nub/800(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.9190961663992645e-3 | 3.935911847927723e-3 | 3.955589744864349e-3 |
| Standard deviation | 4.492401129585707e-5 | 5.5000763452347316e-5 | 7.003941034859953e-5 |
Outlying measurements have slight (3.2222222222222215e-2%) effect on estimated standard deviation.
sort . nub/1000(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.874351888334302e-3 | 5.932358802231678e-3 | 5.982450322651752e-3 |
| Standard deviation | 8.570781347824115e-5 | 1.3000237717213838e-4 | 1.812188406050065e-4 |
Outlying measurements have slight (3.83999999999998e-2%) effect on estimated standard deviation.
sort . nub/1250(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.53996814272472e-3 | 9.59292843214343e-3 | 9.644785406280453e-3 |
| Standard deviation | 8.593507026881297e-5 | 1.0752233814831147e-4 | 1.3357506765839032e-4 |
Outlying measurements have slight (4.986149584487534e-2%) effect on estimated standard deviation.
sort . nub/1500(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.3360412550668072e-2 | 1.3512346053984822e-2 | 1.3663545128137685e-2 |
| Standard deviation | 2.4011379936275234e-4 | 3.086744679702794e-4 | 4.0833387749203014e-4 |
Outlying measurements have slight (5.8593749999999986e-2%) effect on estimated standard deviation.
sort . nub/2000(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.3113009987299694e-2 | 2.3359119621698236e-2 | 2.382495379107152e-2 |
| Standard deviation | 3.789190919579907e-4 | 6.227623448251887e-4 | 8.57701878056939e-4 |
Outlying measurements have slight (7.63888888888887e-2%) effect on estimated standard deviation.
sort . nub/3000(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.0087763419048095e-2 | 5.14381635953873e-2 | 5.362757058126135e-2 |
| Standard deviation | 1.694740087542496e-3 | 2.7213652349580594e-3 | 3.5743856075839396e-3 |
Outlying measurements have moderate (0.11409696739377746%) effect on estimated standard deviation.
sort . nub/4000(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.625528041668151e-2 | 8.888567183333419e-2 | 9.363198833329735e-2 |
| Standard deviation | 1.1714755601801782e-3 | 5.324959495670561e-3 | 7.355890718673692e-3 |
Outlying measurements have moderate (0.14651099959096825%) effect on estimated standard deviation.
sort . nub/5000(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.12532589790000656 | 0.1338080460766446 | 0.15676407239991477 |
| Standard deviation | 1.6603643591565145e-3 | 1.6662662044537577e-2 | 2.1659121955190173e-2 |
Outlying measurements have moderate (0.3677562392214536%) effect on estimated standard deviation.
nub . sort/100(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.63025621354079e-5 | 8.702174800005504e-5 | 8.900726514275967e-5 |
| Standard deviation | 1.8679279367663798e-6 | 2.7305596211644046e-6 | 4.102379647193532e-6 |
Outlying measurements have moderate (0.265151691046096%) effect on estimated standard deviation.
nub . sort/200(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.9874100240243534e-4 | 3.003016553336435e-4 | 3.033046462828185e-4 |
| Standard deviation | 3.077158911107816e-6 | 7.1492935092725536e-6 | 1.113060783602996e-5 |
Outlying measurements have moderate (0.13713788806927185%) effect on estimated standard deviation.
nub . sort/400(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.0655604229045323e-3 | 1.0681490518600967e-3 | 1.0712861944730636e-3 |
| Standard deviation | 7.079385947953758e-6 | 8.912857877561698e-6 | 1.1839111243210442e-5 |
Outlying measurements have slight (1.886094674556213e-2%) effect on estimated standard deviation.
nub . sort/600(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.265728289535712e-3 | 2.2822124911931127e-3 | 2.332621112266842e-3 |
| Standard deviation | 1.7826631892687543e-5 | 6.887652953041216e-5 | 1.367382735131262e-4 |
Outlying measurements have moderate (0.1209751688275105%) effect on estimated standard deviation.
nub . sort/800(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.862550215733859e-3 | 3.8761930528628083e-3 | 3.891935261257112e-3 |
| Standard deviation | 2.6658041827838432e-5 | 3.394616286619869e-5 | 4.3963354741978684e-5 |
Outlying measurements have slight (3.1217481789802035e-2%) effect on estimated standard deviation.
nub . sort/1000(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.898631864445046e-3 | 5.993379915344883e-3 | 6.187179511543979e-3 |
| Standard deviation | 6.716587402534423e-5 | 3.561479139079926e-4 | 5.282605250801659e-4 |
Outlying measurements have moderate (0.26612353365932007%) effect on estimated standard deviation.
nub . sort/1250(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.993574732149779e-3 | 9.13745541258897e-3 | 9.341264036075459e-3 |
| Standard deviation | 2.832295430304527e-4 | 3.6715401213742525e-4 | 4.359526836894163e-4 |
Outlying measurements have moderate (0.1391125409832629%) effect on estimated standard deviation.
nub . sort/1500(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.1359360978066715e-2 | 1.1408981064985339e-2 | 1.1478107524302414e-2 |
| Standard deviation | 1.0041250403384487e-4 | 1.3092813098421449e-4 | 1.6517864699752826e-4 |
Outlying measurements have slight (5.246913580246913e-2%) effect on estimated standard deviation.
nub . sort/2000(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.8964569160080283e-2 | 1.9068451602485315e-2 | 1.937685690698383e-2 |
| Standard deviation | 1.1753860881212423e-4 | 3.035153544543021e-4 | 5.154487498557616e-4 |
Outlying measurements have slight (7.100591715976325e-2%) effect on estimated standard deviation.
nub . sort/3000(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.8971519123456234e-2 | 3.9221489190183395e-2 | 3.940552407629472e-2 |
| Standard deviation | 2.1639096898096194e-4 | 3.012994979622777e-4 | 3.946512769912841e-4 |
Outlying measurements have slight (9.876543209876543e-2%) effect on estimated standard deviation.
nub . sort/4000(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.73082800286119e-2 | 6.748841980920557e-2 | 6.757935383471352e-2 |
| Standard deviation | 6.201823638931874e-5 | 1.69108000598711e-4 | 2.491621547324255e-4 |
Outlying measurements have moderate (0.12244897959183672%) effect on estimated standard deviation.
nub . sort/5000(1->536870911)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.10386349440002504 | 0.10457026893004089 | 0.10533638130002601 |
| Standard deviation | 4.980578208676183e-4 | 8.876834617902416e-4 | 1.1605327176264387e-3 |
Outlying measurements have moderate (0.16000000000000003%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.