criterion performance measurements
overview
want to understand this report?
100% dups/sort . nub 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.72468571221113e-6 | 5.740538558824375e-6 | 5.771020807121755e-6 |
| Standard deviation | 4.730258794424012e-8 | 7.329016692655842e-8 | 1.1632022397707401e-7 |
Outlying measurements have slight (9.706199200709434e-2%) effect on estimated standard deviation.
100% dups/nub . sort 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.9544027740630374e-5 | 1.958828255610482e-5 | 1.9629292926297498e-5 |
| Standard deviation | 1.160204011547391e-7 | 1.3748378189213948e-7 | 1.7070990746339426e-7 |
Outlying measurements have no (6.622222222222322e-3%) effect on estimated standard deviation.
99% dups/sort . nub 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.1051651339931592e-5 | 1.1083096536571557e-5 | 1.11582720900996e-5 |
| Standard deviation | 7.299788073230841e-8 | 1.5326389007487397e-7 | 2.877780230057781e-7 |
Outlying measurements have moderate (0.10488434524730203%) effect on estimated standard deviation.
99% dups/nub . sort 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.153596756270284e-5 | 2.1570148630021358e-5 | 2.1620860907429448e-5 |
| Standard deviation | 1.0425800302559891e-7 | 1.3727470359430007e-7 | 2.130805557856782e-7 |
Outlying measurements have no (6.711102994886817e-3%) effect on estimated standard deviation.
95% dups/sort . nub 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.528436636342486e-5 | 1.5323843214130496e-5 | 1.5363587650418966e-5 |
| Standard deviation | 1.0337804497778894e-7 | 1.277013062700293e-7 | 1.6259081552008595e-7 |
Outlying measurements have no (6.409989594172845e-3%) effect on estimated standard deviation.
95% dups/nub . sort 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.2571169762742954e-5 | 2.2613436092682827e-5 | 2.265852984264909e-5 |
| Standard deviation | 1.2897212809698224e-7 | 1.5545254806711358e-7 | 2.0583871767919872e-7 |
Outlying measurements have no (6.7564440742284146e-3%) effect on estimated standard deviation.
90% dups/sort . nub 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.841604310515107e-5 | 1.8449719091754257e-5 | 1.8478427141829585e-5 |
| Standard deviation | 8.916296279971136e-8 | 1.080047016023811e-7 | 1.3980527897535016e-7 |
Outlying measurements have no (6.535664819944422e-3%) effect on estimated standard deviation.
90% dups/nub . sort 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.346108954564922e-5 | 2.3537910186860734e-5 | 2.3638437931501612e-5 |
| Standard deviation | 2.2922526378892825e-7 | 2.877218588890866e-7 | 3.832675741522739e-7 |
Outlying measurements have slight (7.689219180259152e-2%) effect on estimated standard deviation.
80% dups/sort . nub 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.3740063182615715e-5 | 2.3792073144445597e-5 | 2.386655514140691e-5 |
| Standard deviation | 1.4690091871959718e-7 | 2.0357383978295146e-7 | 3.109344955836961e-7 |
Outlying measurements have no (6.802401951585629e-3%) effect on estimated standard deviation.
80% dups/nub . sort 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.8312901232544135e-5 | 2.8359708741466903e-5 | 2.8403575326606318e-5 |
| Standard deviation | 1.3030932326572963e-7 | 1.568705942701643e-7 | 2.106511868191512e-7 |
Outlying measurements have no (6.9441048462028305e-3%) effect on estimated standard deviation.
75% dups/sort . nub 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.590705767713011e-5 | 2.594246518300186e-5 | 2.598246255224041e-5 |
| Standard deviation | 1.072483994149913e-7 | 1.244270778033459e-7 | 1.552798348791157e-7 |
Outlying measurements have no (6.848989298454211e-3%) effect on estimated standard deviation.
75% dups/nub . sort 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.143692217267703e-5 | 3.1508577698532224e-5 | 3.1580846752830646e-5 |
| Standard deviation | 1.9988238537076e-7 | 2.4994345598731766e-7 | 3.593476226151845e-7 |
Outlying measurements have no (7.0418993008400585e-3%) effect on estimated standard deviation.
70% dups/sort . nub 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.551327599034386e-5 | 2.5570207389177504e-5 | 2.562475699153022e-5 |
| Standard deviation | 1.5751569445345662e-7 | 1.909253390435074e-7 | 2.3571451590736073e-7 |
Outlying measurements have no (6.848989298454212e-3%) effect on estimated standard deviation.
70% dups/nub . sort 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.997272890922733e-5 | 3.0026328201546215e-5 | 3.0134065368813667e-5 |
| Standard deviation | 1.607175229094437e-7 | 2.5653447140778067e-7 | 4.4903890855157066e-7 |
Outlying measurements have no (6.9926601864709675e-3%) effect on estimated standard deviation.
60% dups/sort . nub 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.185819782750739e-5 | 3.190676982163783e-5 | 3.196770278455549e-5 |
| Standard deviation | 1.3974292580984243e-7 | 1.7676913456000688e-7 | 2.472497556582221e-7 |
Outlying measurements have no (7.091836734694061e-3%) effect on estimated standard deviation.
60% dups/nub . sort 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.356117815782447e-5 | 3.361375854687947e-5 | 3.367312851812096e-5 |
| Standard deviation | 1.5552406700165978e-7 | 1.888659488099882e-7 | 2.3378335391346784e-7 |
Outlying measurements have no (7.142487448889722e-3%) effect on estimated standard deviation.
50% dups/sort . nub 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.521002876084381e-5 | 3.528044965820694e-5 | 3.5431171451194184e-5 |
| Standard deviation | 2.0859165344508945e-7 | 3.449596799905052e-7 | 6.635943227042994e-7 |
Outlying measurements have no (7.193866834698293e-3%) effect on estimated standard deviation.
50% dups/nub . sort 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.941434976287914e-5 | 3.950074696083045e-5 | 3.957527054358486e-5 |
| Standard deviation | 2.1909135879046403e-7 | 2.6596656049053246e-7 | 3.257714442528348e-7 |
Outlying measurements have no (7.298875432525926e-3%) effect on estimated standard deviation.
40% dups/sort . nub 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.884778673951425e-5 | 3.8893621480776694e-5 | 3.895924739643769e-5 |
| Standard deviation | 1.3714997226719682e-7 | 1.8572647773115502e-7 | 3.0373781522115793e-7 |
Outlying measurements have no (7.2988754325257815e-3%) effect on estimated standard deviation.
40% dups/nub . sort 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.123101379390096e-5 | 4.1308562470794934e-5 | 4.1423025580294115e-5 |
| Standard deviation | 2.504170460726143e-7 | 3.2710542342978186e-7 | 4.4715759299542766e-7 |
Outlying measurements have no (7.352537722908001e-3%) effect on estimated standard deviation.
30% dups/sort . nub 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.106901995892203e-5 | 4.118408384721611e-5 | 4.1507460105170076e-5 |
| Standard deviation | 1.795999482061194e-7 | 5.833666122817347e-7 | 1.1584404640865124e-6 |
Outlying measurements have slight (9.094719152008954e-2%) effect on estimated standard deviation.
30% dups/nub . sort 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.3466312306134364e-5 | 4.354326668238476e-5 | 4.3634670866955e-5 |
| Standard deviation | 2.2889987893183593e-7 | 2.699695636666835e-7 | 3.2528784029707714e-7 |
Outlying measurements have no (7.406994876364472e-3%) effect on estimated standard deviation.
25% dups/sort . nub 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.3644460543275944e-5 | 4.371746462993035e-5 | 4.381290468115428e-5 |
| Standard deviation | 2.2032996473511694e-7 | 2.685427863064816e-7 | 3.47625295777363e-7 |
Outlying measurements have no (7.406994876364403e-3%) effect on estimated standard deviation.
25% dups/nub . sort 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.9290250844674904e-5 | 4.940153242174351e-5 | 4.960492967876719e-5 |
| Standard deviation | 3.462950487164233e-7 | 5.096690646155473e-7 | 7.419889743762181e-7 |
Outlying measurements have no (7.575316123769107e-3%) effect on estimated standard deviation.
20% dups/sort . nub 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.82734754911963e-5 | 4.836359620704324e-5 | 4.8453763897375164e-5 |
| Standard deviation | 2.579621524525003e-7 | 3.025776543038029e-7 | 3.68318478343286e-7 |
Outlying measurements have no (7.518365472910756e-3%) effect on estimated standard deviation.
20% dups/nub . sort 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.968151032110808e-5 | 4.976561193571652e-5 | 4.9832049917085114e-5 |
| Standard deviation | 1.9582721266648186e-7 | 2.3960375310150606e-7 | 2.9446715010505665e-7 |
Outlying measurements have no (7.5753161237689945e-3%) effect on estimated standard deviation.
10% dups/sort . nub 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.1354852558320616e-5 | 5.14520162536307e-5 | 5.157882718282019e-5 |
| Standard deviation | 3.0745420286215213e-7 | 3.8947948408248126e-7 | 4.921278620968e-7 |
Outlying measurements have no (7.575316123768877e-3%) effect on estimated standard deviation.
10% dups/nub . sort 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.894385692861346e-5 | 4.903844620769715e-5 | 4.9143946917066656e-5 |
| Standard deviation | 2.6364517040876143e-7 | 3.2995117548791405e-7 | 4.2595793143486533e-7 |
Outlying measurements have no (7.518365472910774e-3%) effect on estimated standard deviation.
5% dups/sort . nub 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.1802551215463705e-5 | 5.186665123553926e-5 | 5.194206464723838e-5 |
| Standard deviation | 1.9641181943710368e-7 | 2.3748256444019727e-7 | 3.0183760868310736e-7 |
Outlying measurements have no (7.63313609467447e-3%) effect on estimated standard deviation.
5% dups/nub . sort 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.304995893245247e-5 | 5.3151128808515096e-5 | 5.3296950583084916e-5 |
| Standard deviation | 3.0014671384750563e-7 | 4.192892579590039e-7 | 6.753731601341841e-7 |
Outlying measurements have no (7.633136094674264e-3%) effect on estimated standard deviation.
1% dups/sort . nub 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.521497249095303e-5 | 5.529339407027402e-5 | 5.537308821635711e-5 |
| Standard deviation | 2.220658984229996e-7 | 2.6108834559743324e-7 | 3.1301771693693883e-7 |
Outlying measurements have no (7.691845441980522e-3%) effect on estimated standard deviation.
1% dups/nub . sort 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.5627679578530745e-5 | 5.569722767274859e-5 | 5.576965551790014e-5 |
| Standard deviation | 1.9029048561696877e-7 | 2.316177822886978e-7 | 2.899493574276263e-7 |
Outlying measurements have no (7.691845441980759e-3%) effect on estimated standard deviation.
0% dups/sort . nub 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.5811934077088624e-5 | 5.597102102518009e-5 | 5.623239528921857e-5 |
| Standard deviation | 4.844953018636126e-7 | 6.786191991782874e-7 | 9.481075433128967e-7 |
Outlying measurements have slight (6.575683503943773e-2%) effect on estimated standard deviation.
0% dups/nub . sort 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.582092957949801e-5 | 5.592591847157896e-5 | 5.6116345185376955e-5 |
| Standard deviation | 3.145804770382843e-7 | 4.6803603595465724e-7 | 8.432393694163816e-7 |
Outlying measurements have no (7.69184544198063e-3%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.