criterion performance measurements
overview
want to understand this report?
Words l=6-12/levenshteinMax (1)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.094987995779043e-7 | 5.12047419274794e-7 | 5.146096727084768e-7 |
| Standard deviation | 9.063101526091535e-9 | 1.1080595177695315e-8 | 1.3743448721305266e-8 |
Outlying measurements have moderate (0.28858497084372264%) effect on estimated standard deviation.
Words l=6-12/levenshteinMaxO (1)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.6130322254684783e-7 | 4.638786512435241e-7 | 4.6757500384496316e-7 |
| Standard deviation | 9.006095390923346e-9 | 1.199198409083218e-8 | 1.733226515125828e-8 |
Outlying measurements have moderate (0.37144565484818426%) effect on estimated standard deviation.
Words l=6-12/levenshteinMax (2)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.72828968914981e-7 | 7.771372845948626e-7 | 7.81814915214546e-7 |
| Standard deviation | 1.1476206993813018e-8 | 1.85285456638277e-8 | 2.7942935031797885e-8 |
Outlying measurements have moderate (0.32409526601610733%) effect on estimated standard deviation.
Words l=6-12/levenshteinMaxO (2)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.276034437573004e-7 | 7.301493930482189e-7 | 7.33480582348211e-7 |
| Standard deviation | 1.0421013244925857e-8 | 1.1993877746149059e-8 | 1.3540335835643387e-8 |
Outlying measurements have moderate (0.1845750906832196%) effect on estimated standard deviation.
Words l=6-12/levenshteinMax (3)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.2961468496194909e-6 | 1.3014158712908436e-6 | 1.3076742528659427e-6 |
| Standard deviation | 2.3883131393121512e-8 | 2.6761774595195698e-8 | 2.9612308710049455e-8 |
Outlying measurements have moderate (0.2519626146317031%) effect on estimated standard deviation.
Words l=6-12/levenshteinMaxO (3)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.1401290580910864e-6 | 1.1442552610975473e-6 | 1.1507050822571032e-6 |
| Standard deviation | 1.7700453619795466e-8 | 2.3240719485544236e-8 | 3.186371241716856e-8 |
Outlying measurements have moderate (0.252684694515236%) effect on estimated standard deviation.
Words l=6-12/levenshteinMax (4)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.3047211386533976e-6 | 1.3088431846441061e-6 | 1.312706712524239e-6 |
| Standard deviation | 1.4779881846623868e-8 | 1.744112338485496e-8 | 2.0285848899949218e-8 |
Outlying measurements have moderate (0.12486834988505864%) effect on estimated standard deviation.
Words l=6-12/levenshteinMaxO (4)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.1556035757250523e-6 | 1.159792906280762e-6 | 1.1662758430585339e-6 |
| Standard deviation | 1.541084236445498e-8 | 2.0268737727442e-8 | 2.459093231150613e-8 |
Outlying measurements have moderate (0.19731292584552335%) effect on estimated standard deviation.
Words l=6-12/levenshtein
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.891461130497558e-6 | 2.9045651438729805e-6 | 2.9212511472187948e-6 |
| Standard deviation | 3.657686279051383e-8 | 5.919746654007349e-8 | 7.910509482826237e-8 |
Outlying measurements have moderate (0.23402095305966014%) effect on estimated standard deviation.
Words l=6-18/levenshteinMax (1)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.093908831415694e-7 | 5.113318082209661e-7 | 5.143088853563764e-7 |
| Standard deviation | 6.845229031974509e-9 | 8.150741735514652e-9 | 1.0278055397725646e-8 |
Outlying measurements have moderate (0.179163093540656%) effect on estimated standard deviation.
Words l=6-18/levenshteinMaxO (1)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.727289196592301e-7 | 4.77359446686713e-7 | 4.85593025537201e-7 |
| Standard deviation | 1.1110824007754965e-8 | 2.30567976446259e-8 | 3.938347252403591e-8 |
Outlying measurements have severe (0.6756117410036199%) effect on estimated standard deviation.
Words l=6-18/levenshteinMax (2)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.833477155547966e-7 | 7.860402022241628e-7 | 7.89125901997458e-7 |
| Standard deviation | 9.709294441498233e-9 | 1.1420870272379538e-8 | 1.3390425595799925e-8 |
Outlying measurements have moderate (0.1486379928348229%) effect on estimated standard deviation.
Words l=6-18/levenshteinMaxO (2)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.265481129008179e-7 | 7.30369326791746e-7 | 7.346493966553628e-7 |
| Standard deviation | 8.91496789528264e-9 | 1.4081951238594874e-8 | 1.9663409338836688e-8 |
Outlying measurements have moderate (0.2386767410831963%) effect on estimated standard deviation.
Words l=6-18/levenshteinMax (3)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.0437849697647802e-6 | 1.0474953893574155e-6 | 1.0521261934441389e-6 |
| Standard deviation | 1.2608573623293616e-8 | 1.4229251243386834e-8 | 1.7806688619004076e-8 |
Outlying measurements have moderate (0.13063386729328608%) effect on estimated standard deviation.
Words l=6-18/levenshteinMaxO (3)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.34231019015816e-7 | 9.367129120445083e-7 | 9.413320259405648e-7 |
| Standard deviation | 7.0813312393672645e-9 | 1.1144425374153086e-8 | 1.810500109452167e-8 |
Outlying measurements have moderate (0.1042339500386065%) effect on estimated standard deviation.
Words l=6-18/levenshteinMax (4)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.3166693074719318e-6 | 1.3213056595420056e-6 | 1.3268421916287142e-6 |
| Standard deviation | 2.00475895457818e-8 | 2.2792161700316953e-8 | 2.8166534145877957e-8 |
Outlying measurements have moderate (0.1904524290983281%) effect on estimated standard deviation.
Words l=6-18/levenshteinMaxO (4)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.4368874698306042e-6 | 1.4397065655243028e-6 | 1.4422354158747286e-6 |
| Standard deviation | 8.107185891033322e-9 | 9.655836469137856e-9 | 1.1463979033806e-8 |
Outlying measurements have no (4.566113963470859e-3%) effect on estimated standard deviation.
Words l=6-18/levenshtein
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.396014610654096e-6 | 4.410390615270599e-6 | 4.423689208669972e-6 |
| Standard deviation | 3.5717268092551135e-8 | 5.096991146595085e-8 | 6.71971955902648e-8 |
Outlying measurements have slight (8.719573104253056e-2%) effect on estimated standard deviation.
Words l=12-18/levenshteinMax (1)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.431347058485031e-7 | 9.463087387465415e-7 | 9.529536223845246e-7 |
| Standard deviation | 1.3094621341173746e-8 | 1.7230765911496702e-8 | 2.4962160680977002e-8 |
Outlying measurements have moderate (0.21419013293470707%) effect on estimated standard deviation.
Words l=12-18/levenshteinMaxO (1)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.484535233356154e-7 | 8.511800583547641e-7 | 8.544497338764325e-7 |
| Standard deviation | 8.353698711794795e-9 | 9.932245606156598e-9 | 1.2588798380018643e-8 |
Outlying measurements have moderate (0.10319800620906494%) effect on estimated standard deviation.
Words l=12-18/levenshteinMax (2)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.3908153440721768e-6 | 1.3941518547506313e-6 | 1.3987747890835833e-6 |
| Standard deviation | 1.1342351401667951e-8 | 1.5222239691839626e-8 | 2.0822375342337428e-8 |
Outlying measurements have slight (8.633293483399053e-2%) effect on estimated standard deviation.
Words l=12-18/levenshteinMaxO (2)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.3515481416151932e-6 | 1.3542049018464726e-6 | 1.3603941669848435e-6 |
| Standard deviation | 1.079447462411921e-8 | 1.4278751652134445e-8 | 1.9478707054634116e-8 |
Outlying measurements have slight (8.192759008238043e-2%) effect on estimated standard deviation.
Words l=12-18/levenshteinMax (3)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.9098004595312514e-6 | 1.913692572199834e-6 | 1.9187149416402877e-6 |
| Standard deviation | 1.1656631111441305e-8 | 1.6005271120647826e-8 | 2.4937020575539823e-8 |
Outlying measurements have no (4.694731221075059e-3%) effect on estimated standard deviation.
Words l=12-18/levenshteinMaxO (3)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.245497886978052e-6 | 2.2507503934591177e-6 | 2.256028501719549e-6 |
| Standard deviation | 1.811150565048708e-8 | 2.3657533855075984e-8 | 3.133407871217825e-8 |
Outlying measurements have slight (7.686263410806271e-2%) effect on estimated standard deviation.
Words l=12-18/levenshteinMax (4)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.4407506239871247e-6 | 2.4589256303854897e-6 | 2.492850829503526e-6 |
| Standard deviation | 5.082157923080107e-8 | 9.394071033743994e-8 | 1.7417183729677424e-7 |
Outlying measurements have severe (0.5259227228729124%) effect on estimated standard deviation.
Words l=12-18/levenshteinMaxO (4)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.210835513489359e-6 | 2.2278203316554856e-6 | 2.246155956884535e-6 |
| Standard deviation | 3.292641190929456e-8 | 5.8402610728639657e-8 | 8.264659425118022e-8 |
Outlying measurements have moderate (0.3448429210038298%) effect on estimated standard deviation.
Words l=12-18/levenshteinMax (5)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.918267273882085e-6 | 2.9360418652084485e-6 | 2.95313087124345e-6 |
| Standard deviation | 4.454291352978231e-8 | 6.292965459144917e-8 | 9.364448782413098e-8 |
Outlying measurements have moderate (0.2541539878086313%) effect on estimated standard deviation.
Words l=12-18/levenshteinMaxO (5)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.7432391353413285e-6 | 2.7609658585376507e-6 | 2.796065156661712e-6 |
| Standard deviation | 6.849207731011825e-8 | 8.587636188021735e-8 | 1.1347619217107081e-7 |
Outlying measurements have moderate (0.4192967928025129%) effect on estimated standard deviation.
Words l=12-18/levenshteinMax (6)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.918577915415165e-6 | 3.936670946542688e-6 | 3.960988440692238e-6 |
| Standard deviation | 6.792035614020313e-8 | 9.83224387443597e-8 | 1.3982815260583e-7 |
Outlying measurements have moderate (0.3114536611244236%) effect on estimated standard deviation.
Words l=12-18/levenshteinMaxO (6)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.309190414925994e-6 | 4.325199947436605e-6 | 4.364093371003563e-6 |
| Standard deviation | 6.1471328180886e-8 | 8.572701737892059e-8 | 1.205290944459127e-7 |
Outlying measurements have moderate (0.219907021399332%) effect on estimated standard deviation.
Words l=12-18/levenshteinMax (7)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.935123527069053e-6 | 3.960192676602097e-6 | 4.024173124114128e-6 |
| Standard deviation | 6.601874216320978e-8 | 1.4599819865984434e-7 | 2.500322365675517e-7 |
Outlying measurements have moderate (0.4973511898442842%) effect on estimated standard deviation.
Words l=12-18/levenshteinMaxO (7)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.6948530895024855e-6 | 4.710875261102436e-6 | 4.746419632979524e-6 |
| Standard deviation | 5.746187496672835e-8 | 8.298511061646039e-8 | 1.243634171314583e-7 |
Outlying measurements have moderate (0.1812805811867173%) effect on estimated standard deviation.
Words l=12-18/levenshteinMax (8)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.863362828948513e-6 | 5.892142533135247e-6 | 5.943405230006469e-6 |
| Standard deviation | 1.2938811188391342e-7 | 1.585425464914178e-7 | 2.026007463947298e-7 |
Outlying measurements have moderate (0.33528192281617%) effect on estimated standard deviation.
Words l=12-18/levenshteinMaxO (8)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.7100529039056095e-6 | 4.722111231250186e-6 | 4.736160804040995e-6 |
| Standard deviation | 4.4486882668316925e-8 | 5.272671738308397e-8 | 6.775934335753044e-8 |
Outlying measurements have slight (8.266838674149862e-2%) effect on estimated standard deviation.
Words l=12-18/levenshtein
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.637410585839664e-6 | 8.661773819407198e-6 | 8.719757346344086e-6 |
| Standard deviation | 6.606343311709919e-8 | 1.4865753006120628e-7 | 2.2391341746433267e-7 |
Outlying measurements have moderate (0.16267370605974574%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.