Benchmarking GHC's Integer Libraries
overview
GHC has two official Integer libraries, integer-gmp and integer-simple. This report shows their performance in comparison to a new library that aims to compete.
These tests are being carried out with ghc 7.8.3 and the git HEAD versions of integer-gmp and integer-simple.
New1, New2, New3 and New4 use slightly different definitions of the Integer data structure to investigate how much difference that makes.
want to understand this report?
Small Integer addition and subtraction/GMP
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.7514997327062937e-5 | 1.7640341415710276e-5 | 1.7934254427329073e-5 |
| Standard deviation | 2.1332208374055706e-7 | 5.792332881863418e-7 | 1.1196438237036846e-6 |
Outlying measurements have moderate (0.3737930732605032%) effect on estimated standard deviation.
Small Integer addition and subtraction/Simple
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.5310635921048544e-5 | 5.5477510767358575e-5 | 5.573571155984019e-5 |
| Standard deviation | 5.157093043308087e-7 | 6.813779756837666e-7 | 8.897428652371317e-7 |
Outlying measurements have slight (6.585908992781132e-2%) effect on estimated standard deviation.
Small Integer addition and subtraction/New1
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.8767343789457278e-5 | 2.884684527258814e-5 | 2.8969890698731547e-5 |
| Standard deviation | 2.6712294027125626e-7 | 3.3133211948483457e-7 | 4.5809737723961807e-7 |
Outlying measurements have slight (6.628611581616026e-2%) effect on estimated standard deviation.
Small Integer addition and subtraction/New2
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.886077266244053e-5 | 2.901704508415688e-5 | 2.9223883738575227e-5 |
| Standard deviation | 4.4811318434401943e-7 | 6.250011476455239e-7 | 1.027864294517977e-6 |
Outlying measurements have moderate (0.19433308743035926%) effect on estimated standard deviation.
Small Integer addition and subtraction/New3
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.924655717348622e-5 | 1.933351956234242e-5 | 1.9431583255567034e-5 |
| Standard deviation | 2.536593461307928e-7 | 3.144692835034666e-7 | 3.95697962406203e-7 |
Outlying measurements have moderate (0.12633375912496458%) effect on estimated standard deviation.
Small Integer addition and subtraction/New4
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.781468712984763e-5 | 2.7943514505489774e-5 | 2.815227326959249e-5 |
| Standard deviation | 3.818024113155517e-7 | 5.45468712768126e-7 | 7.677456578598636e-7 |
Outlying measurements have moderate (0.16606093415474202%) effect on estimated standard deviation.
Big Integer addition and subtraction/GMP
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.369126377513535e-5 | 3.3804431976503643e-5 | 3.3948776389175675e-5 |
| Standard deviation | 3.173094788249792e-7 | 4.1286413801220525e-7 | 5.364027406444386e-7 |
Outlying measurements have slight (7.441721272267726e-2%) effect on estimated standard deviation.
Big Integer addition and subtraction/Simple
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.45587894429478e-4 | 1.461352706432436e-4 | 1.4693635646612782e-4 |
| Standard deviation | 1.6617938360086846e-6 | 2.163901762659674e-6 | 3.023039612181347e-6 |
Outlying measurements have slight (8.624042292786656e-2%) effect on estimated standard deviation.
Big Integer addition and subtraction/New1
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.691370414124913e-5 | 5.7167139617072084e-5 | 5.757699123160413e-5 |
| Standard deviation | 7.773740616319971e-7 | 1.067785488361516e-6 | 1.556175508384997e-6 |
Outlying measurements have moderate (0.14072807095633752%) effect on estimated standard deviation.
Big Integer addition and subtraction/New2
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.316457756622487e-5 | 4.349158928384114e-5 | 4.428520167829888e-5 |
| Standard deviation | 8.909152106987711e-7 | 1.6695697477032276e-6 | 2.997257421647786e-6 |
Outlying measurements have moderate (0.41801190164417346%) effect on estimated standard deviation.
Big Integer addition and subtraction/New3
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.753555728048234e-5 | 3.772923016558623e-5 | 3.812574897023247e-5 |
| Standard deviation | 4.865514726239807e-7 | 9.183389568065321e-7 | 1.4348670095814645e-6 |
Outlying measurements have moderate (0.2297063521293558%) effect on estimated standard deviation.
Big Integer addition and subtraction/New4
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.864952039624626e-5 | 3.879167727462824e-5 | 3.896716686534869e-5 |
| Standard deviation | 4.01683327544462e-7 | 5.257451862286895e-7 | 7.208984510265147e-7 |
Outlying measurements have slight (8.336570521316707e-2%) effect on estimated standard deviation.
Small Integer multiplication/GMP
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.421720379663632e-5 | 7.482840176552621e-5 | 7.600796792712341e-5 |
| Standard deviation | 1.8524999679749822e-6 | 2.8624985392640303e-6 | 4.375961745614096e-6 |
Outlying measurements have moderate (0.3997100709190396%) effect on estimated standard deviation.
Small Integer multiplication/Simple
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.662248150736246e-4 | 2.677389881082302e-4 | 2.7008166058578847e-4 |
| Standard deviation | 4.692065307899623e-6 | 6.6162068084675186e-6 | 9.291722071859248e-6 |
Outlying measurements have moderate (0.17648347465824502%) effect on estimated standard deviation.
Small Integer multiplication/New1
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.3929488137260474e-4 | 1.4004332860035614e-4 | 1.420194758588076e-4 |
| Standard deviation | 1.909336822428332e-6 | 3.5970913320353752e-6 | 7.035008820167278e-6 |
Outlying measurements have moderate (0.2079327030087455%) effect on estimated standard deviation.
Small Integer multiplication/New2
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.591595139239573e-4 | 1.6019408813953067e-4 | 1.6195305511793715e-4 |
| Standard deviation | 2.81126411599615e-6 | 4.386441939092864e-6 | 7.563832444652051e-6 |
Outlying measurements have moderate (0.23149103160221277%) effect on estimated standard deviation.
Small Integer multiplication/New3
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.0173368008569251e-4 | 1.0217606391177535e-4 | 1.0278439670727498e-4 |
| Standard deviation | 1.2436149850023235e-6 | 1.7796372884001947e-6 | 2.574825379107786e-6 |
Outlying measurements have moderate (0.1133550246204539%) effect on estimated standard deviation.
Small Integer multiplication/New4
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.0667801509433486e-4 | 1.0695183099066484e-4 | 1.0732839536366174e-4 |
| Standard deviation | 8.537470039515456e-7 | 1.084274398444292e-6 | 1.4225049900170472e-6 |
Outlying measurements have no (8.546373365041507e-3%) effect on estimated standard deviation.
Big Integer multiplication (dropped Simple, too slow)/GMP
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.0508054186471675e-3 | 5.066428267443586e-3 | 5.087398470586369e-3 |
| Standard deviation | 4.2969465536696914e-5 | 5.6538887880872475e-5 | 7.722055136610735e-5 |
Outlying measurements have slight (2.3795359904818507e-2%) effect on estimated standard deviation.
Big Integer multiplication (dropped Simple, too slow)/New1
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.650215118872433e-2 | 4.664540139932031e-2 | 4.683817115669699e-2 |
| Standard deviation | 2.5150567186704525e-4 | 3.241281932725928e-4 | 4.8370894271006107e-4 |
Outlying measurements have slight (6.632653061224483e-2%) effect on estimated standard deviation.
Big Integer multiplication (dropped Simple, too slow)/New2
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.690816319843621e-2 | 4.719511717174225e-2 | 4.751735300192013e-2 |
| Standard deviation | 3.2770639584387865e-4 | 5.459266774436285e-4 | 8.581210651668843e-4 |
Outlying measurements have slight (6.632653061224489e-2%) effect on estimated standard deviation.
Big Integer multiplication (dropped Simple, too slow)/New3
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.1195033019870424e-2 | 4.133497658526373e-2 | 4.1475508310580486e-2 |
| Standard deviation | 2.0234273675531428e-4 | 2.696040787097232e-4 | 4.0848295425667547e-4 |
Outlying measurements have slight (6.222222222222218e-2%) effect on estimated standard deviation.
Big Integer multiplication (dropped Simple, too slow)/New4
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.17570980246601e-2 | 4.195468824202721e-2 | 4.2196237575646424e-2 |
| Standard deviation | 3.1774083245747324e-4 | 4.3158487855443465e-4 | 6.319238618498724e-4 |
Outlying measurements have slight (6.632653061224485e-2%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.