Benchmarking GHC's Integer Libraries
overview
GHC has two official Integer libraries, integer-gmp and integer-simple. This report shows their performance in comparison to a new library that aims to compete.
These tests are being carried out with ghc 7.8.3 and the git HEAD versions of integer-gmp and integer-simple.
New1, New2, New3 and New4 use slightly different definitions of the Integer data structure to investigate how much difference that makes.
want to understand this report?
Small Integer addition and subtraction/GMP
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.5010626990506646e-5 | 2.509793170763304e-5 | 2.5363324296708676e-5 |
| Standard deviation | 1.665398467296502e-7 | 4.618222590902813e-7 | 9.204550358744712e-7 |
Outlying measurements have moderate (0.15061317328745327%) effect on estimated standard deviation.
Small Integer addition and subtraction/Simple
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.211640095387577e-5 | 8.22164759824359e-5 | 8.23411131738838e-5 |
| Standard deviation | 2.7497974818366785e-7 | 3.669730840916224e-7 | 5.366949801145903e-7 |
Outlying measurements have no (8.196161464380609e-3%) effect on estimated standard deviation.
Small Integer addition and subtraction/New1
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.125906501242458e-5 | 4.1319091770531565e-5 | 4.139985105070127e-5 |
| Standard deviation | 1.833079748860793e-7 | 2.3008596252023703e-7 | 2.908178161877941e-7 |
Outlying measurements have no (7.352537722907905e-3%) effect on estimated standard deviation.
Small Integer addition and subtraction/New2
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.440670866328509e-5 | 4.445487730719241e-5 | 4.452947969312423e-5 |
| Standard deviation | 1.453642595275738e-7 | 2.0410624086320996e-7 | 3.286830250943545e-7 |
Outlying measurements have no (7.406994876364469e-3%) effect on estimated standard deviation.
Small Integer addition and subtraction/New3
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.6894250513306927e-5 | 2.6920474083593922e-5 | 2.6949710834042438e-5 |
| Standard deviation | 8.024966708962819e-8 | 9.536410332926305e-8 | 1.1854498479721743e-7 |
Outlying measurements have no (6.896219135802422e-3%) effect on estimated standard deviation.
Small Integer addition and subtraction/New4
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.3655395130653506e-5 | 4.373041298055046e-5 | 4.382066729457033e-5 |
| Standard deviation | 2.144477730091516e-7 | 2.838762696876709e-7 | 3.5901117106912826e-7 |
Outlying measurements have no (7.406994876364392e-3%) effect on estimated standard deviation.
Big Integer addition and subtraction/GMP
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.002656398808534e-5 | 8.016538641796328e-5 | 8.035461099489483e-5 |
| Standard deviation | 3.821759202028868e-7 | 5.484627819439364e-7 | 8.892946240453947e-7 |
Outlying measurements have no (8.196161464380845e-3%) effect on estimated standard deviation.
Big Integer addition and subtraction/Simple
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.407993379150091e-4 | 3.414033798536703e-4 | 3.421118279340613e-4 |
| Standard deviation | 1.8142934939519513e-6 | 2.251687485525029e-6 | 3.111125657238723e-6 |
Outlying measurements have slight (1.0751417769376189e-2%) effect on estimated standard deviation.
Big Integer addition and subtraction/New1
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.304739014081192e-4 | 1.3070115712286026e-4 | 1.3092268340900122e-4 |
| Standard deviation | 6.517866185690538e-7 | 7.701500041655928e-7 | 9.352886821131784e-7 |
Outlying measurements have no (8.927846765684462e-3%) effect on estimated standard deviation.
Big Integer addition and subtraction/New2
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.0560680021865639e-4 | 1.060673745636221e-4 | 1.0652238403925204e-4 |
| Standard deviation | 1.3501545484729153e-6 | 1.5928455419052046e-6 | 1.8970741038882815e-6 |
Outlying measurements have slight (8.945667789916115e-2%) effect on estimated standard deviation.
Big Integer addition and subtraction/New3
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.51649415080561e-5 | 9.54617383259212e-5 | 9.587881357014858e-5 |
| Standard deviation | 7.955761360569145e-7 | 1.1647678978905111e-6 | 1.6976188723036955e-6 |
Outlying measurements have slight (6.370839382798157e-2%) effect on estimated standard deviation.
Big Integer addition and subtraction/New4
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.805936740351089e-5 | 9.822613189931079e-5 | 9.842616496379695e-5 |
| Standard deviation | 4.906051270608268e-7 | 5.984144712763677e-7 | 7.67472810611457e-7 |
Outlying measurements have no (8.473957191906193e-3%) effect on estimated standard deviation.
Small Integer multiplication/GMP
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.274008620568709e-5 | 7.28184815807909e-5 | 7.292015145238549e-5 |
| Standard deviation | 2.239135440387453e-7 | 3.0150296317364503e-7 | 4.1722368271821177e-7 |
Outlying measurements have no (8.063983078855362e-3%) effect on estimated standard deviation.
Small Integer multiplication/Simple
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.201658703531416e-4 | 3.206261603723899e-4 | 3.2123300472691816e-4 |
| Standard deviation | 1.3801791242564591e-6 | 1.7991245641756331e-6 | 2.456926101109583e-6 |
Outlying measurements have slight (1.0637067869117665e-2%) effect on estimated standard deviation.
Small Integer multiplication/New1
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.3676324409030255e-4 | 1.368952827630646e-4 | 1.3704401524187676e-4 |
| Standard deviation | 3.876343487125905e-7 | 4.4860267064797097e-7 | 5.21928940097646e-7 |
Outlying measurements have no (8.92784676568448e-3%) effect on estimated standard deviation.
Small Integer multiplication/New2
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.5400809221879883e-4 | 1.5418550647963518e-4 | 1.5440847625348647e-4 |
| Standard deviation | 5.393561058053353e-7 | 6.766780144411008e-7 | 9.352719202193893e-7 |
Outlying measurements have no (9.17352537722908e-3%) effect on estimated standard deviation.
Small Integer multiplication/New3
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.509303610321351e-5 | 7.520286119894468e-5 | 7.532377432887599e-5 |
| Standard deviation | 3.1001491709040595e-7 | 3.825060414476205e-7 | 4.966369908787442e-7 |
Outlying measurements have no (8.063983078855284e-3%) effect on estimated standard deviation.
Small Integer multiplication/New4
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.0372880688970595e-4 | 1.0391412010337194e-4 | 1.0413801484006541e-4 |
| Standard deviation | 5.528799682929522e-7 | 6.962111126072613e-7 | 9.14661296476633e-7 |
Outlying measurements have no (8.546373365041484e-3%) effect on estimated standard deviation.
Big Integer multiplication (dropped Simple, too slow)/GMP
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.00345591792732e-3 | 5.010552474951389e-3 | 5.020264266350114e-3 |
| Standard deviation | 1.94172204664026e-5 | 2.533928327029626e-5 | 3.5229188647373816e-5 |
Outlying measurements have slight (2.3795359904818562e-2%) effect on estimated standard deviation.
Big Integer multiplication (dropped Simple, too slow)/Simple
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.540878167078916e-9 | 8.55895919160916e-9 | 8.592328180281072e-9 |
| Standard deviation | 4.727936722081037e-11 | 8.25496665240366e-11 | 1.5280391920003871e-10 |
Outlying measurements have slight (9.485246680174772e-2%) effect on estimated standard deviation.
Big Integer multiplication (dropped Simple, too slow)/New1
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.598523354948584e-2 | 4.608881634279361e-2 | 4.623866769986433e-2 |
| Standard deviation | 1.3971565512837643e-4 | 2.2371598172586868e-4 | 3.5484727043105574e-4 |
Outlying measurements have slight (6.632653061224482e-2%) effect on estimated standard deviation.
Big Integer multiplication (dropped Simple, too slow)/New2
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.5925308357504996e-2 | 4.60076198603393e-2 | 4.6114248627183614e-2 |
| Standard deviation | 1.1718623878170662e-4 | 1.7868740389431496e-4 | 2.8733414153774424e-4 |
Outlying measurements have slight (6.632653061224483e-2%) effect on estimated standard deviation.
Big Integer multiplication (dropped Simple, too slow)/New3
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.584701892995844e-2 | 2.5888308990959126e-2 | 2.5937606622292417e-2 |
| Standard deviation | 8.030620576190414e-5 | 1.0521095855560344e-4 | 1.4284576978483617e-4 |
Outlying measurements have slight (4.986149584487534e-2%) effect on estimated standard deviation.
Big Integer multiplication (dropped Simple, too slow)/New4
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.109981626659988e-2 | 4.138886583469803e-2 | 4.229248587415979e-2 |
| Standard deviation | 2.5450296310621217e-4 | 9.160178214166327e-4 | 1.6621622972288262e-3 |
Outlying measurements have slight (6.222222222222219e-2%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.