Benchmarking GHC's Integer Libraries
overview
GHC has two official Integer libraries, integer-gmp and integer-simple. This report shows their performance in comparison to a new library that aims to compete.
These tests are being carried out with GHC 7.10.3 and the git HEAD versions of integer-gmp and integer-simple on x86_64/linux.
New1, New2, New3 and New4 use slightly different definitions of the Integer data structure to investigate how much difference that makes.
The Small Integer benchmarks work on Integer values where the input values and the result fit within a single machine word (eg 32 bits on 32 bit CPUs, 64 bits on 64 bit CPUs). The Big Integer benchmarks operate on values that require multiple machine words.
want to understand this report?
Small Integer addition and subtraction/GMP
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.9508501952839906e-5 | 1.9598054477940657e-5 | 1.9694145317858426e-5 |
| Standard deviation | 2.418107127205221e-7 | 2.912506794193717e-7 | 3.646219864090966e-7 |
Outlying measurements have moderate (0.10727523778901105%) effect on estimated standard deviation.
Small Integer addition and subtraction/Simple
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.940440464461906e-5 | 6.962834626645953e-5 | 6.984458871093858e-5 |
| Standard deviation | 5.745514052531572e-7 | 6.938370809997472e-7 | 9.280069220016786e-7 |
Outlying measurements have no (7.999479708636577e-3%) effect on estimated standard deviation.
Small Integer addition and subtraction/New1
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.286915459950388e-5 | 3.299210349611197e-5 | 3.312308326780057e-5 |
| Standard deviation | 3.363154505440383e-7 | 4.283723556807698e-7 | 6.682618569741471e-7 |
Outlying measurements have slight (8.085048748921865e-2%) effect on estimated standard deviation.
Small Integer addition and subtraction/New2
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.217509830678664e-5 | 3.229138853099786e-5 | 3.238421544456759e-5 |
| Standard deviation | 2.7843031162252867e-7 | 3.4946330357743255e-7 | 4.880582859847416e-7 |
Outlying measurements have slight (5.348905241000567e-2%) effect on estimated standard deviation.
Small Integer addition and subtraction/New3
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.7601126211585917e-5 | 1.8096350578131945e-5 | 1.921439544341538e-5 |
| Standard deviation | 1.0548018737631428e-6 | 2.385251427764402e-6 | 4.466550952868594e-6 |
Outlying measurements have severe (0.9095256517950608%) effect on estimated standard deviation.
Small Integer addition and subtraction/New4
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.131332121174682e-5 | 3.143440070932379e-5 | 3.153933749624994e-5 |
| Standard deviation | 3.19881936583018e-7 | 3.8919587267387964e-7 | 4.949332400954093e-7 |
Outlying measurements have slight (7.361449542075751e-2%) effect on estimated standard deviation.
Big Integer addition and subtraction/GMP
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.909777910671559e-5 | 7.931041500889632e-5 | 7.986678555639738e-5 |
| Standard deviation | 4.613230376165756e-7 | 1.09671262196937e-6 | 2.2989684716744174e-6 |
Outlying measurements have slight (7.746718429324297e-2%) effect on estimated standard deviation.
Big Integer addition and subtraction/Simple
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.004232588599528e-4 | 3.0124149441560866e-4 | 3.020772333968669e-4 |
| Standard deviation | 2.4670380088752667e-6 | 2.909911971746779e-6 | 3.7691396998924828e-6 |
Outlying measurements have slight (1.0415512465374024e-2%) effect on estimated standard deviation.
Big Integer addition and subtraction/New1
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.2413092911306555e-4 | 2.2459160974435234e-4 | 2.250275095995692e-4 |
| Standard deviation | 1.2277151211700408e-6 | 1.493650030300162e-6 | 2.18089045921849e-6 |
Outlying measurements have no (9.802960494069169e-3%) effect on estimated standard deviation.
Big Integer addition and subtraction/New2
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.7377857387333107e-4 | 1.7432703597805541e-4 | 1.7490859993159148e-4 |
| Standard deviation | 1.5278497156362357e-6 | 1.796435770520288e-6 | 2.154902794752755e-6 |
Outlying measurements have no (9.344962620149552e-3%) effect on estimated standard deviation.
Big Integer addition and subtraction/New3
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.11495636122488e-5 | 8.144128670376391e-5 | 8.166566833283577e-5 |
| Standard deviation | 6.565462653866341e-7 | 8.67197224078203e-7 | 1.1237996284038624e-6 |
Outlying measurements have no (8.19616146438101e-3%) effect on estimated standard deviation.
Big Integer addition and subtraction/New4
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.837426844361084e-5 | 8.862208999603015e-5 | 8.885788025180083e-5 |
| Standard deviation | 6.685704259806643e-7 | 8.196917557213248e-7 | 1.1100632275880652e-6 |
Outlying measurements have no (8.332744862650645e-3%) effect on estimated standard deviation.
Small Integer multiplication/GMP
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.889910168630509e-5 | 5.905954053905368e-5 | 5.924931192823258e-5 |
| Standard deviation | 4.810783792963639e-7 | 5.537943747795584e-7 | 6.860460320558393e-7 |
Outlying measurements have no (7.751464843750134e-3%) effect on estimated standard deviation.
Small Integer multiplication/Simple
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.826413106665565e-4 | 2.851960524088455e-4 | 2.9279617972844307e-4 |
| Standard deviation | 3.605508590646345e-6 | 1.2920708925999783e-5 | 2.489654930738116e-5 |
Outlying measurements have moderate (0.4223218359832151%) effect on estimated standard deviation.
Small Integer multiplication/New1
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.2039579693767128e-4 | 1.208381784221025e-4 | 1.211889319111599e-4 |
| Standard deviation | 1.037076815894393e-6 | 1.3157751786258664e-6 | 1.7244510800244794e-6 |
Outlying measurements have no (8.771242853786573e-3%) effect on estimated standard deviation.
Small Integer multiplication/New2
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.0756416859923507e-4 | 1.0805155028579465e-4 | 1.094393088726101e-4 |
| Standard deviation | 1.058377799940141e-6 | 2.5172385321694598e-6 | 5.224851428212607e-6 |
Outlying measurements have moderate (0.18998628009083013%) effect on estimated standard deviation.
Small Integer multiplication/New3
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.704272713507808e-5 | 4.7498517739598194e-5 | 4.9020819316584705e-5 |
| Standard deviation | 1.2110297953201015e-6 | 2.544043232770854e-6 | 5.119561306396279e-6 |
Outlying measurements have severe (0.5837052187380394%) effect on estimated standard deviation.
Small Integer multiplication/New4
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.117802141597239e-5 | 8.184982275145291e-5 | 8.36526863799446e-5 |
| Standard deviation | 1.2761072582063457e-6 | 3.5638707771161867e-6 | 7.005029062465823e-6 |
Outlying measurements have moderate (0.45595948233347783%) effect on estimated standard deviation.
Small-Big Integer multiplication/GMP
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.640589581761487e-5 | 3.6517723312763345e-5 | 3.6614743062162316e-5 |
| Standard deviation | 2.782728943408854e-7 | 3.5528609200233e-7 | 4.7460657278801724e-7 |
Outlying measurements have no (7.193866834698507e-3%) effect on estimated standard deviation.
Small-Big Integer multiplication/Simple
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.332579687558128e-4 | 3.345784981291447e-4 | 3.3864521482500224e-4 |
| Standard deviation | 2.481529158846257e-6 | 6.966380615174672e-6 | 1.4110978020812381e-5 |
Outlying measurements have moderate (0.13348222005792434%) effect on estimated standard deviation.
Small-Big Integer multiplication/New1
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.4604351007690898e-4 | 1.4636679948284992e-4 | 1.467279622590302e-4 |
| Standard deviation | 9.210104040076105e-7 | 1.194305964806162e-6 | 1.4541748639160362e-6 |
Outlying measurements have no (9.090143927278808e-3%) effect on estimated standard deviation.
Small-Big Integer multiplication/New2
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.5163935684198442e-4 | 1.5213740221029152e-4 | 1.5309284065379063e-4 |
| Standard deviation | 1.524047088307656e-6 | 2.336079362338861e-6 | 3.950352982779207e-6 |
Outlying measurements have slight (8.733186799683637e-2%) effect on estimated standard deviation.
Small-Big Integer multiplication/New3
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.937764305597335e-5 | 2.957274337266762e-5 | 2.9819310742341764e-5 |
| Standard deviation | 5.932179923737864e-7 | 7.332695375619711e-7 | 9.404152658961238e-7 |
Outlying measurements have moderate (0.2420179991725011%) effect on estimated standard deviation.
Small-Big Integer multiplication/New4
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.195168333816589e-5 | 3.323522593541996e-5 | 3.466715689624688e-5 |
| Standard deviation | 3.2583874370643987e-6 | 4.361380658715394e-6 | 5.497875868891006e-6 |
Outlying measurements have severe (0.9018095985067999%) effect on estimated standard deviation.
Big Integer multiplication (dropped Simple, too slow)/GMP
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.586732158082434e-3 | 4.7324645741900736e-3 | 5.082821388322336e-3 |
| Standard deviation | 3.9490607937238436e-4 | 6.546744223101139e-4 | 1.0191877995028293e-3 |
Outlying measurements have severe (0.7506199036755055%) effect on estimated standard deviation.
Big Integer multiplication (dropped Simple, too slow)/Simple
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.204451479286109e-9 | 7.2776557967373095e-9 | 7.412624586832105e-9 |
| Standard deviation | 1.831634872850904e-10 | 3.251043593547767e-10 | 4.918453189732454e-10 |
Outlying measurements have severe (0.6988327313234477%) effect on estimated standard deviation.
Big Integer multiplication (dropped Simple, too slow)/New1
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.4921401762397825e-2 | 4.523467146211691e-2 | 4.5675507088933705e-2 |
| Standard deviation | 3.892392859144249e-4 | 7.497942879376458e-4 | 9.857437923540208e-4 |
Outlying measurements have slight (6.632653061224478e-2%) effect on estimated standard deviation.
Big Integer multiplication (dropped Simple, too slow)/New2
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.574736554703078e-2 | 4.620638088258441e-2 | 4.71101615702036e-2 |
| Standard deviation | 6.997120552727267e-4 | 1.2395659108682682e-3 | 1.9757699682176615e-3 |
Outlying measurements have slight (6.632653061224489e-2%) effect on estimated standard deviation.
Big Integer multiplication (dropped Simple, too slow)/New3
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.3827034480957885e-2 | 2.402377406908e-2 | 2.4326594187405022e-2 |
| Standard deviation | 3.763307425985407e-4 | 5.496842383095563e-4 | 8.346837812307599e-4 |
Outlying measurements have slight (4.986149584487534e-2%) effect on estimated standard deviation.
Big Integer multiplication (dropped Simple, too slow)/New4
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.9268130761211044e-2 | 3.961356442844948e-2 | 4.022744958085779e-2 |
| Standard deviation | 4.95266881206096e-4 | 8.95378338683568e-4 | 1.3862301237334291e-3 |
Outlying measurements have slight (6.222222222222223e-2%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.