Not Even My Wife Reads This

I saw there was a new version of PyPy available, and realized I hadn't benchmarked it in a while. So here's a quick test of CPython 2.7 versus PyPy 2.2.1 (the default on Ubuntu 14.04 LTS) and PyPy 2.5 (the latest and greatest, unless you're reading this post in the future).

The benchmark is simply running all of my Python Project Euler solutions on each of these Python versions, throwing away any that didn't work or took longer than 60 seconds on any of them, and then dumping all the runtimes. There is definitely some random noise in the numbers, but it should mostly even out over many programs. (There were 145 that made the time cut this time.)

I ran these on Ubuntu 14.04 on an Amazon EC2 c4.2xlarge instance. (A Haswell Xeon CPU running somewhere in the 2.9 – 3.2 GHz range depending on whether Turbo Mode is engaged, 4 or 8 cores depending on whether you count HyperThreading, 15 GB.)

I'll give the results first in case you don't want to scroll past the big table: PyPy 2.5 177.35s, PyPy 2.2.1 208.11s, CPython 2.7 616.73s. So PyPy 2.5 is about 17% faster than PyPy 2.2.1 on this benchmark, a nice sign that PyPy keeps getting faster even though much of the low-hanging fruit was picked long ago. And PyPy is about 3.5 times as fast, or 250% faster, than CPython 2.7 on this benchmark.

I don't see any interesting cases where CPython significantly beats PyPy. We used to see cases like that, but as PyPy has matured it's become more consistently fast on a larger set of programs.) I guess the closest to an interesting result is euler225.py, where CPython beats PyPy 23.74s to 27.75s. Not a huge difference, but worth taking a look. I may try to figure out what's going on and post a followup.

script
PyPy 2.2.1
PyPy 2.5
CPython 2.7.6

euler1.py
 0.31
 0.10
 0.31


euler2.py
 0.31
 0.10
 0.10


euler3.py
 0.20
 0.10
 0.10


euler4.py
 0.10
 0.10
 0.10


euler5.py
 0.10
 0.10
 0.10


euler6.py
 0.10
 0.10
 0.10


euler7.py
 0.10
 0.10
 0.20


euler8.py
 0.10
 0.10
 0.10


euler9.py
 0.10
 0.10
 0.10


euler10.py
 0.99
 0.71
 3.20


euler11.py
 0.10
 0.10
 0.10


euler13.py
 0.10
 0.10
 0.10


euler14.py
 1.50
 0.90
 1.50


euler15.py
 0.10
 0.10
 0.10


euler16.py
 0.10
 0.10
 0.10


euler18.py
 0.10
 0.10
 0.10


euler19.py
 0.10
 0.10
 0.10


euler20.py
 0.10
 0.10
 0.10


euler21.py
 0.10
 0.10
 0.10


euler22.py
 0.10
 0.10
 0.10


euler23.py
 0.60
 0.60
 4.91


euler24.py
 1.60
 1.00
 2.11


euler25.py
 0.20
 0.10
 0.10


euler26.py
 1.71
 1.10
 1.91


euler27.py
 1.10
 0.60
 4.71


euler28.py
 0.10
 0.10
 0.10


euler29.py
 0.10
 0.10
 0.10


euler30.py
 0.70
 0.30
 2.21


euler32.py
 0.60
 0.50
 1.30


euler33.py
 0.10
 0.10
 0.10


euler34.py
 1.50
 1.00
 4.91


euler35.py
 2.31
 1.20
10.42


euler36.py
 0.30
 0.30
 0.90


euler37.py
 2.41
 1.70
 5.01


euler38.py
 0.40
 0.30
 0.50


euler39.py
 0.10
 0.10
 0.10


euler40.py
 0.20
 0.20
 0.30


euler41.py
 1.70
 0.60
 1.60


euler42.py
 0.10
 0.10
 0.10


euler43.py
 5.91
 4.31
11.52


euler44.py
 0.30
 0.30
 1.10


euler45.py
 0.40
 0.40
 0.60


euler46.py
 0.20
 0.10
 0.40


euler47.py
 0.40
 0.30
 1.20


euler48.py
 0.10
 0.10
 0.10


euler49.py
 0.20
 0.10
 0.20


euler50.py
 0.90
 0.70
27.36


euler51.py
 4.41
 2.61
 9.72


euler52.py
 0.20
 0.10
 0.30


euler53.py
 0.10
 0.10
 0.10


euler54.py
 0.20
 0.20
 0.10


euler55.py
 0.30
 0.10
 0.10


euler56.py
 0.20
 0.10
 0.30


euler57.py
 0.10
 0.10
 0.30


euler58.py
 2.61
 2.41
28.96


euler59.py
 2.21
 1.20
 9.72


euler61.py
 0.20
 0.10
 0.10


euler62.py
 0.20
 0.10
 0.10


euler63.py
 0.10
 0.10
 0.10


euler64.py
 3.11
 2.91
38.38


euler65.py
 0.10
 0.10
 0.10


euler66.py
 0.70
 0.70
 5.71


euler67.py
 0.10
 0.10
 0.10


euler68.py
 0.10
 0.10
 0.10


euler69.py
 0.10
 0.10
 0.10


euler70.py
 0.20
 0.20
 0.30


euler71.py
 0.10
 0.10
 0.40


euler72.py
 3.51
 2.91
21.84


euler73.py
 1.90
 1.71
16.23


euler75.py
 7.12
 0.60
 1.00


euler77.py
 0.20
 0.30
 0.20


euler79.py
 0.10
 0.10
 0.10


euler80.py
 0.10
 0.10
 0.20


euler81.py
 0.10
 0.10
 0.10


euler82.py
 0.20
 0.20
 0.20


euler83.py
 0.20
 0.10
 0.40


euler84.py
 0.90
 0.80
13.83


euler85.py
 0.90
 1.00
 5.11


euler87.py
 0.20
 0.30
 0.40


euler88.py
 3.31
 1.40
 2.21


euler89.py
 0.10
 0.10
 0.10


euler92.py
13.93
 7.72
46.80


euler93.py
 2.11
 1.20
 4.01


euler94.py
 4.31
 3.41
 5.71


euler95.py
11.62
13.33
53.92


euler97.py
 0.20
 0.20
 0.80


euler98.py
 0.20
 0.20
 0.20


euler99.py
 0.10
 0.10
 0.10


euler100.py
 0.10
 0.10
 0.10


euler101.py
 0.20
 0.20
 0.10


euler102.py
 0.10
 0.10
 0.10


euler103.py
 0.10
 0.10
 0.10


euler104.py
 0.30
 0.30
 0.60


euler105.py
 0.80
 0.20
 0.20


euler106.py
 0.80
 0.20
 0.20


euler107.py
 0.20
 0.20
 0.10


euler108.py
 0.50
 0.50
 2.81


euler109.py
 0.30
 0.20
 1.30


euler111.py
 0.80
 0.30
 7.02


euler112.py
 2.01
 0.80
 4.81


euler114.py
 0.10
 0.10
 0.10


euler115.py
 0.10
 0.10
 0.20


euler116.py
 0.10
 0.10
 0.10


euler117.py
 0.10
 0.10
 0.10


euler119.py
 0.10
 0.10
 0.10


euler120.py
 0.10
 0.10
 0.10


euler121.py
 0.10
 0.10
 0.10


euler122.py
13.63
13.73
18.34


euler123.py
 1.10
 1.11
 2.21


euler124.py
 0.40
 0.40
 0.90


euler125.py
 0.20
 0.30
 0.40


euler126.py
 0.70
 0.80
 5.01


euler135.py
 0.20
 0.20
 0.60


euler136.py
 8.12
 6.31
36.97


euler142.py
 0.10
 0.10
 0.10


euler143.py
 0.10
 0.10
 0.10


euler147.py
 0.10
 0.10
 0.30


euler149.py
 9.22
 7.62
17.34


euler150.py
 0.10
 0.10
 0.20


euler162.py
 0.10
 0.10
 0.10


euler171.py
 0.10
 0.10
 0.10


euler172.py
 0.30
 0.30
 0.20


euler173.py
 0.10
 0.10
 0.30


euler174.py
 0.50
 0.70
 0.80


euler181.py
 0.10
 0.10
 0.10


euler188.py
 0.20
 0.10
 0.10


euler190.py
 0.10
 0.10
 0.10


euler202.py
 0.10
 0.10
 0.10


euler205.py
 0.40
 0.40
 0.30


euler206.py
 4.51
 4.21
15.63


euler207.py
 0.20
 0.20
 0.20


euler222.py
 0.10
 0.10
 0.10


euler225.py
27.75
27.75
23.74


euler227.py
 0.10
 0.10
 0.60


euler230.py
 0.10
 0.10
 0.10


euler233.py
 0.10
 0.10
 0.10


euler234.py
 1.30
 1.00
 2.51


euler235.py
 0.10
 0.10
 0.10


euler240.py
 3.41
 1.91
 7.22


euler267.py
 0.20
 0.30
 0.20


euler286.py
 9.02
 8.12
54.39


euler345.py
15.13
21.54
27.76


euler346.py
 1.00
 0.80
 1.80


euler347.py
 6.41
 6.92
20.04


euler371.py
 0.10
 0.10
 0.10


total
208.11
177.35
616.73


wins
87
127
71