2015-02-26

Numeric CombSort Benchmark update!

As I've written before, CombSort are quite good sort algorithm. Let's compare this algorithm when implemented in various programming language. The benchmark should not use any other built-in function other than array generation and printing. The benchmark uses AMD A8-6600K, 16GB RAM with Non-SSD disk.

$ alias | grep 'alias time'
alias time='/usr/bin/time -f "\nCPU: %Us\tReal: %es\tRAM: %MKB"'
$ time --version
GNU time 1.7

g++ --version
g++ (GCC) 4.9.2 20141224 (prerelease)
$ time g++ comb.cpp
CPU: 0.05s      Real: 0.12s     RAM: 19428KB
$ time ./a.out
CPU: 1.94s      Real: 1.97s     RAM: 79804KB
$ time g++ -O2 comb.cpp
CPU: 0.07s      Real: 0.11s     RAM: 21260KB
$ time ./a.out
CPU: 0.88s      Real: 0.90s     RAM: 79804KB

clang --version
clang version 3.5.1 (tags/RELEASE_351/final)
$ time clang++ comb.cpp
CPU: 0.05s      Real: 0.08s     RAM: 33564KB
$ time ./a.out
CPU: 1.83s      Real: 1.86s     RAM: 79764KB
$ time clang++ -O2 comb.cpp
CPU: 0.08s      Real: 0.14s     RAM: 37860KB
$ time ./a.out
CPU: 0.89s      Real: 0.91s     RAM: 79804KB

java -version
java version "1.7.0_71" 
$ time javac comb.java
CPU: 1.05s      Real: 0.73s     RAM: 65952KB
$ time java comb
CPU: 1.32s      Real: 1.32s     RAM: 110488KB

php --version
PHP 5.6.4 (cli) (built: Dec 17 2014 21:45:04)
$ time php comb.php
CPU: 102.69s    Real: 104.20s   RAM: 2497508KB

hhvm --version
HipHop VM 3.5.0 (rel)
$ time hhvm -v Eval.Jit=true comb.php 
CPU: 12.56s     Real: 14.83s    RAM: 362488KB

ruby --version
ruby 2.2.0p0 (2014-12-25 revision 49005) [x86_64-linux]
$ time ruby comb.rb
CPU: 52.87s     Real: 53.02s    RAM: 87892KB

rbx --version
rubinius 2.5.2 (2.1.0 7a5b05b1 2015-01-30 3.5.1 JI) [x86_64-linux-gnu]
$ time rbx comb.rb
CPU: 74.89s     Real: 74.30s    RAM: 135320KB

node --version
v0.10.35
$ time node comb1.js
CPU: 2.64s      Real: 2.64s     RAM: 92240KB
$ time node comb2.js
CPU: 2.68s      Real: 2.72s     RAM: 140612KB

rhino < /dev/null 
Rhino 1.7 release 4 2014 07 01
$ rhino comb2.js
CPU: 87.39s     Real: 61.16s    RAM: 1993848KB

$ pacman -Qo `which jsc-3`
/usr/bin/jsc-3 is owned by webkitgtk 2.4.8-1
$ time jsc-3 comb1.js
CPU: 23.74s     Real: 23.93s    RAM: 93740KB
$ time jsc-3 comb2.js
CPU: 18.99s     Real: 19.16s    RAM: 181644KB

js24 --help | grep Version
Version: JavaScript-C24.2.0
$ time js24 --ion-eager comb1.js
CPU: 2.13s      Real: 2.15s     RAM: 89688KB
$ time js24 --ion-eager comb2.js
CPU: 1.53s      Real: 1.58s     RAM: 92384KB

go version
go version go1.4.1 linux/amd64
$ time go build comb.go 
CPU: 0.14s      Real: 0.17s     RAM: 31568KB
$ time ./comb
CPU: 1.10s      Real: 1.14s     RAM: 79824KB

rustc --version
rustc 1.0.0-dev
$ time rustc comb.rs
CPU: 0.39s      Real: 0.49s     RAM: 106844KB
$ time ./comb
CPU: 10.62s     Real: 10.71s    RAM: 86020KB
$ time rustc -O comb.rs
CPU: 0.41s      Real: 0.49s     RAM: 110204KB
$ time ./comb
CPU: 0.97s      Real: 0.99s     RAM: 86108KB

scala -version
Scala code runner version 2.11.5 -- Copyright 2002-2013, LAMP/EPFL
$ time scala comb.scala
CPU: 5.43s      Real: 6.30s     RAM: 206088KB
$ time scalac comb.scala
CPU: 10.62s     Real: 7.00s     RAM: 143460KB
$ time scala Comb
CPU: 5.49s      Real: 5.05s     RAM: 206300KB

python --version
Python 3.4.2
$ time python comb1.py
CPU: 90.47s     Real: 90.83s    RAM: 403192KB
$ time python comb2.py
CPU: 106.82s    Real: 107.26s   RAM: 87248KB

pypy --version
Python 2.7.8 (c6ad44ecf5d8, Nov 18 2014, 18:04:31) [PyPy 2.4.0 with GCC 4.9.2]
$ time pypy comb1.py
CPU: 5.34s      Real: 5.40s     RAM: 136764KB
$ time pypy comb2.py
CPU: 5.85s      Real: 6.04s     RAM: 204588KB

mcs --version
Mono C# compiler version 3.12.0.0
$ time mcs -o+ comb.cs
CPU: 0.44s      Real: 0.47s     RAM: 45908KB
$ time ./comb.exe
CPU: 1.38s      Real: 1.41s     RAM: 90472KB

lua -v
Lua 5.2.3  Copyright (C) 1994-2013 Lua.org, PUC-Rio
$ time lua comb.lua
CPU: 65.64s     Real: 65.81s    RAM: 264096KB

luajit -v
LuaJIT 2.0.3 -- Copyright (C) 2005-2014 Mike Pall.
$ time luajit comb.lua
CPU: 6.30s      Real: 6.34s     RAM: 132964KB

dart --version
Dart VM version: 1.8.5 (Tue Jan 13 12:44:14 2015) on "linux_x64"
$ time dart scomb.dart
CPU: 2.12s      Real: 2.24s     RAM: 93392KB

The code can be found on my dropbox (folder: num-comb), and here's the summary:

Compiler / InterpreterLanguageCompile DurationCompile RAMRuntime DurationRuntime RAMTotal Duration
g++ (debug)C++50194281940798041990
g++ (-O2)C++702126088079804950
clang++ (debug)C++50335641830797641880
clang++ (-O2)C++803786089079804970
javac, javaJava10506595213201104882370
phpPHP1026902497508102690
hhvmPHP1256036248812560
rubyRuby528708789252870
rbxRuby7489013532074890
node (typed array)Javascript2640922402640
node (untyped array)Javascript26801406122680
rhino (untyped array)Javascript87039199384887039
jsc-3 (typed array)Javascript237409374023740
jsc-3 (untyped array)Javascript1899018164418990
js24 (typed array)Javascript2130896882130
js24 (untyped array)Javascript1530923841530
goGo140315681100798241240
rustc (debug)Rust390106844106208602011010
rustc (-O2)Rust410110204970861081380
scalaScala54302060885430
python3Python 39047040319290470
python3 (array)Python 310682087248106820
pypyPython 253401367645340
pypy (array)Python 258502045885850
mcsC#440459081380904721820
luaLua6564026409665640
luajitLua63001329646300
dartDart2120933922120

Write down your opinion (or pastie if you found a bug on these source, or if you want to add more language implementation) on the comment section ^^)b

Note #1Opal (0.6.8) and JRuby (both 1.7.18 and 9.0.0pre1) failed to run this benchmark (they exceed 300s runtime limit even when using -J-Xmx3000M -J-Djruby.compile.mode=FORCE flag).

Note #2: Yes, it's unfair to compare array of integer and array of double, life is unfair by design, get over it...

5 comments :

  1. You should use php 7 version for comparsions, because using 5.6 is not correct anymore.

    ReplyDelete
  2. Regarding the time for calculations, you are (voluntarily?) using far too high values. These values (10 millions entries) are not realistic at all, not even in the slightest.

    What happens if you use 10 thousand values instead of 10 millions? I'll tell you:
    Node 5.3.0 => 231ms
    PHP 7.0.1 => 133ms
    PHP 5.5.12 => 205ms

    Even an almost two years old version of PHP is faster than a cutting edge version of Node. Let alone a recent version of PHP, which is almost twice as fast as its Node equivalent.

    ReplyDelete
    Replies
    1. Could you post a benchmark that can hold that argument?

      Delete
  3. that would be REALLY awesome if you could repeat those mesurements on newest versions of those environments

    ReplyDelete

THINK: is it True? is it Helpful? is it Inspiring? is it Necessary? is it Kind?