Showing posts with label pypy. Show all posts
Showing posts with label pypy. Show all posts

2021-12-31

String Associative Array and CombSort Benchmark 2021 Edition

Last year, we've done string associative benchmark and lesser string associative benchmark (measuring string concat operation and built-in associative array set and get), numeric comb sort benchmark and string comb sort benchmark (measuring basic array random access, string conversion, and array swap for number and string), this year's using newer server: 32-core running on 64-bit Ubuntu 21.10. This time we will skip programming languages that are no deb packages (unless the install script is just one line and doesn't ruin system directories) or no direct compile-run command like previous one, also only best of 3 runs.

$ alias time='/usr/bin/time -f "\nCPU: %Us\tReal: %es\tRAM: %MKB"

This time, NodeJS failed to complete (after waiting for an hour) with 10x more data compared to last year for assoc benchmark. Here's the spreadsheet and final result (Real duration and RAM):

LanguageCommand FlagsVersionAssocRAMNum CombRAMStr CombRAMTotalRAM
Gogo rungo1.17.512.392,305,8240.4683,0963.33245,89616.182,634,816
Javajava18-ea+15-Ubuntu-410.705,582,3080.96170,2406.93722,15218.596,474,700
Nimnim r -d:release --gc:arc1.4.217.904,200,2121.2179,4445.45633,81624.564,913,472
Pythonpypy7.3.519.703,104,4402.17140,1245.73523,26427.603,767,828
Ctcc -run0.9.2722.682,820,5681.0180,4844.80392,89628.493,293,948
Juliajulia1.6.523.983,714,4480.52255,8444.60861,07629.104,831,368
Vv -prod run0.2.4 a0a180720.368,910,8562.67124,2486.48470,13629.519,505,240
Lualuajit2.1.0-beta316.701,133,8763.27133,50410.80511,46830.771,778,848
Dartdart2.15.125.922,101,1521.40208,0565.75574,85233.072,884,060
Crystalcrystal run --release1.2.213.772,371,3207.73202,55212.08440,74833.583,014,620
Nimnim r -d:release1.4.224.923,864,1001.7679,5487.511,211,20034.195,154,848
PHPphp8.0.89.941,368,8087.74328,34424.84641,44842.522,338,600
Crystalcrystal run0.35.138.852,372,0049.97179,17622.27441,72071.092,992,900
Vv run0.2.4 a0a180751.818,911,0046.6379,71618.43470,42076.879,461,140
Pythonpython33.9.743.114,106,89229.16405,33243.08722,996115.355,235,220
Nimnim r1.4.288.053,864,0482.9379,53632.601,211,260123.585,154,844
Rubyrubyruby 2.7.4p19152.482,970,90827.15100,32052.23708,940131.863,780,168
Javascriptnodev16.13.1999.999,999,9990.71115,0446.11461,8361006.8110,576,879

FAQ

1. Why you measure the compile duration too? because developer experience also important (feedback loop, edit-rebuild/compile-run), at least for me, it would be sucks a lot if we have to wait a minute to compile before we can test something. We could always write precalculated values with C++ template to make runtime faster for example, but the compilation delay would be very sucks.
2. Why not warming up the VM first? each implementation have it's own advantage and disadvantage. We already know, that compiled language mostly faster at runtime, but at cost of relatively slower development feedback loop. Interpreted language mostly slower at runtime, especially if executed using VM that have startup overhead, in exception of one with AOT or JIT optimization. So to make it fair for every kind of implementation, we do it differently by not glorifying the runtime performance (which super make sense for server or long-lived process, but not best for development or CI cost which people often neglected), but by total performance which consist of Compile duration (if any) + VM startup duration (if any) + AOT or JIT duration (if any) + Runtime duration, so every strategy the PL's implementator use can be fairly judged.
3. Why there's no C++, VB.NET, C#, D, Object-Pascal? don't want to compile things (since there's no build and run command in one flag).  
4. Why there's no Kotlin, Scala, Rust, Elixir, Pony, Swift, Groovy, or Zig? Too lazy to add :3 you can contribute tho (create a pull request, then I'll run the benchmark again as preferably as there's precompiled binary/deb/apt/ppa repository for the compiler/interpreter).
5. Why there's no Ruby 3.1? I can't find any PPA for latest Ruby, latest one on Ubuntu 21.10 repo is Ruby2.7.

Contributorsilmanzo (Nim, Crystal, D), inkydragon (Julia)

2018-09-13

String Associative Array Benchmark 2

Redoing old string associative benchmark (3 years ago) but now on Macbook Pro MJLQ2, here's the result:

alias time='/usr/local/bin/gtime -f "\nCPU: %Us\tReal: %es\tRAM: %MKB"'

$ java -version
java version "1.8.0_172"
Java(TM) SE Runtime Environment (build 1.8.0_172-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.172-b11, mixed mode)
$ time javac hash_map.java
CPU: 1.22s Real: 0.60s RAM: 67540KB
$ time java hash_map
6009354 6009348 611297
36186112 159701682 23370001
CPU: 274.83s Real: 53.47s RAM: 3946268KB

$ go version
go version go1.10.2 darwin/amd64

$ go build map.go
CPU: 0.09s Real: 0.14s RAM: 16160KB
$ time ./map
6009354 6009348 611297
36186112 159701682 23370001
CPU: 29.55s Real: 22.12s RAM: 2425316KB

$ dart --version
Dart VM version: 2.0.0 (Fri Aug 3 10:53:23 2018 +0200) on "macos_x64"

$ time dart ./map.dart
6009354 6009348 611297
36186112 159701682 23370001
CPU: 59.29s Real: 60.08s RAM: 1763952KB

$ node --version
v10.10.0
$ time node object.js # ERROR: TOO SLOW
^CCommand terminated by signal 2 
CPU: 855.68s Real: 763.40s RAM: 1517576KB

$ php -version
PHP 7.1.16 (cli) (built: Mar 31 2018 02:59:59) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.1.0, Copyright (c) 1998-2018 Zend Technologies
$ time php -d memory_limit=4G assoc.php
6009354 6009348 611297
36186112 159701682 23370001
CPU: 26.76s Real: 27.89s RAM: 1297824KB

$ mcs --version
Mono C# compiler version 5.10.1.0
$ time mcs dictionary.cs
CPU: 0.21s Real: 0.32s RAM: 42564KB
$ time mono ./dictionary.exe
6009354 6009348 611297
36186112 159701682 23370001
CPU: 33.69s Real: 33.64s RAM: 1609576KB

$ ruby --version
ruby 2.3.7p456 (2018-03-28 revision 63024) [universal.x86_64-darwin17]
$ time ruby hash.rb
time ruby hash.rb
6009354 6009348 611297
36186112 159701682 23370001
CPU: 100.57s Real: 103.63s RAM: 3012148KB

$ jruby --version
jruby 9.2.0.0 (2.5.0) 2018-05-24 81156a8 Java HotSpot(TM) 64-Bit Server VM 25.172-b11 on 1.8.0_172-b11 +jit [darwin-x86_64]
$ time jruby hash.rb # ERROR: TOO MEMORY CONSUMING
Error: Your application used more memory than the automatic cap of 3641MB.
Specify -J-Xmx####M to increase it (#### = cap size in MB).
Specify -w for full java.lang.OutOfMemoryError: GC overhead limit exceeded stack trace
Command exited with non-zero status 1
CPU: 1616.55s Real: 257.91s RAM: 4439172KB

$ python3 --versionPython 3.7.0
$ time python3 dictionary.py
6009354 6009348 611297
36186112 159701682 23370001
CPU: 117.59s Real: 121.63s RAM: 3923812KB

$ pypy --version
Python 2.7.13 (ab0b9caf307db6592905a80b8faffd69b39005b8, Jun 24 2018, 08:19:27)[PyPy 6.0.0 with GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.2)]
$ time pypy dictionary.py
(6009354, 6009348, 611297)
(36186112, 159701682, 23370001)
CPU: 28.82s Real: 31.55s RAM: 3993764KB

$ lua -v
Lua 5.3.5  Copyright (C) 1994-2018 Lua.org, PUC-Rio
$ time lua table.lua
6009354 6009348 611297
36186112 159701682 23370001
CPU: 89.05s Real: 91.35s RAM: 2523332KB

$ luajit -v
LuaJIT 2.0.5 -- Copyright (C) 2005-2017 Mike Pall. http://luajit.org/
$ time luajit table.lua # ERROR: CANNOT ALLOC MEMORY
PANIC: unprotected error in call to Lua API (not enough memory)
Command exited with non-zero status 1CPU: 30.57s Real: 31.56s RAM: 1132648KB

This time PHP7 became the best performing programming language implementation, also the least memory consumption (I'm amazed with what they did in version 7). And as usual LuaJIT limited to 1-2GB so they failed, NodeJS too slow, and I'm unwilling to wait more than 10 minutes, so I terminate the app, JRuby hogging RAM too much.


LanguageVersionCompile DurationCompile RAMRuntime DurationRuntime RAM2015's Duration2015's RAMDuration ImprovRAM Improv
C#5.10.1.00.3242,56433.641,609,57640.981,690,22417.91%4.77%
Dart2.0.060.081,763,952133.352,538,22054.95%30.50%
Go1.10.20.1416,16022.122,425,31624.102,748,7848.22%11.77%
Java1.8.0_1720.6067,54053.473,946,268103.544,119,38848.36%4.20%
Lua5.3.591.352,523,332101.733,079,33610.20%18.06%
PHP7.1.1627.891,297,824failedfailed
Pypy6.0.031.553,993,76479.874,034,95660.50%1.02%
Python33.7.0121.633,923,812157.714,335,56822.88%9.50%
Ruby2.3.7p456103.633,012,148107.913,031,8723.97%0.65%