(download) (home) (papers) (performance) (projects) (team) (user guide) (users)

This page compares the performance of the following SML compilers on a range of benchmarks.

There are tables comparing run time, compile time, and code size.

All of the benchmarks are available here. Some of the benchmarks were obtained from the SML/NJ benchmark suite. Some of the benchmarks use data files in the DATA subdirectory.

Setup

All benchmarks were compiled and run on a 1.6 GHz Pentium 4 with 512M of physical memory. The benchmarks were compiled with the default settings for all the compilers, except for Moscow ML, which was passed the -orthodox -standalone -toplevel switches. The Poly/ML executables were produced by useing the file, followed by a PolyML.commit. The SML/NJ executables were produced by wrapping the entire program in a local declaration whose body performs an SMLofNJ.exportFn.

For more details, or if you want to run the benchmarks yourself, please see the benchmark directory of the MLton sources.

Run time ratio

The following table gives the ratio of the run time of each benchmark when compiled by another compiler to the run time when compiled by MLton. That is, the larger the number, the slower the generated code runs. A number larger than one indicates that the corresponding compiler produces code that runs more slowly than MLton. If an entry is *, that means that the corresponding compiler failed to compile the benchmark or that the benchmark failed to run.

benchmark ML-Kit Moscow-ML Poly/ML SML/NJ
barnes-hut 3.8 25.2 * 0.9
checksum 9.3 * * 6.3
count-graphs 6.4 47.0 4.0 2.3
DLXSimulator 1.4 * * *
fft 2.1 18.7 11.6 0.8
fib 0.8 4.1 1.1 1.0
hamlet * 16.3 2.3 2.2
imp-for 2.4 55.9 5.2 4.9
knuth-bendix 5.5 17.3 3.8 2.5
lexgen 2.3 6.8 2.1 1.4
life 2.4 36.8 8.9 1.4
logic 2.9 9.2 1.2 1.4
mandelbrot 6.1 70.7 85.9 2.3
matrix-multiply 14.2 125.6 31.2 8.8
md5 * * * 14.1
merge * * 1.3 10.5
mlyacc 1.9 8.7 1.3 1.7
mpuz 2.2 68.3 6.3 3.5
nucleic 2.8 21.9 27.0 0.4
peek 15.7 159.5 22.9 10.3
psdes-random 4.7 * * 3.5
ratio-regions 3.5 62.0 3.1 6.2
ray 5.2 25.6 33.4 1.0
raytrace * * 39.2 2.4
simple 1.6 17.3 6.4 1.4
smith-normal-form * * 20.1 102.2
tailfib 1.0 38.1 3.7 1.7
tak 0.9 13.0 1.3 1.6
tensor * * * 5.3
tsp 3.7 44.6 * 2.0
tyan * 20.1 1.3 1.4
vector-concat 0.8 26.7 2.4 6.5
vector-rev 3.1 89.4 4.7 33.7
vliw 2.3 12.1 1.7 1.9
wc-input1 7.9 * 8.8 12.0
wc-scanStream 18.2 * 161.2 6.2
zebra 10.0 48.5 6.0 11.6
zern * * * 1.6

Compile time

The following table gives the compile time of each benchmark in seconds. A * in an entry means that the compiler failed to compile the benchmark.

benchmark MLton ML-Kit Moscow-ML Poly/ML SML/NJ
barnes-hut 1.5 6.1 0.6 * 1.1
checksum 0.4 0.8 * * 0.1
count-graphs 1.0 2.0 0.2 0.1 0.7
DLXSimulator 2.7 8.0 * * *
fft 0.8 1.5 0.1 0.1 0.6
fib 0.4 0.8 0.1 0.0 0.1
hamlet 30.1 * 27.9 11.1 52.2
imp-for 0.3 0.9 0.1 0.0 0.2
knuth-bendix 1.2 4.4 0.2 0.1 1.2
lexgen 3.4 7.2 0.5 0.3 3.1
life 0.8 2.0 0.1 0.1 0.5
logic 1.7 5.0 0.3 0.1 1.2
mandelbrot 0.3 0.9 0.1 0.0 0.2
matrix-multiply 0.4 0.9 0.1 0.0 0.2
md5 0.7 1.8 * * 1.1
merge 0.4 0.8 0.1 0.0 0.2
mlyacc 13.2 39.9 4.5 1.5 15.3
mpuz 0.6 1.3 0.1 0.1 0.3
nucleic 1.5 16.3 1.3 0.4 1.5
peek 0.7 1.0 0.1 0.0 0.2
psdes-random 0.4 0.9 * * 0.2
ratio-regions 1.4 4.1 0.2 0.1 1.2
ray 2.3 3.4 0.2 0.1 0.7
raytrace 6.3 * * 0.6 4.3
simple 4.2 13.2 0.6 0.2 3.1
smith-normal-form 3.9 * * 0.1 2.0
tailfib 0.3 0.8 0.0 0.0 0.1
tak 0.3 0.7 0.1 0.0 0.1
tensor 1.8 * * * 2.2
tsp 0.8 1.9 0.2 * 0.5
tyan 2.2 6.4 0.4 0.2 1.8
vector-concat 0.4 0.8 0.1 0.0 0.2
vector-rev 0.4 0.8 0.0 0.0 0.2
vliw 7.7 29.8 2.0 1.3 11.2
wc-input1 1.1 0.8 0.1 0.0 0.2
wc-scanStream 1.1 0.8 0.1 0.0 0.2
zebra 3.5 2.1 0.1 0.1 0.5
zern 0.7 * * * 0.5

Code size

The following table gives the code size of each benchmark in bytes. The size for MLton and the ML Kit is the sum of text and data for the standalone executable as reported by size. The size for Moscow ML is the size in bytes of the executable a.out. The size for Poly/ML is the difference in size of the database before the session start and after the commit. The size for SML/NJ is the size of the heap file created by exportFn and does not include the size of the SML/NJ runtime system (approximately 100K). A * in an entry means that the compiler failed to compile the benchmark.

benchmark MLton ML-Kit Moscow-ML Poly/ML SML/NJ
barnes-hut 57,119 172,121 94,990 * 328,696
checksum 23,021 72,877 * * 333,528
count-graphs 44,333 88,905 84,575 98,304 355,376
DLXSimulator 88,445 161,869 * * *
fft 32,893 86,121 84,095 65,536 329,736
fib 23,053 16,525 79,878 49,152 307,896
hamlet 1,106,524 * 277,168 5,316,608 1,259,720
imp-for 23,053 17,253 80,040 57,344 308,920
knuth-bendix 64,582 97,609 88,439 180,224 322,528
lexgen 152,293 214,401 104,883 188,416 392,184
life 39,629 79,477 83,390 90,112 318,424
logic 80,205 115,073 87,252 114,688 331,744
mandelbrot 23,085 78,289 81,341 57,344 313,016
matrix-multiply 23,629 96,537 82,419 57,344 339,656
md5 32,678 91,841 * * 332,816
merge 24,301 26,001 80,091 49,152 308,928
mlyacc 471,909 493,345 148,286 2,908,160 690,216
mpuz 27,437 76,357 82,381 81,920 321,208
nucleic 62,061 221,585 207,154 204,800 352,240
peek 31,622 61,245 81,618 57,344 312,040
psdes-random 24,397 25,985 * * 314,040
ratio-regions 42,765 98,729 87,485 73,728 334,832
ray 85,788 112,709 89,860 147,456 383,048
raytrace 237,545 * * 524,288 502,888
simple 182,141 202,945 94,397 475,136 637,984
smith-normal-form 137,679 * * 131,072 484,424
tailfib 22,765 16,685 79,939 57,344 307,896
tak 23,149 16,441 79,928 57,344 303,800
tensor 56,910 * * * 342,048
tsp 38,022 100,409 86,140 * 332,784
tyan 85,894 145,605 91,587 204,800 368,656
vector-concat 23,853 24,853 80,191 49,152 318,152
vector-rev 23,821 25,049 80,073 57,344 318,152
vliw 300,117 471,721 135,386 696,320 618,576
wc-input1 48,798 132,565 86,900 49,152 313,016
wc-scanStream 49,726 132,997 87,076 49,152 314,040
zebra 109,926 45,077 83,419 90,112 323,560
zern 30,524 * * * 365,296


MLton
Last modified: Mon Sep 16 12:02:46 PDT 2002