Reproducing the benchmark

[croqaz]

Hi folks. I would like to reproduce the results from this article about execution performance: https://www.graalvm.org/latest/reference-manual/python/Performance/#execution-performance

I have a fork of the The Python Performance Benchmark Suite: https://github.com/croqaz/py-bench which allows me to run most of the benchmarks without additional dependencies and running tests one by one, eg:

python bench/pystone.py
# 100000

Obviously, pystone.py is a silly example, there are serious tests in there. I also tried some private code.
When I compared the results of CPython 3.11, 3.12 or 3.13 with GraalPy 24.1.2 from the releases of this repo, GraalPy is roughly 5 times slower, which is similar to what the article above says.

I would like to run the JIT compilation, so I can benefit from the 4x speedups in the article, but I just don't know what flags to enable in the graalpy executable. Does it even work with the standalone executable ? or I need to run it within a JAVA app ? I would really like to run it standalone.

Thank you very much!

[msimacek]

With benchmarks that take just a few hundred miliseconds, you're mostly measuring the difference in interpreter startup time. GraalPy's startup is much slower than CPython's.
The JIT is on by default, but you need to give it enough time to kick in. It takes some time for functions to get hot enough to be marked for compilation and it takes some time to compile them. The JIT is meant to optimize long running code, it doesn't help on short one-shot runs. Benchmarking tools like pyperformance account for this by running the benchmark loop a few times to let it warm up before starting the actual measurement (in the same process). You can't get that kind of measurement with time or hyperfine. But you can try increasing the number of iterations to amortize the startup and warmup time. Try python bench/pystone.py 20000000 - on my machine CPython 3.13 takes 27s and GraalPy takes 2s.

[croqaz]

Thank you very much for your reply! I can confirm that at least pystone is much faster with GraalPy compared to CPython, exactly like you said!
I have to play with this a bit more, with different benchmarks.