intertwingly

Three months ago I put the same small Rails application through a 2×2: stock Rails and Roundhouse's emitted Ruby, each on CRuby+YJIT and on JRuby. The claim I drew from it was that the emit — Rails with every request-invariant decision made ahead of time, so what remains is monomorphic sends, resolved constants, and direct string building — is the input the JVM's JIT had been waiting for: stock Rails gained about 2× from the JVM, the emit gained 5–6×, and the two effects multiplied.

Benoit Daloze, who leads TruffleRuby, asked whether it held on TruffleRuby's JVM mode, and was politely doubtful about two of my parameters: "I'm not sure if the 20s warmup or 512MB heap is enough, but no better way to know than just trying it."

He was right about both, and the answer to the larger question turned out to be no.

Nothing to build

The first thing to say is what the experiment did not involve. The JRuby row in June needed its own emit target — the sqlite3 gem is a C extension with no JVM build, so the JRuby tree swaps in a JDBC backend. TruffleRuby compiles C extensions. The published ruby.tgz — the CRuby target, byte for byte — bundles, boots, and serves under TruffleRuby with no changes: sqlite3, nokogiri, puma, nio4r all build, the JSON output is byte-identical to CRuby's, and the HTML differs only in the CSRF token. There is no TruffleRuby target in Roundhouse and there won't be one; the tarball you can download today is the reproduction.

That is also why this experiment is cleaner than June's. The stock-Rails-on-JRuby baseline is still pinned to Rails 8.0 by the only AR-JDBC adapter that drives JRuby 10; the TruffleRuby baseline is the same Rails 8.1 fixture the CRuby lane serves. Same app, same Rails, same emitted code, three runtimes.

The 2×3

These are from the benchmark page, which now carries truffleruby and rails-truffleruby lanes and is regenerated nightly; the figures below are one night's, and the ratios are the point. The machine is a bare-metal Ryzen 5 3600 (6 cores / 12 threads, governor pinned, boost off), wrk at 64 connections, three 20-second runs per cell after a warmup the harness now sizes per lane. Every Ruby runtime gets Puma's 5 threads; Spinel — the same emitted framework compiled to a native binary — gets its autodetected workers.

/articles, the HTML index:

/articles/1.json:

Reading the columns

Read down the stock Rails column first, because it is the one nobody has to take my word for: it is Rails 8, unmodified, on three runtimes. JRuby is worth 1.9× on both pages. TruffleRuby is worth 3.4× on HTML and 5.2× on JSON — the best runtime for a stock Rails application in this table by a wide margin, and on the JSON endpoint it comes within 15% of Roundhouse's emit running on CRuby. If your question is "which Ruby runs my Rails app fastest without changing a line of it," the answer here is TruffleRuby, and it isn't close.

Now read down the emit column. JRuby is the fastest interpreter-or-JIT for the emitted code, 3.3× and 4.5× over CRuby+YJIT. TruffleRuby is 1.8× and 2.7× — and, on the same code, trails JRuby by 1.7–1.9×.

Then read the two together, which is the point of a 2×3. In June the JVM rewarded the emit more than it rewarded Rails (3.3× against 1.9× on this machine). On TruffleRuby the sign flips: the runtime swap is worth 3.4× to stock Rails and 1.8× to the emit. The emit still wins on TruffleRuby — 5.1× and 3.1× over Rails on the same runtime — but by a third of the JVM's margin. The page states this in a sentence per runtime now, because it changed my reading of the June result: "the input was the limit" was a fact about HotSpot reached through JRuby, not a fact about JITs.

I think the reason is the one TruffleRuby's authors would give. Truffle is a partial evaluator: it specializes an AST interpreter with respect to the program it is running, which is the first Futamura projection, and Rails' metaprogramming — the dynamic dispatch, the method_missing, the per-request re-derivation of the same answer — is exactly the kind of thing specialization dissolves once the shapes stabilize. Roundhouse makes the same move at a different layer: it specializes Rails with respect to the application, ahead of time, and writes the residue down as source. When the runtime is already doing most of that work, the compiler's residue has less left to buy. The two projections don't multiply the way the emit and HotSpot did; they overlap.

That is a more interesting result than a second confirmation would have been, and a more humbling one.

Winners and trade-offs

The table above is throughput. The rest of the trade is memory, startup, and warmup, and the three runtimes sit in very different places.

Throughput, latency and RSS are the box's; boot and time-to-plateau are from the laptop runs where I watched the warmup curve in 30-second slices, and the box, with slower cores sharing time with Graal's compiler threads, plateaus later still.

Benoit's two doubts split cleanly by codebase. 512 MB is enough for the emit under TruffleRuby — it costs 3% against an unpinned heap that otherwise grows to nearly 4 GB — and it is not enough for stock Rails, which runs out of a 512 MB heap within a minute of load; it needs 1 GB, at −9%. 20 seconds of warmup is not enough for either: the emit under TruffleRuby plateaus in one to two minutes, stock Rails in two or more, against JRuby's half-minute. The benchmark harness warms those lanes longer now and records the figure it used in every row — and the first night on the box showed the laptop-derived floors were still short. The page's own stability section named the cells: Rails on TruffleRuby went 979 → 1,756 → 1,874 req/s across three runs warmed 120, 240 and 360 seconds, a median still climbing; the emit on TruffleRuby went flat only from its second run; and stock Rails on JRuby, which had never had a floor, ran its first cell 42% under its third. Commit 9b97fe28 moves each floor to the point its curve went flat on this machine — 120 s for the emit on TruffleRuby, 240 s for stock Rails on it, 60 s for stock Rails on JRuby — and the check on the next nightly is simple: those three cells' run-to-run spread drops under 3%, and the first run stops being the slowest. Until it does, read the TruffleRuby rows above as floors, and the stock-Rails one as the loosest of them.

So: if you want the fastest Ruby for the code Roundhouse emits and you have the memory, JRuby. If you want the fastest runtime for Rails as it is, TruffleRuby, with more memory still and a startup that wants a warm pool in front of it. If you want the smallest, cheapest-per-request deployment of the emit — 65 MB, sub-second start, no warmup — Spinel, which is 2.4× JRuby's throughput on this page at a sixteenth of its memory, with the caveat that it is using all six cores where Puma's five threads are not, a comparison I'll come back to. And CRuby+YJIT with the emit is the quiet one: 10× stock Rails at a third of stock Rails' memory, on the runtime you already have.

Native or JVM?

TruffleRuby ships as two standalones now — the native image, and a JVM build with Graal as the JIT — and Benoit asked specifically about the JVM one. On the emit, native led by 10–15% and booted in a third of the time. On stock Rails, JVM mode edged native by about 10% on HTML at the same heap and needed the longest warmup of anything I measured. Neither difference changes a ranking above, so the nightly runs the native build and names it in its environment appendix; the JVM figures are in the write-up notes if anyone wants them.

What we found along the way

Three things, none of which I went looking for.

TruffleRuby found a bug in my runtime. Its threads run in parallel — no GVL — and the first table I produced had 13% of requests failing with undefined method '[]' for nil. The cause was a statement-handle counter in the emitted SQLite shim that was shared across threads without a lock, which CRuby's GVL had been hiding for months. JRuby had been reporting the same race the whole time, as one can't add a new key into hash during iteration a minute in the bench logs, low enough that nobody chased it. A GVL-free runtime is a race detector for GVL-shaped code; the fix was the design the JRuby backend had already adopted, two files away. That fix is in today's tarball.

An error page is a plausible benchmark. wrk counts every response toward its requests-per-second, and a Rails 500 is cheap to serve. Chasing the TruffleRuby failures taught me to read the Non-2xx line, and reading it on the box turned up something worse: the rails-jruby JSON cell on the public page had been 100% error pages since the day the lane was added — JRuby 10.1.0.0 raised on every Rails datetime read after warmup, a bug I had filed myself in July, fixed in 10.1.1.0, on a box that had never been upgraded. June's post said JRuby and CRuby were "at rough parity" on JSON, 1,272 against 1,080. The 1,080 was the error page. The corrected figure has JRuby at 1.9× on JSON, the same as on HTML, and that paragraph of the June post is wrong. The harness now records a cell with any non-2xx response as no data, with the count beside it, and says so on the page.

The governor wasn't pinned. The page's environment line had honestly said governor=schedutil, boost enabled since June 11, when a reboot silently dropped a cpupower setting nothing re-applied. Every number from June through last night was taken that way; the page told anyone who read the appendix. A systemd unit now re-pins it at boot, and the figures above are the first on the restored configuration — which is also why they are not directly comparable to the ones the page showed the night before.

What to explore next

Why the emit trails JRuby on TruffleRuby by 2×. It doesn't on a Mac, where the two tie on JSON; on the Linux box it trails on both pages, with the worst tail latencies of the emitted lanes. The HTML page makes many more calls across the Db boundary than the JSON page — a step? plus a column read per cell — and if the cost is TruffleRuby's C-extension boundary, the fix is a coarser-grained shim that fetches a row per call, which would pay on every runtime where that boundary is foreign. I have the tarball and two wrk lines for anyone with a profiler and an opinion; Benoit has first claim.

Spinel per core. Its 2.4× over JRuby comes from twelve workers on six cores against Puma's five threads. Per physical core that is still ahead, but the honest metric is CPU time per request, not throughput divided by a worker count, and the same run with SPINEL_WORKERS=5 is the cheapest experiment on this list.

Campfire. Everything here is the five-endpoint blog. Roundhouse's larger specimen is Basecamp's Campfire, and TruffleRuby has no fork, so its clustered-Puma deployment shape can't run — only the one-process, threads-only shape can, which happens to be the shape where GVL-free threads matter most. That is a different experiment, and the next one.

Reproduce it

curl -sL https://rubys.github.io/roundhouse/browse/ruby.tgz | tar xz
cd ruby
truffleruby -S bundle install
make seed
WEB_CONCURRENCY=0 RAILS_MAX_THREADS=5 truffleruby -S bundle exec puma -C config/puma.rb config.ru

WEB_CONCURRENCY=0 because there is no fork; give it a minute of load before believing a number; then, from another terminal:

wrk -t2 -c64 -d20s --latency http://127.0.0.1:3000/articles

The same steps under CRuby and under jruby.tgz give the other rows, and stock Rails is the fixture the benchmark serves. If your TruffleRuby numbers don't look like these — on either side — Discussions is where I'd like to hear about it.

Coda

June's post ended by saying the Ruby JRuby was built to run turns out to be a Ruby you can generate from the Rails application you already have. That is still true, and JRuby is still the runtime that rewards it most. What TruffleRuby adds is a correction to the moral: a runtime that specializes the interpreter to the program is already doing much of what a compiler that specializes the framework to the application does, and when you stack them the gains overlap rather than multiply. Roundhouse's emit is worth 5× on TruffleRuby, not 17×, because Graal got there first.

That seems like exactly the kind of thing the people who built it should check.

Roundhouse is open source: dual-licensed MIT / Apache-2.0. Issues and discussion welcome. The TruffleRuby logo is © 2017 Talkdesk, Inc., CC BY 4.0.