intertwingly

Three days ago I ran the five-endpoint blog fixture through a 2×3 — stock Rails and Roundhouse's emitted Ruby, each on CRuby+YJIT, JRuby and TruffleRuby — and the result inverted June's: Graal rewarded stock Rails 3.4× and the emit only 1.8×, because a runtime that specializes the interpreter to the program is already doing much of what a compiler that specializes the framework to the application does. The post ended by saying the next experiment was Campfire, Basecamp's chat app, on the same three runtimes, and that TruffleRuby's lack of fork would make it a different experiment.

It is a different experiment, and it has a different answer.

The same tree, twice more

The Campfire benchmark page has carried three lanes per side: Rails as its own Dockerfile deploys it (Thruster in front of Puma, eight forked workers on this box, jemalloc, resque-pool alongside, the message fragments cached in Redis), the emit in the same Puma shape, and the emit compiled to a native binary by Spinel — plus a one-process twin of each, so the deployed rows can be read against a shape that pins one CPU. It now carries three more.

jruby is the emit re-targeted for the JVM. As in June, that is the same tree with the SQLite shim swapped for a JDBC one, because the sqlite3 gem is a C extension with no JVM build; the application code, the framework runtime and the Puma front are byte-identical to the CRuby tree. truffleruby is the CRuby tree, unmodified, with its bundle compiled by TruffleRuby — sqlite3, bcrypt, nokogiri, nio4r, websocket-driver all build. rails-truffleruby is Campfire itself, the same checkout and the same Rails revision the rails lane serves, with a second bundle TruffleRuby compiled beside the first.

Both TruffleRuby lanes are one process, because there is no fork — a GraalVM process has compiler and GC threads that a child would not inherit — so Campfire's own worker formula and resque-pool cannot run. They are read against ruby-1p and rails-1p, the same one-process shape on CRuby. JRuby has the same constraint and runs one process too; its ladder peaked at one Puma thread per hardware thread (12 → 1,730 req/s; the 40 that Rails' worker formula implies costs 19%), so that is the shape it runs.

TruffleRuby's own thread ladder went the other way — 4 threads, 458 req/s; 12, 384; 40, 293 — and the first version of this post explained that wrongly, as a global interpreter lock. TruffleRuby has not had one for over a decade. What serializes those threads is the global C-extension lock, which TruffleRuby gives to any extension that has not declared itself thread-safe, and sqlite3 2.9.3 declares nothing. Turning it off is one documented flag, and the section below measures what that is worth. The lanes on the page still run the default, locked, because that is what a bundle install gives you. Heaps are pinned, as the blog's are: 1 GB for the emit (512 MB holds it, at a 24% cost to garbage collection), 2 GB for stock Rails.

The machine, harness and app are the page's: a bare-metal Ryzen 5 3600 with the governor pinned and boost off, wrk at 16 connections, three 20-second runs per cell after a warmup the harness sizes per lane — 20 s for the CRuby and Spinel lanes, 60 for JRuby, 120 for the emit on TruffleRuby, 240 for stock Rails on it — with the median reported. Campfire at ed0f9a5, Rails 8.2.0.alpha at the revision its lockfile pins, Ruby 4.0.5, JRuby 10.1.1.0 on JDK 25, TruffleRuby 40.0.0 native. Every request is signed in; every lane renders the same 3,963 tags on the room page, which the page checks before it measures anything. The figures below are one run's — the nightly of September 22.

The 2×3, on a chat app

/rooms/1, the room page — one process per runtime, which is the only shape all three can run:

and the deployed rows, for scale:

/rooms/1/messages, the fragment-cached message list, reads the same way: TruffleRuby is worth 1.18× to stock Rails (134 → 158) and, as shipped, 0.82× to the emit (985 → 803); JRuby's one process does 3,545. Hold that 0.82× loosely — the next section takes it apart.

Read the TruffleRuby row against the blog's. On the blog TruffleRuby was the best runtime for stock Rails by a wide margin — 3.4× on HTML, 5.2× on JSON — and I wrote that if the question was "which Ruby runs my Rails app fastest without changing a line," the answer was TruffleRuby and it wasn't close. On Campfire's room page it is worth 1.8×, and on the message list 1.18×. And where the blog's emit gained 1.8× from Graal, Campfire's emit loses: 0.9× on the room page, 0.82× on the messages. The one thing that survives from the blog is that the emit is still well ahead of Rails on TruffleRuby, 3.6× — but the dividend that was 7.4× on CRuby has been halved, which is the same overlap the last post found, only now it overlaps with a runtime that is no longer winning.

Where the overhead went

The blog post's "what to explore next" already had the suspect: the HTML page makes many more calls across the Db boundary than the JSON page, and if TruffleRuby's cost is the C-extension boundary, the fix would be a coarser-grained shim that fetches a row per call.

Half of that was right, and there turn out to be two costs at that boundary rather than one.

The per-call cost is real and the shim can't dodge it: it already fetches a row per call — one step into the sqlite3 gem, then column reads out of a Ruby array — so there is no coarser grain to move to. Stepping 100 message rows out of Campfire's own database, two thousand times, single-threaded, is 0.83 µs a row on CRuby+YJIT and 2.4 µs a row on TruffleRuby. The same script's pure-Ruby half — building a few hundred thousand HTML fragments with string appends — runs more than an order of magnitude faster on TruffleRuby. That is what its authors would predict: it specializes Ruby superbly and pays a toll at each crossing into a C extension.

The concurrency cost is much larger, and it is not TruffleRuby's to keep. C extensions there take a global lock unless they declare themselves thread-safe with rb_ext_ractor_safe() or rb_ext_thread_safe(); sqlite3 2.9.3 does neither, so every row every thread steps is serialized against every other thread's. TruffleRuby documents the escape hatch for exactly this case, and on the room page it is worth two and a half times:

Unlocked, the emit on TruffleRuby is 2.3× CRuby's one-process shape, not 0.9× — the lock, not the runtime, was the loss. Single-threaded the lock costs about 3% (2.44 µs a row against 2.36), which is how it hid: it is invisible in a microbenchmark and decisive in a server. Stock Rails barely notices it — 89 req/s against 93 with the flag, inside that row's own 11.7% spread — because at eighteen requests per second per thread it is nowhere near the lock, so the 1.8× the runtime swap buys it stands.

The blog's /articles steps a handful of rows per request. Campfire's room page steps a few hundred — the messages, their creators, boosts and attachments — and stock Rails steps the same rows through Active Record, through the same gem, across the same boundary. So the interpretive overhead Graal dissolved on the blog is a smaller share of this page's cost, and the per-call toll it cannot touch is a larger one; the runtime swap that was worth 3.4× to a page made of dispatch is worth 1.8× to a page made of rows. That part of the blog-to-Campfire comparison is unchanged.

What changed is the emit's row, and the lesson is narrower and more useful than the one I first drew: on a row-heavy page under TruffleRuby, the gem that reads your database is the benchmark. Mark it thread-safe, or disable the lock, or reach the database without a C extension at all — which is exactly what the JRuby tree does, over JDBC, and it is no coincidence that JRuby is the row that scales. That swap was a portability workaround in June; on this page it is the reason the JVM lane uses twelve threads and the TruffleRuby lane, as shipped, cannot use five.

The other JVM

The result I did not expect to be the useful one is JRuby's, and it is useful for a reason the numbers alone don't show.

The page's eight-worker ruby row — 2,366 req/s, 7.4× stock Rails in the same shape — has carried a caveat since it was first published: it is a matched-shape runtime comparison, not a deployment claim. The emitted tree keeps its Action Cable registry and its job queue in-process, so with eight forked workers a message posted through one worker reaches only the subscribers whose sockets happen to be in that worker. Campfire could not actually run that shape; the deployable CRuby shape today is one worker, the 478 row, and cross-worker pubsub is a seam I have not built.

One JRuby process with twelve real threads has one registry, one fragment cache, one job queue, shared by every thread, because the JVM runs them in parallel and needs no fork to use the machine — and because its SQLite driver is JDBC, with no C extension to serialize behind. So jruby at 1,712 req/s is the first row on the page that is both multi-core and something Campfire could deploy as it stands: 3.6× the one-worker CRuby shape, 5.4× the deployed Rails row, at 8 ms median against Rails' 44, and 0.72× of the eight CRuby workers whose shape it can't have. The price is memory — 1.6 GB against ~1 GB for the eight workers, because there is no copy-on-write sharing to be had and the JIT's code and metadata live outside the 1 GB heap — and a boot of five seconds to accept and six to a rendered room, against 1.7 for the CRuby cluster and a third of a second for the binary.

That reorders the choices the last post ended with. For the emitted Campfire on a Ruby runtime you already have: CRuby+YJIT, one worker, 478 req/s in 232 MB, or the full eight in a gigabyte once the pubsub seam exists. For the emitted Campfire on every core today, with no seam: JRuby, 1.6 GB, six seconds to boot. For the smallest and fastest: the Spinel binary, 2,570 req/s in 140 MB, a third of a second to a rendered page, which is the row the page exists to publish. For the emitted Campfire on TruffleRuby, the flag above is not optional: 432 req/s without it, 1,078 with. And for stock Campfire, TruffleRuby's 1.8× is real but it comes at 2.9 GB — 41 requests per second per gigabyte, against 172 for the same Rails deployed on CRuby and 18,683 for the binary — and a boot of nine seconds. It is also the least settled row there: its three timed runs went 91, 118, 119 req/s even after four minutes of warmup, an 11.7% spread the page flags in its own stability section, so read it as a floor.

What we found along the way

The JRuby target had drifted from the CRuby one. Roundhouse's JVM tree is built by the same code as its CRuby tree with two swaps — the database shim and the markdown renderer — except that it wasn't: the JVM builder was a hand-copied subset, written once, that had missed every stdlib swap added since. The blog never exercised the gaps. Campfire hit the first one at boot, a superclass mismatch on a Concurrent:: error class defined once by the emit's own port and once by the real gem sentry-ruby had already loaded, and behind it were four more, including the app's initializer mixins never being applied on the JVM at all. The fix is one builder with a flavor switch; the CRuby tree it emits is byte-identical to before. "Features land once" is a rule I wrote down for runtime files, and it applies to the code that assembles them.

A precompiled gem in the lockfile is a landmine for a second runtime. Campfire's Gemfile.lock resolves nokogiri, sqlite3 and thruster to their precompiled x86_64-linux variants. TruffleRuby can't load those; bundle install under it quietly adds the source variants to the lock — in place, in Campfire's lockfile, which the benchmark re-exports pristine every night. Against the pristine lock bundle check says everything is satisfied, and bundle exec puma then re-resolves at startup, which loads Bundler's HTTP fetcher, which requires openssl, which activates TruffleRuby's default openssl 4.0 over the lock's 3.3, and the lane dies with "You have already activated openssl." My manual runs had all passed because their bundle install had already rewritten the lock. The first night with the lane on the page, the row was a hole and a banner; the second bundle now owns a lockfile of its own beside the tree, re-seeded whenever Campfire's changes, so the gem versions and the Rails revision stay exactly the CRuby lane's. If you run a Rails app under two Rubies from one checkout, this one will find you.

bundle check is not a gate. It passed on the lock that bundle exec then refused. The harness runs bundle install every night now; it takes two seconds when there is nothing to do.

Reproduce it

The blog's trees are downloadable tarballs; Campfire's Ruby and JVM trees are not published yet — the harness emits them from the repository — so this one needs a checkout and the Campfire source:

git clone https://github.com/rubys/roundhouse ~/roundhouse && cd ~/roundhouse
git clone https://github.com/basecamp/once-campfire ~/once-campfire
scripts/campfire-oracle prepare --app ~/once-campfire     # bundles, migrates, seeds 50 users / 5 rooms
cargo run --release --bin roundhouse -- --target jruby ~/once-campfire -o /tmp/campfire-jruby --allow-unsupported
cd /tmp/campfire-jruby && jruby -S bundle install
mkdir -p storage && cp ~/roundhouse/build/campfire-oracle/storage/db/production.sqlite3 storage/development.sqlite3
RAILS_ENV=production RAILS_MAX_THREADS=12 WEB_CONCURRENCY=0 PORT=3000 \
  JRUBY_OPTS="-J-Xmx1g -J-Xms1g" jruby -S bundle exec puma -C config/puma.rb config.ru

--target ruby in place of jruby gives the tree that runs under CRuby and, with truffleruby -S bundle install, under TruffleRuby — where the flag is the whole point:

TRUFFLERUBYOPT="--vm.Xmx1g --experimental-options --cexts-lock=false" \
  truffleruby -S bundle exec puma -C config/puma.rb config.ru

Sign in as user1@example.com / secret123456 (the seed's), carry the cookie, and give the JVM a minute of load before believing a number:

wrk -t2 -c16 -d20s --latency -H "Cookie: $COOKIE" http://127.0.0.1:3000/rooms/1

Stock Campfire under TruffleRuby is scripts/campfire-oracle bundle --runtime truffleruby followed by serve --runtime truffleruby, which does the lockfile dance above for you. If your numbers don't look like these, Discussions is where I'd like to hear about it. The remaining open question is the per-call one the flag does not touch: 2.4 µs a row against CRuby's 0.83, single-threaded, warmed. Benoit's EuRuKo talk is on this exact subject, and the better fix is upstream — an extension that declares rb_ext_thread_safe() needs no flag from anyone.

Coda

The last post's moral was that a runtime which specializes the interpreter to the program and a compiler which specializes the framework to the application overlap rather than multiply, and that Graal got there first. Campfire adds the boundary condition: both of them can only remove overhead that is there to remove, and on a page made of rows there is less of it. That part held up.

The part that did not held a better lesson. I measured a threading collapse, reached for the nearest explanation, and picked one that had been false for a decade — and the true cause was not in either compiler but in a gem's failure to say one thing about itself. Neither the runtime nor the emit was the variable. The database driver was, and it was the variable in the JRuby row too, which scales because a JDBC driver has no C extension to lock. Two posts of mine have now described that JDBC swap as a portability chore. It was the measurement all along.

Roundhouse is open source: dual-licensed MIT / Apache-2.0. Issues and discussion welcome.