intertwingly

In 2023 Shopify's Rails at Scale team turned Lobsters into a Rails benchmark for YJIT: a real application, real routes, a fake-data database, SQLite in memory, so that work on the JIT would be measured against Rails as people write it rather than against a microbenchmark. It still lives in ruby-bench, maintained since, now on Rails 8.1. I've been using it for a different question.

Where this comes from

Roundhouse reads a Rails application and writes it out again. For a small blog it writes it out in a dozen languages, each checked page by page against Rails, but that works because a blog depends on almost nothing. For Campfire it can go further than Ruby: Matz's Spinel compiles the output to a single native binary. In August, Spinel compiled Lobsters itself and ran this same benchmark at 5.2× Rails in a fifth of the memory; that lane is paused while Spinel changes how it shares strings. None of those scales to an application with hundreds of gems, many of them C extensions.

Keep the Gems argued what does: transpile Rails to Ruby, keep every gem, and remove the cost Rails adds on every request. For a codebase the size of Shopify's, rewriting by hand, once or on every release, isn't on the table, and neither is a compiler that has to bring every gem along. That post didn't use the term, but earlier ones did: it is the first Futamura projection. Rails is, operationally, an interpreter for your application. Routes, associations, callbacks and templates are data it consults on every request. Specialize that interpreter with respect to the program it interprets, and what comes out is your application with the interpretation done ahead of time.

Roundhouse now has a target with that mandate, named futamura, the most literal form yet of not having to change your application. It leaves your application as it is, Rails and gems included, and replaces the expressions it can prove static with code that does the static part once. Everything it can't prove stays Rails, and runs on Rails.

That raises the obvious question, especially for a benchmark that exists to measure a JIT, and two days after ZJIT caught up with YJIT, using Lobsters as its proxy: isn't removing per-request overhead what YJIT and ZJIT are for?

Both, multiplied

In July I argued that Roundhouse composes with YJIT rather than replacing it. Here is that claim measured, with both JITs: the two stacks on the Lobsters benchmark, with no JIT, with YJIT, and with ZJIT, on one machine and one build of Ruby:

Milliseconds per pass of the benchmark's request sequence (106 logged-in requests), lower is better, on Ruby master as of October 11 (2d08b5f42f) and the Ryzen 5 3600 that runs the nightly bench page; in parentheses, each JIT's gain over no JIT. Roundhouse without a JIT is three times as fast as Rails with either one, and either JIT then speeds Roundhouse up again. The two multiply.

The bench page runs on Ruby 4.0.5, where Roundhouse's lead under YJIT measures 3.6× to 4.0× from run to run. Ruby master widens it a little, to 4.1×.

They overlap too, and the gains show where: both JITs gain less on Roundhouse's output than on Rails, because some of what they accelerate in Rails, Roundhouse has already removed.

Where the time goes

To see why, I profiled both. The table splits one pass of the benchmark by layer, Rails on the left and Roundhouse's Ruby output on the right, both under YJIT on Ruby 4.0.7, on a slower machine than the one above. (Times are from the profiled runs; it is the proportions and the gaps that matter.)

The application's own code costs about the same on both sides, as it should (Roundhouse's figure also includes the row-loading code it generates for each model). Almost everything else is work Rails does around it.

What a JIT can't know

A JIT makes the work a program does cheaper per instruction. It can't decide that the work doesn't need doing, because it has to preserve what Ruby means at every call: any method can be redefined, any association can gain a scope, any callback can be added at runtime. It sees one method at a time, and the types that have flowed through it so far.

Roundhouse reads the whole application before it runs. That's what lets it decide things a JIT can't:

• What SQL to send. Rails spends 36 ms per pass preparing statements; much of the SQL text it builds differs from request to request (lists, for one, are inlined into it). Roundhouse spends 3 ms, because it knows which statements are the same and keeps them prepared. The stepping difference says it asks SQLite for less as well, though I haven't broken that down yet. No JIT changes the query.
• That a relation is static. Rails builds Relation objects and compiles Arel to SQL on every request. A JIT runs that code faster. A specializer that can see the chain is fixed builds it once.
• What to preload, and how. Rails' Preloader works out the association tree and its scopes on every call. With the tree known ahead of time, that work happens once.
• Which filters, callbacks and routes apply. They're declared, so they can be resolved ahead of time instead of consulted per request.
• What not to allocate. Garbage collection costs the same 48 ms on both sides, and most of what Roundhouse avoids allocating is framework objects: relations, attribute sets, safe buffers, time objects. ZJIT's roadmap includes partial escape analysis, starting with short-lived arrays and *args, and that will help. But a JIT can only skip an allocation it can prove never escapes, and framework objects usually do: a relation is stored, a record is returned, a buffer is handed to the next layer.
• What the C underneath does. The ZJIT post is candid that it "can only reason about Ruby code." SQLite's prepare and step are C. So are many of the string and hash operations Rails leans on. A specializer can change how often they're called and with what; a JIT can only call them.

What a JIT does that Roundhouse doesn't is just as real. It generates machine code, specializes each call site for the types it actually sees (which depends on data no compiler can see ahead of time), and removes the interpreter's own dispatch. Roundhouse's output is Ruby, and still needs YJIT or ZJIT for all of that.

There's overlap at the edges. The 2023 post found about 30% of Lobsters' method calls polymorphic, which is part of what a JIT pays for; Roundhouse's output is more monomorphic, so it does some of what a smarter JIT would. And ZJIT's inlining and load-store elimination will reach into framework code a little further every release.

As for the two JITs against each other: in this harness ZJIT trails YJIT on Rails, 575 ms to 548 (the Rails at Scale post, with its own harness and settings, has the two level), and edges ahead of it on Roundhouse's output, 126 ms to 132. Whatever the reason, the difference between the two JITs on either stack is about 5%. Under either JIT, the difference between Rails and Roundhouse is more than 4×.

Where the futamura target is today

We've just started, so on Lobsters it is close to a no-op: 1.00× Rails, within noise. Its specializations so far are database ones (static query chains handed to Rails' own StatementCache, preload trees planned once), and they barely touch this benchmark's busiest routes. On Campfire they help a little: a few percent more requests per second under load across its pages, much of it from two lookups that run on every request.

What the work has produced so far is a way to tell, plus a direction:

• Every specialized site keeps the original Rails expression as a fallback, and a verify mode runs both and raises on any difference, so an application's own test suite becomes a differential test of every specialization it reaches. Campfire's suite and every Lobsters benchmark request give the same answers either way.
• Profiles of the two lanes put the ceiling for this approach, keeping real Active Record objects so that gems hooked into them keep working, at an estimated 2.5–3× Rails. The full lowering gets about 4×, by also giving up those objects. Both are on top of YJIT.
• The next steps, in order: the hot query chains it doesn't yet handle, controller dispatch, the overhead of executing a query, and the date arithmetic Rails code does with Active Support objects.

Complementary

The Rails at Scale team made Ruby fast. What remains is Rails doing work on every request that it could have done once, and that's a question about the application, not about the instructions that run it. A JIT answers the second question, and a specializer the first. On this benchmark, built to measure the JIT, the two answers multiply.

One note on the numbers: they are not directly comparable to the 2023 post's. The benchmark has moved to Rails 8.1, the markdown gem changed, our harness runs SQLite's ANALYZE after seeding (without it SQLite picks a bad plan for the hottest-stories query in every lane), and it replays one fixed, logged-in request sequence, after 15 warmup passes, so that every lane sees exactly the same requests.