intertwingly

Six days ago I published a ledger that didn't flatter: the compiled Lobsters served ten of twenty-six routes, thirty-nine of 354 of its own tests passed, and the benchmark page opened with a banner declining to stand behind its own timings. The deal I made was to keep publishing the ledger either way.

This is the update, under the same rules.

The numbers, as of this afternoon

The compiled binary now serves twenty-four of twenty-six routes rendering what Rails renders — compared as parsed documents against Rails' own output, not as status codes. The two exceptions are named on the page with the reason and the work that closes each, and their visits are subtracted from every lane's timing, Rails included, so all five lanes still measure identical work.

On that identical work, under the ruby-bench harness's own timing rules — Shopify's benchmark, their warmup discipline, their statistic — the compiled binary runs the frozen sequence at 5.2× the speed of Rails on YJIT, in about a fifth of the memory (72 MB peak against 346 MB). The plain-Ruby emit — Rails semantics compiled to ordinary Ruby, no AOT involved — sits at 3.8×.

That 3.8× is worth a sentence, because it used to be 4.0×, and it went down when the comparison got fairer. Making a benchmark honest moves numbers in both directions, and a page that only ever moves in its own favor is telling you something.

What "trustworthy" took

The banner six days ago was automatic: the page red-flags itself whenever its own data shows the lanes didn't do the same work. Earning its removal took three rounds, each of which found something quietly wrong.

First, a one-line bug that let the Rails lanes time eight visits the other lanes had deferred — three published runs' worth of ratios divided different work before the page caught it. Second, the timing loop matched ruby-bench's rule approximately rather than exactly — warmup counts and the reported statistic have now been made verbatim. Third, and my favorite: the five lanes turned out to be quietly running three different builds of SQLite. Nothing recorded it, so a library-version difference could wear a runtime difference's clothes. Every lane now asks its own linked library what it is at runtime, the page attests the match — and the largest "compiler loss" in the per-route table, a query-heavy route the profiler showed spending 94% of its time inside SQLite itself, turned out to be an eighteen-month SQLite version gap. Aligned, it's a win.

The pattern across all three: every explanation anyone guessed — including mine, including ones I'd published — died when measured. The instruments are what's left standing, and they're on the page: which commit, which compiler, which library, which work.

The forcing function, promoted

The last post said a forcing function does its job and then stops being interesting — that's why the blog demo chapter closed. Lobsters' job turns out to be bigger: it is currently forcing a generational garbage collector into the compiler.

The short version. With everything aligned, the seven routes where the compiled binary still loses to YJIT — all of them cheap, sub-millisecond requests — trace to a single cause: Spinel's collector re-walks the entire live heap on every collection, so a request that does almost no work of its own still pays rent on the whole application's memory. We published the analysis with the workload attached; Matz confirmed the mechanism with his own measurements, landed the write barrier a generational collector needs (cost on this workload: about 1.4%), and landed the minor mark itself behind an opt-in flag — citing the issue thread at each step. I ran it the same day; his verification tooling caught a remaining coverage gap on this app in seconds, and the report was filed while Tokyo slept. The prediction, on the record: when that work completes, the compiled binary wins every route on the table.

That loop — publish the workload, measure together, fix on whichever side of the fence the cause lives — has run four times in six days. It is the most productive collaboration mode I know, and all of it happens in public issues.

The other lane

The conformance lane — upstream Lobsters at HEAD, running its own test suite against the transpiled output, the lane that tests whether you have to change your application — went from thirty-nine passing examples to 126 of 354. Still a failing grade, still published, still the finer oracle: each failure names a specific construct, and that page's own rules bar the shortcuts (a pass rate achieved by propping up the subject is not a pass rate about the subject).

One item from the last post deserves its payoff. I wrote then that the compiled lane's Rails.cache was a no-op — every fetch recomputed, "a different program from the one Rails is running." That's fixed: the compiled binary now runs the real cache, and serves Lobsters' heaviest cached page from a hit in a quarter of a millisecond.

Still true

It's still early, and the page still moves underneath this post — there is an intermittent crash in the compiled lane under investigation right now, and on any given afternoon you may click through and find a lane honestly reporting itself dead. That's the design. The question from six days ago hasn't changed: whether the application you already have can come with you. What changed is the evidence.

Roundhouse is open source: dual-licensed MIT / Apache-2.0. Issues and discussion welcome.