What the Campfire Benchmark Measures
Goodhart's law: when a measure becomes a target, it ceases to be a good measure. For Campfire it comes down to one sentence: a single-viewer benchmark rewards a whole-page cache that a busy chat deployment may rarely hit.
I owe a correction first. Taking Flight said the Ruby that Roundhouse emits from Campfire's Rails source beats Tobi's hand-written port on the room, messages, sidebar and search routes. It does, on the benchmark, but that isn't a rendering win. On those routes the emitted app is serving pages from Campfire's whole-page response cache, which arrived in Rails on October 7. Tobi's port was last updated on October 6 and has no such cache. With the cache turned off, his port renders those pages 2.8 to 4.2 times faster than ours (numbers below).
That isn't a complaint about the cache, or about anyone's benchmark. It's a sign that the benchmark has become the target, and it's worth asking what we'd want it to measure instead.
What the benchmark measures today
The shared verification harness behind the table in Campfire's README is careful work. Every response is checked for status, type, length, encoding and content; every acknowledged write is audited against the database and the full-text index; foreign SQL edits and revoked sessions are exercised. Nothing below is a criticism of that.
What it says about the load, in its own words:
The 16 read clients share one freshly authenticated fixture viewer/session per app and round. Warm reuse for one viewer does not measure multi-viewer eviction pressure.
and, about the profile with a writer:
This measures cross-room invalidation and concurrent writes; it does not model every reader following one actively written room or WebSocket fanout.
Every one of the eight implementations in that table (Rails, Django, Laravel, Express, Elixir, Go, Rust and C) puts a whole-page cache in front of the room, messages, sidebar and search pages, keyed per user (and in all but C, per session) and cleared by any database commit. With one viewer, every read after the first is a hit, so the table measures what a cache hit costs: authentication, the room check, building the key, the lookup and writing the response. Those differences are real (Rails 4,101 room pages a second, Express 42,636, C 137,524), but they say little about how fast each implementation renders a page.
The paced-writer profile doesn't change that much. A writer posting ten messages a second clears the cache ten times a second, but one viewer re-renders once and then hits again for the next tenth of a second. Rust goes from 106,494 room pages a second to 99,660. Rails, whose renders cost more, goes from 4,101 to 2,222.
How the implementations cache
The page cache hides the next layer: each message in a room is a cached fragment, and the implementations key those fragments very differently.
| implementation | message fragment key | loaded before lookup | survives unrelated commits | room ETag |
|---|---|---|---|---|
| Rails (since Oct 8) | SHA-256 of the message, room, creator, rendered body and boosts | everything | yes | hash of the whole body |
| Rails (before Oct 8) | id + updated_at, inside a commit epoch |
(not checked) | no | hash of the whole body |
| Go | id + updated_at, inside a commit epoch |
id, updated_at |
no | hash of the parts |
| Rust | template digest + id + updated_at, inside a commit epoch |
id, updated_at |
no | hash of the parts |
| Elixir | template digest + id + updated_at, inside a commit epoch |
the message | no | hash of the whole body |
| Express | template digest + id + updated_at, inside a commit epoch |
the message, creator, room | no | hash of the whole body |
| Laravel | SHA-256 of id, room name and the updated_at of message, creator, boosts, boosters |
message, creator, room, boosters | no | none |
| Django | no fragment cache | — | — | none |
| C | no fragment cache | — | — | hash of the whole body |
| Tobi's Ruby port | id + updated_at |
id, updated_at |
yes | hash of the parts |
| Roundhouse (emitted from Rails) | Rails' key, transpiled | everything | yes | hash of the whole body |
"Inside a commit epoch" means the key includes SQLite's data_version,
so any write anywhere makes every fragment unreachable. Rails' fragment
keys also include the user and a digest of the session token; none of
the other implementations scope fragments per viewer.
There are three designs here:
- Cheap lookup, wiped on every commit (Go, Rust, Elixir, Express, Laravel, and Rails until October 8). A hit costs almost nothing; a write anywhere re-renders everything.
- Content keys that survive commits (Rails since 74381e8, "Retain content-validated message fragments across unrelated commits"). Fragments outlive writes, and the key can't go stale even when, in the code's words, "foreign SQL writers need not touch Rails timestamps", which the harness tests. The price is that every render loads every association and encodes and hashes every message before it can look anything up.
- Cheap lookup that survives commits (Tobi's port). It trusts
updated_at, and keeps it honest with SQLite triggers for the relationships Rails declares withtouch: true.
Roundhouse transpiles Campfire as it is, so the emitted app carries Rails' current choices exactly: the page cache, content keys, per-viewer fragments and a full-body ETag. In a profile of one room render with the page cache off, loading associations and building those content keys are about a third of the time, and the full-body ETag is another tenth.
What the cache hides
On my workstation (Ryzen 7, 16 threads, Ruby head, 11 workers or Ractors, 16 clients), with fragment caches on everywhere and only the whole-page cache varied:
| route | Roundhouse, cache on | Roundhouse, cache off | Roundhouse, one process, cache off | Tobi's port |
|---|---|---|---|---|
| room | 16,673 | 3,581 | 750 | 10,053 |
| messages | 17,859 | 4,534 | 947 | 18,850 |
| sidebar | 26,067 | 6,917 | 1,405 | 21,536 |
| search | 17,098 | 3,539 | 784 | 10,399 |
Requests per second. Tobi's port has no page cache, so its number is the same either way. The one-process column is there because it's the only shape in which the emitted app can actually be deployed today: its Action Cable registry and job queue live in the process, so a broadcast from one worker wouldn't reach subscribers in another. Tobi's port is one process with worker Ractors, which is the shape he can ship.
The page cache is worth four to five times on these routes for the emitted app. Whether it's worth anything to a real Campfire depends on how often the same viewer asks for the same page between two writes, and that's the part the benchmark doesn't measure.
What a benchmark should cover
A proposal, to argue with:
- Many viewers. Hundreds of distinct users with their own sessions, spread unevenly across rooms. A cache keyed per session only helps a viewer who comes back before the next write.
- Writes where people are reading. Messages posted into the rooms being read, at rates a busy deployment sees, not only into a neighbouring room.
- What a write causes. Whatever the browsers actually do after a message arrives (the broadcast over the socket, and any requests that follow it), driven by the same clients.
- Hits and renders reported separately. The cache-hit cost and the render cost are both real and both worth optimizing. A single number that mixes them in an unknown ratio can't tell you which one you improved.
- Tails and memory. p99 latency next to throughput, and memory for the deployment shape you'd ship (one process or several, with every cache and queue it needs), since a self-hosted chat server often runs on a small box.
- The connections. WebSocket sockets held open, and the cost of fanning one message out to everyone in a room.
- The existing contracts, unchanged. Exact responses and audited writes are what make the current numbers trustworthy, and any new profile should keep them.
My guess is that under a load like that, the design that wins is fragments that are shared across viewers, survive unrelated commits and can be looked up from cheap columns, with an ETag built from the parts, and that the whole-page cache matters much less. But that's the point: it should be measured, not guessed.
Next
Roundhouse's own nightly benchmark will carry the one-process lane, and Tobi's port on the WebSocket tier, starting with the next few runs, and I'm building the many-viewer profile there. I'd much rather it lived in the shared harness, where every implementation is measured the same way. If the Campfire maintainers agree on what a benchmark should cover, I'll contribute the profile, and the pull requests that follow from it can point to numbers everyone has already agreed on.