intertwingly

It’s just data

Campfire on Spinel

Roundhouse now has four applications in flight. Status for each is here.

Each one is there to test something different. Blog is the conformance oracle — small enough to support completely, and the only one where every target compiles clean, passes its tests, and matches live Rails. Lobsters is real-world coverage. Campfire is live and interactive. Mastodon is scale, and is frankly a moonshot.

None of the other three are complete. Even when they are, they may never get beyond Spinel as a target — that depends on whether anyone wants the others.

Campfire is the most recent addition and the surprise of the bunch. It is about half Lobsters' size in application Ruby — 3,700 lines against 7,600 — but it leans on far more of Rails: Action Cable, Action Text, Active Storage, Turbo Streams. More framework surface in fewer lines is the harder problem, and I expected it to go slower. It has gone faster.

I should be careful about what "faster" means here, because the ahead-of-time lane is not a countdown. Lobsters serves twenty-six of twenty-six routes with content matching Rails, at 3.63× Rails' speed under YJIT — that part is solid. Its native compile is not: it built clean one day and had two fresh C compiler errors the next, which I have not yet triaged. That is not a setback. It is what both projects look like right now. Roundhouse keeps changing what it emits, Spinel keeps changing how it compiles, and a tree that built yesterday is not evidence about today. Compiling a real Rails application ahead of time is an ongoing effort on both sides of that seam, and neither side has finished maturing.

So this is not a race between the two applications, and Campfire's interest does not rest on a status number. It rests on the kind of application it is.

What it is

ONCE Campfire is a chat application Basecamp sells, self-hosts, and ships as a product, written by the people who wrote Rails. It is MIT-licensed and small enough to read whole: roughly 3,700 lines of application Ruby across seventeen models, forty-three controllers, and seventy-eight views. It is not the first thing Roundhouse has pushed over a WebSocket. The blog fixture does that too: broadcasts_to on two models, a turbo_stream_from in the view, a /cable endpoint, and a test in which five live sockets get the prepend. What the blog does not have is anything that makes a socket layer hard. It uses Turbo's stock channel with a signed stream name, nothing on the connection knows who you are, no client ever sends anything back, and the connection count is however many the test opened.

Campfire is qualitatively different. Seven application-defined channels, a connection identified by a session cookie, an authorization module prepended onto the stock channel to close a bypass, client-to-server actions for typing notifications and presence, and a published requirement of up to ten thousand concurrent users. The blog proved the plumbing exists. Campfire is where the plumbing's design decisions start to cost something.

The part I care most about: zero changes to Campfire's source, and no RBS. The tree is pinned to upstream commit 2aa41410 and fetched on every CI run; nothing patches it, and there is no sig/ directory, no .rbs file, no annotation of any kind anywhere in the application. Every type Roundhouse knows about that app, it inferred. The compiler's job was to grow until it could read what Basecamp actually wrote.

219 of its 240 tests pass, across 40 of 52 files. On the HTTP side the emitted tree renders these pages about 4.8× faster than Rails renders them — that is our output against Rails with its fragment cache turned off, which is the honest operand, because Rails' deployed advantage on these pages is entirely its Redis fragment cache and not its renderer.

The number Basecamp publishes

once.com states Campfire's system requirements as a table:

concurrent users RAM CPU
250 2 GB 1
1,000 8 GB 4
5,000 32 GB 16
10,000 64 GB 32

Perfectly linear: about 6.5–8 MB and a three-hundredth of a core per idle connection. No economy of scale is claimed. That is a CRuby-shaped budget — a connection object graph, a share of the database pool, a Redis subscriber, and copy-on-write decay across forked workers.

It is an assertion about hardware, and Roundhouse's benchmark methodology measures resident memory, so the experiment worth running is that table read backwards: hold the user count fixed, run both stacks, and read the RAM off the result rather than provisioning to it.

Three things surfaced while getting ready to run it. All three surprised me.

Redis was not buying pubsub

Campfire uses Redis three ways: the Action Cable adapter, the fragment cache, and Resque. The interesting one is the first.

Redis is in cable.yml because puma.rb forks (processor_count * 0.666).ceil workers. The subscriber for room 5 lives in worker 3, and the message was posted to worker 7 — separate heaps, so a broker is the only way across. That is a Global VM Lock tax, not a chat-architecture requirement. The chain runs:

GVL → must fork for cores → separate heaps → need a broker → Redis

Spinel has no GVL. Remove the first link and the rest falls out: one process holds every connection, the subscriber list is a local array, and the fan-out is a function call. Nothing is given up in the trade, either — Campfire is pinned to a single machine by adapter: sqlite3 and Active Storage's service: Disk before Redis is even considered. There is no distributed tier that Redis was protecting.

What you are billed instead is subtler, and I did not see it coming: Redis was buying a memory model. Separate processes are why cross-worker traffic needed a broker, and also why nobody had to think about locking. One process with real threads and no GVL inverts both halves at once. The subscription registry and the cache become shared mutable state, and they need a concurrency design that the forked architecture made unnecessary.

91% cannot see the thing I care about

Twenty-one tests still fail. Not one of them is a socket, fan-out, or concurrency failure. They are Active Storage variants, Action Text attachments, signed global IDs, a view helper — content, not topology.

That sounds like good news and isn't, quite. Campfire ships channel tests, and they pass. But Rails' ActionCable::Channel::TestCase asserts that subscribing called stream_for — it never opens a socket. A green presence_channel_test.rb is entirely compatible with having no connection identity at all, which is in fact where we are: ApplicationCable::Connection#connect does not yet run, so nothing on a connection knows who the user is.

So the conformance ladder and the socket lane are close to disjoint. 240 out of 240 would still tell me nothing about whether 250 connections stay connected. The suite has to grow a dimension it does not have, rather than the number going up.

That reframes the goal. The gate is no longer complete conformance; it is sufficient conformance — enough to stand up a binary that holds sockets, so the architecture can be matured against something real. The rule I am using to sort a gap into or out of that gate: it is in if it changes who is connected, who is subscribed, or how many frames move; it is out if it only changes what a rendered page contains.

The limit may not be in the runtime at all

I went in expecting the interesting constraint to be the event loop. It may instead be four lines of application code:

def present
  membership.present          # a database WRITE, on connect
  broadcast_read_room
end

def absent
  membership.disconnected     # a database WRITE, on disconnect
end

Connection lifecycle is a write workload. Campfire's database.yml sets default_transaction_mode: immediate on SQLite — a single writer. Ten thousand connects is ten thousand serialized writes, and a deploy or a network blip is a disconnect storm of the same size, plus ten thousand broadcasts.

No amount of kqueue, fiber scheduling, or M:N threading touches that. It is worth measuring before optimizing anything underneath it, because if the write path dominates at the top tier then the runtime work stays necessary but stops being the interesting number. That is a good lesson to relearn periodically: the bottleneck is not always in the layer you have been staring at.

Where the work actually is

Some of it is in Spinel's C, and reading that code turned up two walls that had nothing to do with the transpiler. The readiness set was capped at 256 file descriptors and overflowed silently — Campfire's smallest published tier is 250 concurrent users, so it was reached before tier one. And socket writes retried EAGAIN by blocking inside C, below any caller's scheduler, so one slow subscriber could stall a whole worker's event loop.

Past tense because both were fixed the same day I filed them. The cap is gone. Readiness is registered once per connection rather than rebuilt every tick, so the caller's work is proportional to events instead of to connections. The receive buffers are per-worker instead of file-static, which is what the no-GVL threading story needed. And there is a partial write that hands the backpressure decision back to the caller — the right place for it, since only the caller knows whether to buffer the frame, drop it, or hang up on the peer.

That is the other face of the churn I complained about earlier. The same velocity that makes yesterday's clean build no evidence about today is what turns a wall into a merged commit before dinner. I would rather have both than neither. The work did not disappear, either — it moved to my side of the seam, where a fiber scheduler now has a better contract to be rewritten against.

For anyone who wants more

The last progress report was about a compiled application getting faster. This one is mostly about finding out which questions I had been asking were the wrong ones. When it builds — and it will build, then break, then build again — the answers get to be measurements instead of arguments.

Roundhouse is open source: dual-licensed MIT / Apache-2.0. Issues and discussion welcome.