intertwingly

In The Browser Half I looked at the part of a Hotwire app where nothing checks anything: the strings that wire Stimulus controllers to each other and to Ruby. A GraphQL server is the opposite case. Its contract is already typed and explicit, and graphql-ruby validates every incoming query against it.

What it doesn't check ahead of time is the other side: whether the Ruby behind each field returns what the field declares. A field declared null: false that comes back nil, or a field with no method behind it, is found when a request reaches it. That's the half Roundhouse can see, because it already infers the types of a Rails app's Ruby without annotations.

Following the schema down

A graphql-ruby type doesn't say what Ruby object it wraps. UserType is a User because some field returned a User and declared its type as UserType. So check reads the field declarations, gives each one the value graphql-ruby would resolve (the type's own method if it has one, otherwise object.<name>, or object.<method:>), and lets inference carry the record class from the schema's query and mutation roots down every edge. Cycles are fine: Link → postedBy → User → links → Link.

Fields that take arguments are called the way graphql-ruby calls them, with keywords typed from the argument declarations: String and ID as Strings, Int as an Integer, enums as their value's name, input objects as their class, nilable unless required. Resolver and mutation classes are followed through resolve, and search_object resolvers through their scope { } block.

None of this is written into the app or into the compiled output. It exists only for the analysis.

What it finds

The test case is howtographql's graphql-ruby example, the Hacker News clone from the How to GraphQL tutorial, unmodified: Rails 6.0, graphql 1.10, six object types, four mutations. check follows all of it:

roundhouse-check: graphql: 6 object type(s), 20 field(s): 20 checked, 0 on types nothing reaches, 0 take arguments

Eighteen of those fields are declared null: false, and every one holds. Three are associations: posted_by (method: :user), and a vote's user and link. A belongs_to can return nil in general. Here each column is NOT NULL with a foreign key, so for a stored row it can't, and check accepts it on exactly those grounds and no others.

To make sure a clean result means something, I added three mistakes to LinkType:

app/graphql/types/link_type.rb:17:7: error[incompatible_binop]: `+` with incompatible operand types: Integer + String
app/graphql/types/link_type.rb:8:5: error[send_dispatch_failed]: no known method `title` on Link
app/graphql/types/link_type.rb:9:5: warning[graphql_nullable_field]: `field :first_voter` is declared `null: false` but can resolve to nil; when it does, the response carries an error and nulls its parent (User?)

The first is an ordinary type error inside a field's method (object.votes.size + "x"), which check now sees because it knows what object is there. The second is graphql-ruby's "Failed to implement", reported at the field line instead of at request time. The third is object.votes.first&.user behind a non-null field.

It also found a real bug in the check itself: record[:name] was typed as the method name rather than what it returns, whenever the reader was declared through RBS or Sorbet. That fix applies to every app, not just GraphQL ones.

What it can't follow yet

That line is the part I care most about. A field this can't follow reports nothing, so a quiet run proves only what was followed. check prints the denominator, and the reasons for the rest.

On a second app, a Rails 8.1 storefront built to measure N+1 strategies, it reads:

roundhouse-check: graphql: 10 object type(s), 38 field(s): 3 checked, 9 on types nothing reaches, 0 take arguments, 26 skipped (computed include 26)

That app picks its resolvers at load time from an environment variable (include Resolvers.for(:product)), so it can compare three strategies. check can't see which module that is, so it doesn't guess: those fields are skipped, with that as the reason. Until this week, Roundhouse's ingest dropped a computed include without a trace. That's fixed too; the compiled Ruby had been losing the mixin.

Not modeled yet, and each would show up on that line with its own reason:

• Connections, which a real API uses everywhere.
• Interfaces and unions.
• hash_key: and dig:.
• Field extensions, and options from a custom field_class. Any option check doesn't recognize makes it skip the field rather than guess what the option does.
• loads: arguments, which are typed as unknown for now.

A request

If you have a Rails app that serves GraphQL through graphql-ruby, I'd like one line from it. The latest release predates this work, so install from the repository (you need a Rust toolchain):

cargo install --locked --git https://github.com/rubys/roundhouse --bin roundhouse
roundhouse check --continue /path/to/your/app

Nothing is booted and nothing in the app changes. Near the end of the output is the graphql: line. Send me that, by opening an issue or however you'd reach me. It holds counts and skip reasons, not your schema; a reason can name an included module or an option, so edit those out if they're private.

I expect most of a real schema to be skipped on a first run, and that's fine. The reasons, by count and across many apps, say what to build next better than anything I could guess from one tutorial app. Any errors or graphql_nullable_field warnings it does report are worth a look too, and if one is wrong, that's a bug I want to hear about.

One app in particular. Tobi has been running roundhouse check on Shopify core and fixing what got in his way; the first of those PRs took a full run from a projected two hours to seven minutes. Shopify's Admin API is GraphQL, so core is about the most demanding test of this there is, and whatever it does that this doesn't model yet will show up on that line by name.

For anyone who'd rather build what's missing themselves, as Tobi did with #171, it's in two files: src/ingest/graphql_ruby.rs, which reads the schema, and src/analyze/graphql.rs, which reports. The check guide describes what's read and why, and tests/graphql_ruby.rs has 19 tests to extend.

What it could become

Two things become possible once every field knows what it returns.

A map of what an anonymous query can reach. In howtographql, UserType exposes email, and nothing in the schema checks who is asking. Reading the schema, { allLinks { postedBy { email } } } should list every poster's address without signing in. I haven't run that against the app, so I'm not claiming it. But "which columns can be reached from the root without passing an authorization check, and by which paths" is a question this analysis can answer for a whole schema, and it would be a good question for the MCP server to answer.

Compiling persisted queries. When the set of query documents is fixed, each one can be specialized: straight-line code, a preload plan worked out ahead of time instead of per request, and JSON written directly. That's the same move Roundhouse already makes for views, and the same differential check applies: send the same query to Rails and to the compiled app and compare the JSON byte for byte. None of that is built.

Both depend on the analysis following real schemas, which is why the first step is the line above.

Roundhouse is open source: dual-licensed MIT / Apache-2.0. The changes described here are commits c2d0664e, 67c39ac5 and 7db5b10e.