Rules as tests: write the rule once, run it on every commit

The pattern in one picture

A test checks a piece of code. You write the code, then you write its test, one by one.

A codebase also has rules: "entities in a test reuse the slice's mocks", "every failure case asserts its error code". Those used to live in a document and in a reviewer's head. Now each rule becomes a test of its own. You write it once, and it runs against every file in its scope, including the ones an agent writes tomorrow.

That is the whole idea. The rest of this post is how it works, what the test looks like, and why it changes the cost of reviewing AI-written code.

Why I needed it

My AI agents commit through a gate of guardians: reviewer subagents, one per area of the codebase, that read the staged diff and grade it.

Guardians are good reviewers, and they are paid on every pass. A one-line fix reopens the round, because any edit in a guardian's area voids its score. Plans that should take minutes took an afternoon.

Most of what a guardian checks is a single judgement over a single file. That kind of question needs a model that reads the file, and only the file.

Jev, in two lines

Jev is TypeSafe's evaluation model. You give it a context and a set of yes/no questions, and it returns the probability of yes for each one, at the pace of a linter, charging only for input.

How it works

A rule is a declarative file that lives in the repo. It says what to look at and what to ask:

export default {
  title: 'The test reuses the slice’s mocks and leaves them untouched',
  on: 'src/**/*.test.ts',
  // The pieces of context Jev receives for each file.
  state: ['file', 'subject', 'mocks'],
  questions: {
    reuses_slice_mocks: {
      // Below this probability the test fails. Set from a real red and a real green.
      barrier: 0.6,
      // Code decides who gets asked: no mocks, no question.
      applies: { mocks: true },
      instructions:
        'Does every entity-shaped object in `file` ' +
        'name an imported mock or spread one?',
    },
  },
}

The engine builds the pieces for each file (the test, the module it tests, the mocks it could use), skips the files outside the question's scope, and sends Jev one call per rule with all its questions.

What the test looks like

It is a plain Vitest suite: a describe per file, a test per rule, a row per question. A red names the file, the rule and the exact question that failed, so whoever reads it (a person or an agent) knows what to fix.

Built once, runs everywhere

The rule's test is built once and runs in the pre-commit hook, on push, in CI/CD, in a sweep of the whole tree, and inside the agent's own loop while it writes code. Each run takes under 500 ms.

Results are stored by content hash. The commit gate only reads that record, so it works offline and instantly, and the same file with the same questions is asked exactly once.

What it saves: time, tokens, rounds

The difference is where the loop closes. With a guardian, each fix goes back through a reasoning agent. With the rule's test, the agent that wrote the code runs the rule, reads the red, fixes the line and runs it again, all before it asks to commit.

Why the architecture works

The key design choice is the split. Rules live in the repo and get reviewed like code; the engine that runs them lives outside and evolves on its own. Adding a question is one entry, adding a scope is a folder of rules, and a false alarm is fixed with one line of scope.

Where the line is

Jev is strong at one judgement over one file. Anything that compares two lists, counts, or depends on order belongs to plain code. The rule of thumb that came out of calibrating: code enumerates, Jev judges one thing at a time.

How to start

1. List what your reviewer checks. Every guardian, every review checklist.
2. Sort each item into code (a grep, an AST check), one judgement over one file, or real reasoning.
3. Write the middle bucket as rules. One yes/no question each, with the mechanism visible in the text.
4. Calibrate each question with a real red and a real green before it can block anything.
5. Wire the suite into the hook and CI, and keep the reviewer agent for the last bucket.

The reviewer agent stays, and it gets called for the questions that need reasoning, with a red row as its brief. Every other rule becomes a test you write once.