Before You Add an AI Reviewer, Add These Checks

The Claude Code team published how they do code review on October 15. I read the whole thing, and I think their model is right for them. I also think a team of five to twenty engineers that copies the reviewer and the revert-rate guardrail, without the volume and internal users underneath them, will be measuring the wrong thing.

Here is what the post says. Claude writes about 90 percent of Claude Code’s own code. From June to September, merged pull requests more than doubled while the share that had to be reverted fell by more than 40 percent. In Anthropic’s monorepo, changes landing per day went from about 2,500 in May to more than 30,000 in September. Their stated goal is that most PRs merge with at most one human in the loop, who can be the author, with revert and incident rates as the guardrail.

That goal works because of what sits around the reviewer. New behavior usually ships behind a flag to Anthropic employees first. Public releases go to the latest channel before stable. Flags turn off without a deploy, broken builds get reverted, and there is a weekly on-call rotation. The review is one layer in a stack, and the bottom of the stack is a fast way to undo a mistake.

A five-person team can copy the flags and the one-step rollback. What it usually lacks is a company of internal users running each build first, and enough merges for a revert rate to mean much.

What the Anthropic post gets right

I’m going to push back on one part of it, the guardrail, so first here is how much of the post I agree with.

They do not trust the AI reviewer alone. Before a PR exists, Claude is asked to run the tests, the build or a screenshot comparison and show the output. Then required CI blocks the merge: lint, types, tests across platforms, build and packaging. A merge queue re-runs every check against the latest main. Required CI and the merge queue are deterministic, and both block the merge.

They also don’t treat reverts as the whole picture. Incident reviews ask which layer should have caught the issue, and past incidents and reverts become an eval set that the reviewer is scored against every day. Review agents run nightly over existing code. They say plainly that noticing a bad merge still depends on the author and on-call as well as the dashboards, and that they don’t yet have circuit breakers that tighten approval when the numbers move.

CI workflow files always need a person. A bot’s approval never fills a slot that calls for a human. A classifier reading the change sets the risk tier, and only authors on the repo’s trusted list get the lighter paths.

Their own adoption list puts checks and required CI before the reviewer, and ends with flags, staged releases and one-step rollback. My argument is about what the guardrail can see on a small team. My read is that revert and incident rates work for Anthropic because the volume is huge, and because, in their words, “flags, reverts and staged releases undo most mistakes in minutes.” Take away the volume and the rollback and the same metrics tell you much less.

A revert rate measures noticing, not correctness

A revert only happens when someone sees the bug.

A crash gets reverted. A page that returns a 500 gets reverted. A report that now adds up to the wrong total does not, until someone compares it to another number. A migration that writes bad data into a column nobody queries does not. A permissions check that lets the wrong role through works perfectly for everyone who is supposed to be there. These ship, and they stay shipped. They also never show up in the revert rate, so the revert rate can fall while the number of these goes up.

Anthropic’s post doesn’t hide from this. The post says their heavy internal use often surfaces regressions before users do, and elsewhere credits telemetry. A small team doesn’t have that audience, so telemetry and outcome checks are what’s left. If your reviewer misses a silent bug and nothing checks the outcome, nobody finds it until a customer does.

An AI reviewer also has a failure mode that a deterministic check does not: it can be persuaded. Hand it a PR description that says the edge case is handled and it may take your word, because reading the description is part of its job. A schema test has no job like that. It doesn’t read the description. It compares the response to the contract and fails on a renamed field.

I wrote about the description half of that problem in AI PR descriptions nobody reviews. Here I’m after the checks that make the description irrelevant.

What to copy first

Before the reviewer, make the agent prove outcomes with tools that cannot be talked into a pass. I’d build three layers, in this order.

Contract and schema tests. Anything that crosses a boundary gets a test on the shape of the data. Your API’s response, the events your services publish, the columns your jobs write. An agent that renames a field to make its own code tidier breaks the contract, and the test fails in CI whatever the PR says. This layer is a different thing from grading model output with a judge. For that, see testing AI agent outputs with ExUnit. I’m talking about the plain tests around your own interfaces.

Post-deploy outcome checks. This is the layer most small teams skip, and it’s the one that catches the silent bugs. After a deploy, something runs and asserts the outcome you wanted. A known order produces a known invoice total. A smoke job confirms the new row landed in the table and the count is within range. Whatever the check is, it runs against production data and fails loudly when it’s wrong. A failing outcome check is a small team’s substitute for the internal users Anthropic ships to first.

Branch protection and required CI. Main accepts nothing that hasn’t passed the contract tests and the rest of required CI, and nobody, human or agent, can push around it. CI workflow files, auth and billing code need a person on the review. Anthropic does this too; it’s on their list of what to adopt. The review rule and the branch protection setting are in the code review policy post, so I won’t repeat them here.

At EnergyConnect, I’m the founding engineer. We use deterministic tools to verify outputs and outcomes. The deploy pipeline there runs contract and schema tests, post-deploy outcome checks, and branch protection with required CI, which is the same three layers.

These layers also make an AI reviewer much easier to trust later, because it’s no longer the only thing between a bad change and your customers.

The check existed and the plumbing around it lied

I run this site with a content pipeline built from the same idea, and one of its gates failed in September in a way worth showing.

The pipeline is mostly small deterministic gates. A PostToolUse hook in .claude/settings.json blocks any edit that leaves priv/content/site_content.json as invalid JSON, because the app loads that file at compile time and a bad edit breaks the build. A second hook, .claude/hooks/seo-fast-check.py, re-checks the SERP title length, the description byte cap and the merged-out tags every time a post file is written. Neither one reads the post and forms an opinion. They parse and compare.

There is also a citation-verifier agent, and that one exists because of a failure: a draft once cited figures its source didn’t contain. So the pipeline now has a verifier that fetches every external link and marks each claim supported or unsupported. It is an agent, so it sits on the reviewer side of this line.

The better example is the merge gate. On September 30 a merge command looked like this:

gh pr checks N --watch | tail -1 \
  && gh pr merge N

CI had failed. The PR merged anyway. The && tested the exit code of tail, which was 0, and the failing exit code from gh pr checks went into the pipe and disappeared. The commit that shipped was 282d581, a homepage layout change. It went to main with a formatting diff and failed the --check-formatted step, and a push to main deploys this site. The follow-up, 01c28bc, ran mix format on the file, and its message says 282d581 merged with a formatting diff and failed CI.

The check ran, failed and said so, and the pipe threw the answer away. The fix is to test the command itself:

if gh pr checks N --watch; then
  gh pr merge N
fi

No pipe before the test. For any gate you build for agents, break something on purpose after you write it and watch the merge get refused.

Then add the AI reviewer

Once the three layers are in, add the reviewer. It reads intent and architecture, and finds the problems a schema can’t describe. Anthropic’s post has several habits worth taking whole.

Cap the rounds. An agent author and an agent reviewer can keep each other busy indefinitely; one of their CLI PRs went through more than fifty rounds of review and fixes overnight before its author stopped it. Their fix is that after three rounds the reviewer asks the author to step back and reconsider the approach.

Re-review only the new diff after the first few passes, and don’t repeat findings already posted. Without this, the reviewer keeps rediscovering old code. When they ran git blame on what it kept flagging, about 59 percent was code that had been there since its first pass.

Turn incidents into evals. When something breaks, add that PR to a set, confirm where the bug lived, and score the reviewer against the set whenever its prompts, tools or model change. For a small team the set will have five cases for a long time.

Start on one repo and tune its instructions for a few weeks before widening, which is also in their adoption list.

Notice what the incident-as-eval habit depends on, though. You can only add an incident to the set if you detected it. The post-deploy outcome checks are what turn a silent bug into an incident you know about, so they’re also what fills the eval set.

Read this next: what a fractional Chief AI Officer does for dev teams. Setting up these checks before the reviewer is the kind of work I do as a fractional Chief AI Officer.