Mirza Ishraq
← All writing

By · · 12 min read

Automated Code Review Pipeline with Claude Code

Automated code review doesn’t work by asking the same AI that wrote a change to also grade it. That’s not review, that’s the same reasoning agreeing with itself.

My own pipeline for Claude Code runs eight independent stages before anything merges: a different model, a fresh context with zero memory of why any line was written the way it was, or a real running system that can’t be talked out of its own behavior.

It didn’t come from one clean rewrite. It grew stage by stage, over months, every time a single pass let something through that a second one would have caught.

None of the stages get final authority on their own. Each one produces a verdict, a human reads it, and merge approval is a separate step from everything else passing, on purpose, even on a Tuesday when everything is green.

That human read isn’t a formality either. I read the actual diff myself, not just the review verdicts, and if something doesn’t sit right, I ask Claude to explain it directly before I’ll approve anything.

An explanation that doesn’t hold up under a direct question is its own finding, even when nothing upstream flagged it.

What automated code review with Claude Code actually runs

Here’s the full chain, in order.

Stage What runs What it checks
1. Plan A capable model, reading real source File-and-line-level plan, not a description of intent
2. Adversarial plan review A fresh context, read-only tools Every load-bearing assumption in the plan, against the actual files
3. Implement Whatever model is fast enough Follows the reviewed plan
4. Pre-push code review Independent model, fresh context The diff, cold, no memory of the implementation
5. CI GitHub Actions Test suite, real build, smoke test against the built artifact
6. Holistic PR review Independent model, fresh context Diff + description + CI, checked against each other
7. Functional test A real running instance The behavior, exercised by hand, not just asserted
8. Human approval Me Explicit, every time, never inferred

Stages 2, 4, and 6 are real Claude Code subagents on my machine, not a metaphor. plan-adversary runs before I exit plan mode, on a strong model, and its entire job is to refuse to believe the plan.

Its own rule, verbatim from its prompt: “every factual claim in the plan is a hypothesis until you open the file.” It won’t accept “this is already verified” or “nothing else uses this” without grepping for every consumer itself, and it separates what it confirmed from what it merely couldn’t check, because letting “unverifiable” quietly become “confirmed” is exactly how a stale plan gets implemented.

code-reviewer runs before every commit and push, on a fixed strong model, with a fresh context that has never seen the implementation decisions or the rationalizations that came with them. It’s the same model split I use for the rest of the day’s work too, a capable model reserved for judgment, a faster one for typing, which I broke down properly in Claude Code cost.

pr-reviewer runs once a PR is open, and its job is different in scope: not “is this diff correct,” but “is this the right, complete, honestly described unit of work to merge.” It checks whether the PR description’s “what changed” actually matches the diff, and whether the “tested” claims were actually run or just judged plausible.

There’s a fourth agent, docs-librarian, that isn’t one of the eight stages. It triggers whenever a change touches a service that talks to an external API, and its only job is checking the implementation against the actual current documentation instead of trusting anyone’s memory of how that API works, including the model’s own.

None of this runs through a hook. The instruction is just: run these yourself, every time, unasked, even on the smallest edit. No hooks, no markers, just do it.

That’s a deliberate choice and not a shortcut I haven’t gotten around to. A hook fires on a file pattern; it doesn’t know a one-line change to an auth check deserves the full chain while a copy edit doesn’t. An instruction that’s expected to be followed every time, unprompted, handles that judgment call in a way a static trigger can’t.

It’s the same bet I made writing about Claude Code auto mode: don’t beg the model with a prompt, build something that either runs or doesn’t. Here the “something” is a standing instruction treated as non-negotiable, not a classifier, but the reasoning is the same. Begging doesn’t scale, structure does.

Any check that gates something also has to fail in the right direction. A permission check, a classifier, a review stage that can’t reach a verdict should default to deny, not allow. A check that returns “nothing is restricted” when it can’t read its input is a vulnerability waiting for exactly that failure to happen in production, and it’s worth testing the error path on purpose, not just the happy path where everything loads correctly.

Stage 7 gets the same independence rule, just applied to people instead of models. I’m not the one who functional-tests a change before it ships.

Someone who has never seen the implementation clicks through it instead. They don’t know which code path was touched, so they can’t unconsciously avoid the spot that’s actually fragile, the same reason a fresh-context code review catches things a self-review doesn’t.

Familiarity with your own reasoning is exactly what makes you bad at finding the gaps in it.

What it actually catches

The categories that keep showing up: a regex that matches more, or less, than it was supposed to. A race condition where two requests hit the same check-then-act sequence against Redis or Postgres, so the second one clobbers a state the first one assumed was still true. A cache key that doesn’t capture everything that actually affects the cached value, so a stale or cross-context result gets served back as current.

None of those fail a build. They fail under a timing pattern or an input shape a single read-through doesn’t reproduce, which is exactly why a fresh-context review pass matters more than a more careful first pass would.

The person, or the model, who wrote the regex already believes it’s correct. It takes a reviewer with no stake in that belief to ask what it matches when the input isn’t the one it was tested against.

The technique that actually finds these is boring: grep for every consumer of whatever’s changing, not just the one the plan or the diff talks about.

A plan almost always describes the happy-path caller. The break is nearly always in the variant it didn’t mention, the retry path, the second client, the call site that passes an option nobody else does.

Reading the described change carefully doesn’t surface that. Searching for who else touches the same thing does.

Findings also get graded, not just listed: blocker, should-fix, or question, the same way across every stage.

A blocker is something that can cause wrong results, lost data, or a security hole. A should-fix is a real defect that isn’t a merge risk. A question is something that looks suspicious but can’t be confirmed from the code alone, so it’s phrased as one instead of asserted as fact.

Grading turns a review into a triage list instead of a wall of text, and it stops a genuinely dangerous finding from getting lost between ten nitpicks about naming.

The one rule I don’t break: docs over prompting

I don’t try to control the model through clever prompting. I lean on what’s actually documented and on structured output instead.

That’s the whole reason docs-librarian exists. Prompting an agent to “be careful with the Stripe webhook signature” is a suggestion the model can rationalize past. Pointing it at Stripe’s actual current webhook documentation, and having it check the implementation line against that page, isn’t a suggestion, it’s a fact the model either matches or doesn’t.

The failure mode I’m avoiding isn’t the model being careless. It’s the model being confidently wrong from memory about an API that shipped a breaking change since its training data, and no amount of “please double-check” in a prompt fixes that. Only the current page does.

It checks two layers, in order. First, whatever pinned reference docs already exist for that integration in the repo itself, because that’s the contract the code was actually built against, and a mismatch between the code and that pinned file is a real finding even when the live API would also accept what the code currently does.

Then it goes to the live documentation for anything the pinned reference doesn’t cover, and it says plainly when no pinned reference existed to check against, rather than quietly filling the gap from memory.

When the pinned file and the live docs disagree with each other, it reports both instead of picking one silently, because that disagreement is itself information: either the pin is stale, or the code drifted from a deliberate decision, and only a human reading both knows which.

The same distrust of memory applies to security rules that get computed rather than declared. If “is this restricted” is derived by parsing something else, a naming convention, a list extracted from documentation prose, instead of being an explicit rule, a well-intentioned edit to that other artifact can silently change what’s protected with nothing failing to signal it. So a derived rule like that gets a test that pins the expected result directly, independent of the artifact it’s derived from.

The CI stage: test suite, smoke test, Docker or Ubuntu, and when the minutes run out

Stage 5 isn’t one job. On GitHub Actions it’s a test suite, a real build of the deployable artifact, and at least one smoke test against that built artifact rather than against source running in a dev harness, because a test suite can pass while the packaging step silently drops a file the app needs at runtime.

Whether that build is a Docker build or a plain Ubuntu build depends on the project: a container image if that’s what actually ships, a bare Ubuntu runner if it doesn’t. Same principle either way, only the artifact changes.

GitHub’s own free tier gives a private repo 2,000 Actions minutes a month on GitHub-hosted runners. Past that, I move the project to a self-hosted runner instead of paying per-minute overage. Self-hosted runners are free to use with GitHub Actions, the cost is just whatever machine you point at the job, so it’s a straight swap once a project’s usage pattern makes that machine cheaper than the overage.

I run this same shape of pipeline on Tradion, the AI trading research platform I built, and on other projects I can’t name here. The stages don’t change with the project. What changes is only the build target in stage 5.

The four agents, generic versions

These are stripped-down, generic versions of the four agents, not the verbatim prompts running on my own projects. Swap in your own stack’s file paths, frameworks, and conventions; the structure and the rules are what matter, not the specifics.

---
name: plan-adversary
description: Reviews a proposed plan against the actual codebase before any
  code is written. Never writes code itself.
tools: Read, Grep, Glob, Bash
model: opus
---
Every factual claim in the plan is a hypothesis until you open the file.
Never accept, without checking: a quoted line number or function name,
"this is already the case," "nothing else uses this," or a claim that a
change is safe. Open every file the plan touches. Grep for every consumer
of what's changing, not just the one the plan describes. Record each
assumption as confirmed, false, or unverified, and never let unverified
quietly become confirmed.
Output: WILL BREAK, CLAIMS THAT ARE FALSE, UNVERIFIED, NOT ACCOUNTED FOR,
CORRECT AS WRITTEN. Verdict: safe to implement / safe with changes / do
not implement as written.
---
name: code-reviewer
description: Independently reviews a diff after implementation, before
  push. Fresh context, no memory of implementation decisions.
tools: Read, Grep, Glob, Bash
model: opus
---
You have no memory of why any line was written the way it was. Review the
diff cold. Check correctness, access control, injection risks, race
conditions and check-then-act patterns, and regressions against existing
tests. Every finding needs file:line, the exact code, a concrete failure
scenario, and a fix. Grade each: BLOCKER, SHOULD-FIX, QUESTION. If nothing
is wrong, say so plainly; don't manufacture findings to look thorough.
---
name: docs-librarian
description: Verifies integrations against real, current documentation
  instead of trusting model memory.
tools: Read, Grep, WebFetch, WebSearch
model: sonnet
---
Check any pinned reference docs in the repo first, that's the contract the
code was built against. Then check live documentation for anything not
covered. If the two disagree, report both, don't silently prefer one.
Never rely on memory for version-specific API details. Quote directly from
documentation when flagging an issue, with a direct URL for every one.
---
name: pr-reviewer
description: Reviews an opened pull request holistically, diff,
  description, and CI, before merge. Fresh context.
tools: Read, Grep, Glob, Bash, WebFetch
model: opus
---
Check CI status first; don't review behind a red check. Compare the PR
description against the actual diff: does "what changed" match, were the
"tested" claims actually run or just judged plausible? Review the
implementation for correctness, security, and regressions. Verdict: ready
to approve / ready to approve with nits / request changes / blocked.
Return the verdict, never post, merge, or approve anything yourself.

What it costs, and when I skip it

The full gate, all eight stages, takes about 25 to 30 minutes end to end on a typical change. That’s real time added to every merge, and it’s not free in tokens either.

Running a pipeline this heavy only makes sense if I’m not also leaking tokens everywhere else. I keep Claude Code itself tightly configured for cost outside the pipeline, the model splits, the context hygiene, the settings I actually run are in how tightly I keep Claude Code’s day-to-day cost down. Spending deliberately in one place and carelessly everywhere else isn’t a strategy, it’s just a bigger bill with extra steps.

So I don’t run it on everything. Pure content or docs changes, or a single-property config or style tweak, skip the plan-review and pre-push review stages entirely and go straight to CI plus the holistic review. A small follow-up fixing something a reviewer already flagged gets the same treatment.

Anything touching permissions, shared state, money, or data deletion, or anything else expensive to undo, runs the full eight stages, no shortcuts, no matter how small the diff looks.

The sizing rule isn’t “is the diff small.” It’s “could this plausibly cause a wrong behavior or a security issue.”

A one-line diff that flips an authorization check is small and gets the full chain. A two-hundred-line copy change gets none of it.

Honestly, I don’t always get the sizing right. There are days a three-line fix goes through the full 25 to 30 minutes because I didn’t stop to ask whether it needed to, and it’s hard to tell in the moment whether that’s discipline or just not trusting my own judgment about what counts as small.

I also don’t think this pipeline is the only good way to do this. It’s the shape that’s worked for me so far, not a claim that it’s optimal.

I’ve mistaken process for progress before: 347 commits didn’t save the open source AI project that never found a user, and a review pipeline this elaborate deserves the same question I didn’t ask early enough there, whether it’s actually earning its cost or just feels like it is.

I’m genuinely curious what other people’s version of this looks like, and what a leaner or smarter setup would catch that mine doesn’t. What does your own review pipeline actually look like?