LLM Evals for My Claude Code Agent (Walkthrough + Prompts)

I ran LLM evals on plan-adversary, the Claude Code subagent that reviews my plans before any code gets written. Eight practice plans, three runs each, once with the agent and once without it. The agent passed 24 of 24 runs. Plain Claude, meaning Claude Code given the same prompt and the same practice repo but without plan-adversary installed, passed 19.

What I didn’t expect is that plain Claude caught every planted bug too. A planted bug is a mistake put into a practice plan on purpose, so the right answer is known before the review starts. The agent earned its place on quieter jobs. It approved good plans without crying wolf, and it said “I can’t check this” instead of guessing.

I asked Claude Code to rate the agent out of 10. The original scored about 6, and the rewrite about 9. A model grading a prompt it just rewrote is homework grading itself, so I had it test the agent too.

The eval tool priced all the test runs at $34.39. That’s the API list price, not what I pay. I’m on Claude Max at $200 a month, so I never see a per-token bill.

Below, I cover what changed in the agent and why, how the test kit works, the two times my tests were wrong, the full agent to copy, and what these evals can’t tell you.

What is plan-adversary, and why make it global?

plan-adversary is stage 2 of my automated code review pipeline. Before Claude writes code, it reads the plan against the real files and tries to prove the plan wrong. Its core rule is that every factual claim in a plan is a guess until it opens the file that settles it. Before I built my Claude Code and Codex review setup, it found six real problems in that plan.

It started as a project agent inside Tradion, so my other repos never got it. Claude Code looks for subagents in several places, and the subagent docs rank them. A project’s .claude/agents/ folder beats your personal ~/.claude/agents/ folder.

So I moved a general version to ~/.claude/agents/plan-adversary.md. Every repo gets it now, and a repo with its own copy still wins.

An agent nobody calls does nothing. So I also added a rule to my global CLAUDE.md, the instructions file Claude Code reads in every project.

## Plans
- **Before `ExitPlanMode`, in every repo** → run the `plan-adversary` agent on the plan. This is the one standing exception to "only spawn subagents for large work".
  - It can't see this conversation. Brief it with: the plan file path, the repo path, my goal in one sentence, decisions already made (so it doesn't relitigate them), and any facts it can't find in the repo.
  - Fix every Blocker and High it reports, list what's still UNVERIFIED in the plan, and put its verdict in one line of the plan. Its OPTIONAL items are suggestions: don't chase them unless they're cheap and clearly right.
- **The repo wins.** If a repo's own `CLAUDE.md` sets a different review pipeline, follow that instead. If a repo has its own `.claude/agents/plan-adversary.md`, Claude Code uses it over this global one automatically.

Here is how the pieces fit together on a normal change.

Then I asked Claude Code for four things. Keep the model on Opus. Read Anthropic’s docs, engineering posts and cookbooks on prompts and reviewers. Rate the agent out of 10 and keep working until it reaches 9. And run Claude Code’s eval command to test the prompt.

In Claude Code cost I wrote that Opus earns its price on the planning step, when the question is what to build. Reviewing a plan happens at that same step.

What are LLM evals?

An LLM eval is a test for software that answers a little differently every time. A normal unit test checks one input against one exact output. An LLM eval runs a realistic task several times and scores each answer with a check.

Four parts do the work.

• A case. A realistic prompt, plus any setup it needs, like a small repo.
• Graders. Pass or fail checks on the answer. Some are plain code, like “does the reply mention digest.js?” Others send the answer and a rubric to a second model, which votes pass or fail.
• Repeats. One run tells you little. claude plugin eval runs each case three times by default.
• A baseline. The same case without the thing you’re testing. For a plugin, the tool runs it by default, so you see what the plugin adds instead of a bare score. This post calls that arm plain Claude.

Anthropic’s guide to defining success criteria and building evaluations says good criteria are specific, measurable, achievable and relevant. Mine were built around planted problems, so here is one.

The missed-caller practice plan makes this claim.

`getNote` is only called from the GET route, so nothing else needs to change.

And this line in the practice repo proves it wrong. The nightly digest job calls getNote too.

// src/jobs/digest.js, line 13
    const note = await getNote(row.id);

A review passes if it names the digest job. A review that approves the plan fails.

Before any run, Claude Code wrote this “Done when” section into its plan, and I approved it.

## Done when
- The rubric is at least 9 (I'll show the table again).
- On the final round:
  - every seeded case is caught in all 3 runs
  - `sound-plan` and `fp-trap` are approved in all 3 runs, with no false blockers
  - the average score is at least 0.9
  - the agent beats plain Claude (Δ > 0)
- If it plateaus after 3 iterations, I stop and report honestly.

In plain words, a seeded case is one with a planted problem, sound-plan and fp-trap are the two good plans in the test kit, and the rubric is the 1 to 10 rating below. Δ is the agent’s score minus plain Claude’s.

The same guide ranks three kinds of graders. Code-based grading is the fastest and most reliable. Human grading is the most flexible and highest quality, but slow and expensive, and the guide says to avoid it if possible. For model-based grading, it says to “test to ensure reliability first then scale.”

I skipped the testing part at first. More on that below.

How did the agent score out of 10?

Claude Code read the sources and scored the original agent on ten qualities they keep coming back to. The main sources were Claude Code’s best practices, which recommend an adversarial review step in a fresh context, the subagent docs, Anthropic’s prompting best practices, and the engineering post Demystifying evals for AI agents.

The original scored 5.8. The rewrite scores 9.15. That rating is a judgment call against the sources, not a measurement, which is why the rest of this post exists.

What changed in the prompt, and why?

There were ten changes, each tied to a gap in the table.

1. A definition of a finding. A Blocker breaks correctness, data, security, a stated requirement or a written repo rule. A High is likely to cause a real problem but hinges on something unconfirmed. A Medium is a real gap that won’t break anything on day one. Style, naming, linter catches and hardening wish-lists are out. A reviewer told to find problems will find some, even in a good plan, unless you tell it what counts.
2. Proof, or it moves. Every Blocker and High cites file:line and quotes the code. Without proof, it goes under UNVERIFIED.
3. Refute before reporting. Before reporting a Blocker, the agent looks for the guard, test or caller that would make it wrong. Only code it actually opened counts.
4. A sound plan is a normal outcome. Approving a good plan, with a list of what was checked, counts as a result.
5. Better inputs. The agent starts with no memory of the conversation. It now expects the plan, the repo path, the user’s goal and decisions already made, and it says so when it has to infer the goal.
6. A boundary on inherited rules. Subagents read your CLAUDE.md files too. Mine says “When you find a bug, fix it.” That’s right for the main session and wrong for a reviewer. The agent now treats repo rules as review criteria and leaves the actions to the main session.
7. Calmer wording. Anthropic’s prompting guide says newer models follow the system prompt more closely, so aggressive words like “CRITICAL” can make them overtrigger. The rewrite explains the reason behind each rule and adds three short examples, showing a real blocker, a nitpick to drop and a clean approval. Yes, you, with “MUST” on every other line of your prompt.
8. Effort that fits the plan. A one-line change gets the touched code and every caller. A multi-file plan gets all seven steps. The agent stops once every claim and every consumer is classified. The frontmatter sets effort: high.
9. A report the main session can act on. Verdict first, a severity tag on every finding, about 1,000 words, a MINIMAL CORRECTION section and one closing “if I could change one thing” line. Version 3 also caps the pre-existing problems at three lines and the “checked and sound” list at the three to six checks that matter most.
10. A shorter description. The main session reads the description to decide when to call the agent. It went from about 1,800 characters of transcript-style examples to 573 characters of plain sentences.

How do you make a subagent actually read-only?

The original said “read-only” in its prompt, and nothing enforced it. Two things could write. Bash can run any command. And the memory: user setting, which gives the agent a notes folder at ~/.claude/agent-memory/plan-adversary/, automatically turns on Read, Write and Edit, per the subagent docs.

A rule in a prompt is a request. A hook is a lock. Hooks are scripts Claude Code runs at fixed moments. A PreToolUse hook runs before every tool call and can deny it. A subagent can declare hooks in its own frontmatter, and they only run while that subagent runs.

The guard denies Write and Edit everywhere except the agent’s own notes folder. For Bash, it blocks output redirects, anything that deletes, moves or installs, git commands that change the repo, package scripts, migrations and cloud tools. Read-only commands like git log, ls and grep pass. If it can’t read its input, it denies.

I tested it two ways. A test file feeds it 81 sample tool calls, and all 81 come out right. Then one live session asked the real agent to touch a file and to Write another. Both were blocked, git log went through, and no new files appeared.

The evals below can’t test this lock. The subagent docs say plugin subagents ignore the hooks field, and the eval tool loads the agent as a plugin. That’s why the lock got its own test.

Here is the full guard script, ~/.claude/hooks/plan-adversary-guard.mjs.

#!/usr/bin/env node
// PreToolUse guard for the plan-adversary agent (wired in ~/.claude/agents/plan-adversary.md frontmatter).
// The agent reviews plans and must not change anything, so this denies shell commands that write or have side
// effects, and Write/Edit anywhere except its own memory folder. Read-only commands get no decision and run as normal.
// Tests: ~/.claude/agent-evals/plan-adversary/guard.test.mjs
import { readFileSync } from 'node:fs';
import { homedir } from 'node:os';
import path from 'node:path';

const MEMORY_DIR = path.join(homedir(), '.claude', 'agent-memory', 'plan-adversary') + path.sep;

// git subcommands that only read.
const GIT_READ = new Set([
  'status', 'log', 'show', 'diff', 'blame', 'annotate', 'rev-parse', 'ls-files', 'ls-tree', 'grep', 'describe',
  'shortlog', 'cat-file', 'merge-base', 'for-each-ref', 'rev-list', 'whatchanged', 'name-rev', 'check-ignore',
  'count-objects', 'help', 'version',
]);
// Programs that write, delete, install, deploy, or run arbitrary code. Wrappers that hide the real command
// (sh -c, eval, xargs) are here too, because the guard can't see inside them.
const BLOCKED = new Set([
  'rm', 'rmdir', 'mv', 'cp', 'ln', 'touch', 'mkdir', 'chmod', 'chown', 'truncate', 'dd', 'tee', 'install', 'rsync',
  'scp', 'sudo', 'kill', 'pkill', 'killall', 'launchctl', 'crontab', 'open', 'osascript', 'defaults', 'security',
  'sh', 'bash', 'zsh', 'fish', 'eval', 'source', '.', 'xargs', 'make', 'npx', 'pnpm', 'yarn', 'bun', 'deno',
  'brew', 'apt', 'apt-get', 'pip', 'pip3', 'poetry', 'uv', 'cargo', 'gem', 'bundle', 'composer', 'wget',
  'docker', 'kubectl', 'helm', 'terraform', 'psql', 'mysql', 'mongosh', 'mongo', 'redis-cli', 'sqlite3',
  'vercel', 'netlify', 'railway', 'fly', 'flyctl', 'heroku', 'aws', 'gcloud', 'az', 'stripe', 'supabase',
  'prisma', 'knex', 'sequelize', 'alembic', 'rails', 'rake',
]);
const NPM_READ = new Set(['ls', 'list', 'view', 'info', 'explain', 'why', 'outdated', '-v', '--version']);
// Prefixes that just run the next word as the command.
const PASSTHROUGH = new Set(['env', 'time', 'nice', 'nohup', 'command', 'exec', 'builtin']);

const deny = (reason) =>
  process.stdout.write(JSON.stringify({
    hookSpecificOutput: {
      hookEventName: 'PreToolUse',
      permissionDecision: 'deny',
      permissionDecisionReason: `plan-adversary is read-only: ${reason}. Read the code instead, or list the step under VERIFICATION.`,
    },
  }));

let input;
try {
  input = JSON.parse(readFileSync(0, 'utf8'));
} catch {
  deny('the guard could not parse its input, so it is failing closed');
  process.exit(0);
}
const reason = check(input.tool_name, input.tool_input ?? {});
if (reason) deny(reason);

function check(tool, toolInput) {
  if (['Write', 'Edit', 'MultiEdit', 'NotebookEdit'].includes(tool)) {
    const raw = toolInput.file_path ?? toolInput.notebook_path ?? '';
    const target = path.resolve(raw.replace(/^~(?=\/)/, homedir()));
    return target.startsWith(MEMORY_DIR) ? null : `${tool} is only allowed inside ${MEMORY_DIR}`;
  }
  if (tool === 'Bash') return checkCommand(String(toolInput.command ?? ''));
  return null;
}

function checkCommand(command) {
  // Quoted text can't run anything by itself, and stripping it keeps '>' or ';' inside strings from tripping the checks.
  let s = command.replace(/'[^']*'/g, "''").replace(/"(?:\\.|[^"\\])*"/g, '""');
  // Redirects that write nothing: fd duplication (2>&1) and /dev/null.
  s = s.replace(/\d*>&\d+/g, ' ').replace(/&?\d*>>?\s*\/dev\/null/g, ' ');
  if (/>/.test(s)) return 'output redirection writes a file';
  // Split into simple commands, including the ones inside $(...) and backticks.
  for (const segment of s.split(/\|\||&&|[;|&\n`()]|\$\(/)) {
    const reasonForSegment = checkSimpleCommand(segment.trim().split(/\s+/).filter(Boolean));
    if (reasonForSegment) return reasonForSegment;
  }
  return null;
}

function checkSimpleCommand(words) {
  while (words.length && (/^[A-Za-z_][A-Za-z0-9_]*=/.test(words[0]) || PASSTHROUGH.has(words[0]))) words.shift();
  if (words[0] === 'timeout') words.splice(0, 2);
  if (!words.length) return null;
  const [cmd, ...args] = words;
  const name = path.basename(cmd);
  const has = (...flags) => args.some((a) => flags.includes(a) || flags.some((f) => f.endsWith('=') && a.startsWith(f)));

  if (BLOCKED.has(name)) return `\`${name}\` can change files or systems`;
  if (name === 'git') return checkGit(args);
  if (name === 'npm') return NPM_READ.has(args[0]) ? null : `\`npm ${args[0] ?? ''}\` runs scripts or changes packages`;
  if (name === 'node' || name === 'python' || name === 'python3') {
    const inline = ['-e', '--eval', '-p', '--print', '-c', '--version', '-v', '-V'];
    return args.length && inline.includes(args[0]) ? null : `only inline \`${name}\` probes (-e / -c) are allowed`;
  }
  if (name === 'sed' && args.some((a) => /^-[a-zA-Z]*i/.test(a) || a.startsWith('--in-place'))) return '`sed -i` edits files';
  if (name === 'perl' && args.some((a) => /^-[a-zA-Z]*i/.test(a))) return '`perl -i` edits files';
  if (name === 'awk' || name === 'gawk') {
    if (args.some((a, i) => a === 'inplace' && args[i - 1] === '-i')) return '`awk -i inplace` edits files';
  }
  if (name === 'find' && has('-delete', '-exec', '-execdir', '-ok', '-okdir', '-fprint', '-fprintf', '-fls')) {
    return '`find` with -delete/-exec can change files';
  }
  if (name === 'curl') {
    if (has('-o', '--output', '-O', '--remote-name', '-T', '--upload-file')) return '`curl` writing or uploading files';
    if (has('-d', '--data', '--data-raw', '--data-binary', '--data-urlencode', '-F', '--form', '--json')) return '`curl` sending data';
    const i = args.findIndex((a) => a === '-X' || a === '--request');
    if (i !== -1 && !/^(GET|HEAD|OPTIONS)$/i.test(args[i + 1] ?? '')) return '`curl` with a write method';
  }
  if (name === 'gh') {
    if (args[0] === 'api') {
      return has('-X', '--method', '-f', '-F', '--field', '--raw-field', '--input') ? '`gh api` writes' : null;
    }
    return ['view', 'list', 'diff', 'checks', 'status'].includes(args[1]) ? null : `\`gh ${args.slice(0, 2).join(' ')}\` can change GitHub`;
  }
  return null;
}

function checkGit(args) {
  const a = [...args];
  // Global options before the subcommand: -C <dir>, -c key=value, --no-pager, --git-dir=..., and so on.
  while (a.length && a[0].startsWith('-')) {
    const opt = a.shift();
    if (opt === '-C' || opt === '-c') a.shift();
  }
  const [sub, ...rest] = a;
  if (!sub || GIT_READ.has(sub)) return null;
  const flagsOnly = rest.every((x) => x.startsWith('-'));
  const ok = {
    branch: flagsOnly && !rest.some((x) => /^-(d|D|m|M|c|C|u)$|^--(delete|move|copy|set-upstream|unset-upstream|edit-description)/.test(x)),
    tag: rest.length === 0 || rest.some((x) => x === '-l' || x === '--list'),
    stash: ['list', 'show'].includes(rest[0]),
    remote: rest.length === 0 || rest[0] === '-v' || ['show', 'get-url'].includes(rest[0]),
    config: rest.some((x) => /^--(get|get-all|get-regexp|list)$|^-l$/.test(x)),
    worktree: rest[0] === 'list',
    notes: ['show', 'list'].includes(rest[0]),
    reflog: !rest.length || rest[0] === 'show' || rest[0].startsWith('-'),
  }[sub];
  return ok ? null : `\`git ${sub}\` changes the repository`;
}

The full plan-adversary agent

Save this as ~/.claude/agents/plan-adversary.md next to the guard above. If Claude can’t find it, restart Claude Code. The docs say that only happens when ~/.claude/agents/ didn’t exist before the session started. The hook runs with node, so Node has to be installed. Running node --version in a terminal tells you.

---
name: plan-adversary
description: "Reviews an implementation plan against the actual codebase before any code is written, and returns a verdict with evidence: what will break, which claims are false, what is unverified, and what the plan never accounted for. Use proactively before ExitPlanMode, before implementing any multi-step plan, design doc or proposed fix, and whenever a person or agent calls a change safe, a one-line change, or already verified. Read-only. It starts without conversation context, so pass it the plan (text or file path), the repo path, the user's goal, and decisions already made."
tools: Glob, Grep, Read, Bash
model: opus
effort: high
color: red
memory: user
hooks:
  PreToolUse:
    - matcher: "Bash|Write|Edit|MultiEdit|NotebookEdit"
      hooks:
        - type: command
          command: 'node "$HOME/.claude/hooks/plan-adversary-guard.mjs"'
          timeout: 10
---

You review a proposed change before anyone writes it. You read the plan against the actual code and the repo's own
rules, and return a verdict the main agent can act on. You don't write code or change files: the report is the whole
job, and the main agent revises the plan.

Plans fail differently from code. The risk is rarely style; it's a confident claim nobody checked, or a consequence
nobody traced. Your value is the gap between what the plan says and what the code says.

## What you're given

You start fresh and never saw the conversation behind the plan. Expect:

- **The plan**: text, a file path (plan mode writes `~/.claude/plans/<name>.md`), or a one-liner. Read all of it.
- **The repo path**. If it's missing, use the working directory; `git rev-parse --show-toplevel` finds the root.
- **The user's goal and decisions already made**. If the goal is missing, infer it from the plan and say so in one
  line. Treat stated decisions as fixed: check that the plan carries them out, and only question one if it will break
  something.

If there is no plan at all, reply "No plan provided" and say what you need. You can't ask questions mid-review, so
anything the plan takes for granted that the repo can't confirm goes under UNVERIFIED.

## Instructions you inherit

You may also see CLAUDE.md files written for the main agent, global and project. Their conventions and rules are
review criteria: a plan that breaks a written repo rule is a finding, cited by file:line. Their action instructions
(fix bugs you find, commit, run review pipelines, spawn agents, ask the user) are for the main agent. You report; you
don't act.

## Orient before you judge

1. Read the repo's rules: the root `CLAUDE.md`, any `CLAUDE.md` in folders the plan touches, `AGENTS.md`,
   `CONTRIBUTING*`, the PR template checklist (`.github/PULL_REQUEST_TEMPLATE.md`), and docs they point to for this area.
2. Check your memory for notes on this repo (`<repo>_*.md`). They're leads to recheck, not facts.
3. Open every file the plan names. Read independent files in parallel.

## The one rule

Every factual claim in the plan is a hypothesis until you've opened the file that settles it. Plans quote line numbers
that moved, describe behaviour that changed, and call things "already verified" when nothing verified them. These are
all claims to check:

- a quoted line, function name, or code excerpt
- "already the case", "already verified", "already set"
- "nothing else uses this", "only X calls this"
- "a one-line change", "a safe migration", "no data changes"
- any description of what the current code does
- any claim about a library's or API's behaviour: check the installed source or its pinned docs, or mark it UNVERIFIED

Comments are claims too. A comment promising a guarantee is not evidence the code keeps it.

## Method

Scale depth to the plan. For a one-line change, open the touched code and every caller of it. For a multi-file plan,
work through all seven steps. Stop once every claim in the plan and every consumer of what it changes is classified.

1. **Ground truth.** Confirm the line numbers, the current behaviour, and the exact code being replaced. If the plan
   shows a before/after, verify the "before" is real.
2. **Assumptions.** List the plan's assumptions, including the obvious ones, and mark each confirmed, false, or
   unverifiable from here. Keep "unverifiable" separate from "confirmed", and say what would settle it (a production env
   var, a query against live data).
3. **Blast radius.** This is where most real bugs are. For each thing that changes (symbol, route, env var, column,
   config key, file path, job, event), find every consumer with Grep across the repo, excluding node_modules, dist,
   build, vendor, .venv and .git. Include tests, scripts, CI and docs. For each consumer, ask whether the change still
   holds. Look hardest at consumers that use it differently from the plan's example: different arguments, a separate
   process (worker, cron, webhook, CLI) that skips the request middleware, a retry or fallback path, a second client.
4. **New mechanisms.** For each flag, lock, guard or state the plan adds: is it scoped to the whole unit of work it
   protects, or does it reset each iteration? What happens on the failure path: a clean stop, or a silent fall-through
   into a half-written state? Does one error message cover causes that need different responses?
5. **Scope.** Read the out-of-scope list as claims: is anything there load-bearing for what's in scope? Check the
   plan's own quality too: files and interfaces named, scope stated, an end-to-end check included. A vague step
   ("we'll sort out the migration later") is a finding when the rest of the plan depends on it.
6. **Verification.** For each test or check in the plan, ask: if the change were broken the way I suspect, would this
   fail? A check that passes while the bug is present is worse than none, because it manufactures confidence. Name what
   is untested: the second consumer, the failure path, the valid input next to the invalid one.
7. **Rollback.** "Revert the commit" doesn't undo an applied migration, written data, a changed queue payload, a sent
   email or webhook, or anything already consumed downstream. Say so when it applies.

Use Bash for read-only inspection (`git log`, `git show`, `git diff`, `git blame`, `ls`) and for inline probes that
create nothing (`node -e`, `python3 -c`) when you need to see how a library actually behaves. Don't run tests, builds,
migrations or installs: they can write files and touch databases, so list them under VERIFICATION GAPS instead. A guard
hook blocks commands that write; if it blocks one, get the answer by reading.

## What counts as a finding

- **Blocker**: breaks correctness, loses or corrupts data, opens a security hole, misses a stated requirement, or breaks
  a written repo rule. Must be fixed before implementing.
- **High**: very likely to cause a real problem, but it hinges on something you couldn't fully confirm. Should be fixed.
- **Medium**: a real gap that won't break anything on day one, such as a missing test for the second consumer.

Leave out style and naming preferences, anything a linter or type checker will catch, failures that need unusual input
with no sign it ever occurs, hardening wish-lists, and redesigns of an approach that works. If the plan picks an
approach you wouldn't, one line under OPTIONAL is enough. Problems that already exist, and that the plan neither causes
nor makes worse, get one line under PRE-EXISTING.

Every Blocker and High cites file:line and quotes the line or a short excerpt that proves it. If you can't point to the
code, it isn't a Blocker: move it to UNVERIFIED.

Before reporting a Blocker, try to refute it. Look for what would make it wrong: a guard, test, handler or caller you
haven't read yet. Only a mitigation you actually opened counts; "the framework probably handles this" doesn't. If you
find one, drop or downgrade the finding.

A sound plan is a normal outcome. A reviewer asked to find problems tends to find some even when the work is fine, and
the main agent will chase whatever you report. Approving a good plan, with a specific list of what you checked, is as
valuable as stopping a bad one.

## Common traps

Check each one the plan comes near:

- a comment claiming a guarantee the code doesn't keep
- check-then-act on a shared store (database, cache, queue, file); if the repo already has an atomic pattern (a
  conditional update, a transaction, a lock), a new counter or state change should copy it
- a `catch` for a database constraint error where the constraint doesn't exist in the schema
- charging, sending or committing before the work is safe to interrupt
- a new route or handler missing the auth or ownership checks its siblings have
- origin, cookie and CORS reasoning that ignores whether the frontend and API share an origin in production
- separate processes (workers, cron, serverless functions) that don't share the API's middleware
- a check that passes vacuously: a script that logs "skipped" and exits 0, a test that asserts nothing useful
- config, env vars and secrets that exist locally but not in the deployed environment, or the reverse

The repo's own traps come from its CLAUDE.md, its PR template checklist, and your memory notes for it.

## Output

Your final message is the whole deliverable; the main agent sees nothing else. Lead with the verdict, keep the report
under about 1,000 words, and leave out empty sections, except WILL BREAK, where you write "Nothing found."

- `safe to implement`: no Blocker or High items.
- `safe with changes`: Blocker or High items, each fixable with a small change you can state.
- `do not implement as written`: a core claim is false, or the fix needs a different approach.

<output_format>
VERDICT: <safe to implement | safe with changes | do not implement as written>: <one sentence why>
GOAL: <as given, or "inferred: ...">

WILL BREAK
  [Blocker|High] <what breaks and who notices>: <file:line> `<quoted code>`

CLAIMS THAT ARE FALSE
  [Blocker|High|Medium] Plan says <X>; the code says <Y>: <file:line> `<quoted code>`

UNVERIFIED
  <assumption>: <why it can't be checked from here>; <what would check it>

NOT ACCOUNTED FOR
  [severity] <consumer, path or case the plan never mentions>: <file:line>

VERIFICATION GAPS
  [severity] <a check that would pass while the bug is present, or what is untested>

OPTIONAL
  <non-blocking suggestion, one line each>

PRE-EXISTING
  <a problem the plan doesn't cause or worsen, one line each, at most 3>

CHECKED AND SOUND
  <the 3-6 checks that matter most, specifically, with file:line; not every file you read>

MINIMAL CORRECTION
  <the smallest change to the plan that clears every Blocker and High: code or exact plan text.
   Omit when the verdict is safe to implement.>

IF I COULD CHANGE ONE THING: <one line; omit when nothing needs changing>
</output_format>

<examples>
<example>
A Blocker, with the evidence that makes it one:

  [Blocker] Refunds from the Stripe webhook stop applying: the plan makes `reason` a required second argument, and the
  webhook still calls applyCredit with one, so it throws before writing: src/webhooks/stripe.ts:88
  `await applyCredit(event.data.object.customer)`
</example>
<example>
A comment to leave out, because nothing breaks:

  "The plan names the helper `fmtDate`; `formatDate` would read better." Naming preference. Not reported.
</example>
<example>
A clean approval:

  VERDICT: safe to implement: the new column is nullable, both writers go through the function the plan updates, and
  the planned test covers the null case.
  GOAL: inferred: let users archive notes without deleting them.

  WILL BREAK
    Nothing found.

  CHECKED AND SOUND
    Migration adds `archived_at` as nullable, so no backfill: db/migrations/014_archive.sql:3
    Both writers (api/notes.ts:40, jobs/import.ts:112) call saveNote, which the plan changes.
    The list query the plan edits is the only reader of the new column (grep: 1 match, api/notes.ts:22).
</example>
</examples>

## Memory

Your memory folder is shared across every repo, so file notes by repo.

- After a review, save only verified, non-obvious traps that a future plan in the same repo is likely to walk into:
  things that shipped broken, mechanisms that work differently from how they look, checks that pass vacuously.
- One file per repo topic, named `<repo>_<topic>.md`, where `<repo>` is the repo folder name. Start the note with the
  date you verified it and "Recheck against the code before reusing."
- Don't save plan-specific details, anything the repo's CLAUDE.md already says, or anything you didn't verify.
- Keep `MEMORY.md` as an index: one line per note, grouped by repo.
- If memory writes are unavailable, skip this step; the review is what matters.

How do you use plan-adversary day to day?

Let it run by itself. Open Claude Code in a repo and press Shift+Tab until the status bar shows ⏸ plan mode on, as the best practices describe. Ask for your change. Before Claude leaves plan mode, the CLAUDE.md rule makes it call plan-adversary and fix every Blocker and High it reports. You don’t type anything extra.

Call it yourself. In the Claude Code prompt, type @agent-plan-adversary, or name it in a sentence like “Use the plan-adversary subagent to review my plan.” The subagent docs list both. The agent can’t see your conversation, so tell it the plan, the goal and anything already decided.

Here’s an example briefing, built from the edit-a-note plan in the test kit.

@agent-plan-adversary Review the plan in PLAN.md against this repo.
Goal: let a note's owner edit its title and body.
Already decided: another user's note returns 404, same as GET. Edit history is out of scope.

What comes back. This is the start of a real report from the missed-caller test. I trimmed the eval’s temporary folder from the file path, and “…” marks a skipped line.

VERDICT: safe with changes. The claim that "`getNote` is only called from the GET route" is false. The nightly digest job also calls it with only an id. After this change the digest would quietly stop emailing anyone, and `npm test` would still pass.
…
WILL BREAK
  [Blocker] **The nightly digest email stops going out, with no error.** `src/jobs/digest.js` calls `getNote` with one argument:
  - src/jobs/digest.js:13 `const note = await getNote(row.id);`

The verdict comes first, the Blocker names who notices (nobody, which is the point), and the proof is the exact line of code.

What a block looks like. When the agent tries to change anything, the guard stops it. These are the two lines the agent saw in the live test, with my home folder shortened to ~.

PreToolUse:Bash hook error: plan-adversary is read-only: `touch` can change files or systems. Read the code instead, or list the step under VERIFICATION.
PreToolUse:Write hook error: plan-adversary is read-only: Write is only allowed inside ~/.claude/agent-memory/plan-adversary/. Read the code instead, or list the step under VERIFICATION.

How does the test kit work?

claude plugin eval tests plugins, not loose agent files. So the suite is a tiny plugin wrapper in ~/.claude/agent-evals/plan-adversary/. Before every run, run.sh copies the live agent into it, so the suite always tests what is installed.

~/.claude/agent-evals/plan-adversary/
├── .claude-plugin/plugin.json   # named plan-adversary-eval
├── agents/plan-adversary.md     # copied from ~/.claude/agents/ by run.sh
├── run.sh
├── evals/
│   ├── _fixture/                # the fake notes app every case starts from
│   ├── missed-consumer/         # one folder per case
│   │   ├── case.yaml
│   │   ├── fixture.sh
│   │   ├── PLAN.md
│   │   ├── prompt.md
│   │   └── graders/
│   └── ...                      # 7 more cases
└── evals/results/               # one JSON file per round

A fake app with traps. Every case starts from the same small notes API. Its CLAUDE.md has four rules. Every /api route goes through requireAuth, every query filters by user_id, every migration has a down file, and jobs run in a separate cron process. The traps below were planted on purpose. This is what each one looks like in the code.

The auth middleware checks the signature and never checks whether the token has expired.

// src/middleware/auth.js, lines 7-11
  if (!token || !verifySignature(token)) {
    return res.status(401).json({ error: 'unauthorized' });
  }
  req.user = decodeToken(token);
  next();

Tokens store their expiry time in seconds, while JavaScript’s Date.now() counts milliseconds.

// src/lib/tokens.js, lines 5-7
// Tokens are base64url(JSON payload) + "." + HMAC. `exp` is a Unix timestamp in SECONDS, like a JWT.
export function issueToken(user, ttlSeconds = 3600) {
  const payload = { id: user.id, email: user.email, exp: Math.floor(Date.now() / 1000) + ttlSeconds };

getNote looks a note up by id only, and a comment says the routes check ownership. The digest job’s call, shown earlier, skips the routes entirely.

// src/services/notes.js, lines 3-5
// Looks a note up by id only. Callers that serve a user check ownership themselves (see routes/notes.js).
export async function getNote(id) {
  const { rows } = await db.query('SELECT * FROM notes WHERE id = $1', [id]);

The seed data has two notes titled “Groceries” for the same user, so a migration that adds a unique (user, title) rule fails on any database loaded with this seed.

-- db/seed.sql, lines 4-5
  (1, 'Groceries', 'milk, eggs'),
  (1, 'Groceries', 'coffee'),

And .env.example, shown here in full, has no APP_URL.

# .env.example, lines 1-3
DATABASE_URL=postgres://localhost:5432/notes
JWT_SECRET=change-me
SMTP_URL=smtp://localhost:2525

Eight practice plans. Each case’s setup script copies the app, commits it to git and drops in a PLAN.md. Each plan has one answer two reviewers would agree on.

The last two are held out. Nothing in the prompt was tuned while looking at them. It’s the same idea as the test set in overfitting vs underfitting, where a score only means something on cases you didn’t tune against. Lance Martin’s article Automating eval design and hillclimbing with Claude describes the same split, with “a test set that is never seen.”

Here is one full case, the missed caller. This is the setup script.

#!/usr/bin/env bash
# Builds the workspace: the shared notes-api fixture committed to git, plus this case's PLAN.md (left uncommitted).
set -euo pipefail
here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
cp -R "$here/../_fixture/." .
git init -q
git add -A
git -c user.name=eval -c user.email=eval@example.com commit -qm "notes-api"
cp "$here/PLAN.md" PLAN.md

This is the plan it reviews.

# Plan: scope getNote to the owner

## Why
`getNote(id)` looks notes up by id only. Ownership is checked in the route, which is easy to forget in new routes. Move the owner check into the service.

## Changes
- `src/services/notes.js`: change `getNote(id)` to `getNote(id, userId)` with
  `SELECT * FROM notes WHERE id = $1 AND user_id = $2`.
- `src/routes/notes.js`: `GET /api/notes/:id` calls `getNote(req.params.id, req.user.id)` and drops its own
  `note.user_id !== req.user.id` check (the query now does it).

`getNote` is only called from the GET route, so nothing else needs to change.

## Tests
Update `tests/notes.test.js` to pass a userId to `getNote`.

## Verification
`npm test` passes.

This is the prompt, the same for every case. It asks for the report word for word, so the graders see what the agent wrote.

---
tags: [dev]
max_turns: 40
timeout_seconds: 900
allowed_tools: [Read, Glob, Grep, Agent]
---

I'm about to implement the plan in PLAN.md in this repo. Before I do, check it against the code: will it break anything? Give me the full review word for word: if a review agent wrote it, paste its report as-is.

Graders. The missed-caller case has three kinds. A judge rubric checks the substance.

---
type: llm
weight: 2
---

Context for the judge: the plan changes getNote(id) to getNote(id, userId) and claims only the GET route calls it. In the repo, src/jobs/digest.js (the nightly digest, run from the separate cron process) also calls getNote(row.id) with one argument, so after the change it would get no rows and silently send empty or missing digests.

PASS if the review identifies that the digest job (src/jobs/digest.js) also calls getNote and would break or silently lose notes after the change, and it does not approve the plan as safe without changes.
FAIL if the review never mentions the digest job's call to getNote, or approves the plan as written.

A judge model votes three times, and two passes win. My judge was Sonnet, while the agent runs on Opus. Lance’s article says plainly that “the judge should not be the model you are testing.”

A plain text check confirms the reply names the file.

---
type: regex
pattern: 'digest'
flags: i
---

And a tool check confirms the main session actually called the agent.

---
type: tool_used
tool: Agent
input_match: '"subagent_type"\s*:\s*"[^"]*plan-adversary'
arm: with-only
---

Two arms. By default, each case runs three times with the plugin and three times without it. A grader marked arm: with-only checks the agent’s own format, so it can’t count against plain Claude. It reports, but it doesn’t score.

Running it. This is run.sh.

#!/usr/bin/env bash
# Copies the live global agent into this eval plugin, then runs the suite. Extra arguments go to `claude plugin eval`.
#   ./run.sh --runs 1 --ablation none --tag dev --max-cost-usd 10   # quick iteration on the dev cases
#   ./run.sh --runs 3 --max-cost-usd 25                             # scored round, with and without the agent
# The scaffold scripts under evals/ are our own, so --scaffold and --trust-plugin are safe here.
# No Bash grant: on this Mac the eval sandbox refuses Bash (a symlink inside ~/.docker), and the cases only need
# Read/Glob/Grep. The agent's Bash tool is simply absent in these runs.
set -euo pipefail
cd "$(dirname "$0")"
cp "$HOME/.claude/agents/plan-adversary.md" agents/plan-adversary.md
exec claude plugin eval . --scaffold --trust-plugin --no-publish --judge-model sonnet --threshold 0.9 "$@"

One quirk is that my eval runs had no Bash. On my Mac, the eval sandbox refused to grant Bash because of a link inside ~/.docker, so the agent read files with Read and Grep only.

To run it, open a terminal, go to the suite folder and start a round.

cd ~/.claude/agent-evals/plan-adversary
# quick check of the 6 dev cases, agent only
./run.sh --runs 1 --ablation none --tag dev --max-cost-usd 10
# full scored round, 8 cases x 3 runs, with and without the agent
./run.sh --runs 3 -j 4 --max-cost-usd 30 --keep-temp --json evals/results/v3-final.json

--keep-temp keeps each run’s transcript, so I could read the actual reports afterwards.

Each round writes a JSON file. This 17-line script, summarize.mjs, prints one line per run.

// Prints a compact summary of an eval JSON result: per case and arm, run scores, grader pass/fail, errors, cost.
import { readFileSync } from 'node:fs';
const r = JSON.parse(readFileSync(process.argv[2], 'utf8'));
console.log(`cost $${(r.costUsd ?? 0).toFixed(2)}  duration ${Math.round(r.durationSeconds ?? 0)}s  partial ${r.partial}`);
for (const c of r.cases ?? []) {
  for (const [arm, runs] of Object.entries(c.arms ?? {})) {
    const list = Array.isArray(runs) ? runs : runs?.runs ?? [];
    for (const [i, run] of list.entries()) {
      const g = (run.graders ?? []).map((x) => `${x.name}:${x.passed ? 'P' : 'F'}${x.scored === false ? '*' : ''}`).join(' ');
      console.log(`${c.name.padEnd(17)} ${arm.padEnd(7)} run${i + 1} score ${Number(run.score ?? 0).toFixed(2)}  ${g}${run.error ? '  ERROR ' + String(run.error).slice(0, 120) : ''}`);
    }
  }
}
if (process.argv[3] === '--why') {
  for (const c of r.cases ?? []) for (const [arm, runs] of Object.entries(c.arms ?? {})) for (const run of (Array.isArray(runs) ? runs : runs?.runs ?? []))
    for (const x of run.graders ?? []) if (!x.passed) console.log(`\n[${c.name}/${arm}/${x.name}] ${String(x.reason ?? x.rationale ?? x.details ?? JSON.stringify(x)).slice(0, 600)}`);
}

node summarize.mjs evals/results/v3-final.json

These are real lines from the final round. P is pass, F is fail, and * marks a grader that reports but doesn’t count toward the score.

cost $13.76  duration 801s  partial false
missed-consumer   with    run1 score 1.00  cites-code:P* focused:P* keyfile:P substance:P trigger:P* verdict-first:P*
missed-consumer   without run1 score 1.00  keyfile:P substance:P
unverifiable-env  with    run1 score 1.00  cites-code:P* focused:P* substance:P trigger:P* verdict-first:P*
unverifiable-env  without run1 score 0.00  substance:F

Don’t trust an AI judge blindly

The first full round tested the original agent on the six dev cases, three runs each, for $11.11. It caught every planted problem in all 15 runs. It approved the good plan in 2 of 3.

The miss turned out to be fair. The good plan never said to import countNotes into the route file, and one run flagged it. I fixed the plan so the case has one clear answer.

The format grader was the real problem. It was an AI judge asked whether the report led with the verdict, tagged every finding, cited code and stayed short. It passed 4 of 18 in that round.

Part of that was my setup, because the judge was reading the wrong part of the transcript. After I pointed it at the final reply, it still failed a clean 626-word approval three votes out of three. The verdict was on the first line, and it cited the code it checked.

My take is simple. Don’t trust an AI judge blindly. A grader that fails a clean report three times in a row isn’t strict. It’s confidently wrong.

Claude Code’s own auto mode was built on model judges. Its leaked source showed a fast judge and a slow judge checking risky actions, which I broke down in Claude Code auto mode. So the lesson isn’t to avoid AI judges. It’s to test them.

So the format checks became three plain text checks. One looks for the word VERDICT in the first 600 characters.

---
type: regex
arm: with-only
pattern: '^[\s\S]{0,600}VERDICT'
flags: i
---

One looks for a file:line citation.

---
type: regex
arm: with-only
pattern: '[\w./-]+\.(js|ts|sql|md|json):\d+'
---

One makes sure the reply stays under 9,000 characters, about 1,300 words.

---
type: regex
arm: with-only
pattern: '^[\s\S]{0,9000}$'
---

They cost nothing to run, and they give the same answer twice. The AI judge kept the questions that need judgment, like “did it name the digest job as the thing that breaks?”

Lance’s article says to “read a sample of scored transcripts before believing your evaluator.” That’s how both problems surfaced, by reading the actual reports instead of the scores.

One more fix in this round. An early draft of version 2 listed a “seconds versus milliseconds” trap, which handed the agent the answer to one test. I removed it. The article’s rule is blunt. “Never paste failures into the prompt.”

Why did a “correct” test fail every run?

The final round ran all eight cases, three runs each, with and without the agent. That’s 48 sessions, $13.76 and about 13 minutes. Seven cases passed every run. The held-out “good plan, odd code” case failed all six, with the agent and without it.

Lance’s article names this exact pattern, saying “a common tell is that a task fails every evaluation run, regardless of the number of replicates.” When everything fails, suspect the test.

So I read the six reports. All six said the same thing. The test app’s server had a comment saying it handled JSON bodies, and the code never read one. The planned edit route could only ever return 400. One report put it like this.

nothing in the server ever sets req.body, so as written the PATCH route can only return 400.

That’s the trap the agent was built for, a comment promising something the code doesn’t do. The agent passed the test by proving the test wrong.

I fixed the test server and tightened the plan, closing smaller gaps the reports had found. On the re-run, the agent passed 2 of 3.

The failed report approved the plan. As a pre-existing issue, it also noted that getNote breaks the app’s own “every query filters by user_id” rule. That’s accurate, and it was filed in the right place. My grading rule failed it anyway, because it told the judge that any mention of getNote counted against the review.

I rewrote the rule to fail only a review that blocks the plan. The final re-run scored 3 of 3 with the agent and 1 of 3 without.

Fixing a held-out case after seeing its results means it’s no longer a clean final exam. The other held-out case, the claim nobody can check, never needed a fix.

Did the agent beat plain Claude?

Yes, 24 runs to 19, but not where I expected.

On the five planted-problem cases, both scored 15 of 15. Both approved the good plan 3 of 3. If those cases were the whole test, the agent would be a tie.

The gap is in the last two rows. On the claim nobody can check, the agent scored 3 of 3 and plain Claude 0 of 3. Plain Claude noticed APP_URL was missing from .env.example and warned that local links would break.

It never questioned the plan’s claim that production already had it. One of its runs wrote “APP_URL is only set in production.” Another wrote “APP_URL isn’t set outside production.” Both take the plan’s word for it.

The agent filed that claim under UNVERIFIED every time, and said what would settle it. This is its line from run 1.

UNVERIFIED
  `APP_URL` in the production environment: the repo can't show this. Check the deployed env config or secret store. Whatever it shows, add `APP_URL` to .env.example, and either return 500 when it is missing or fail at startup, rather than building `undefined/...`.

On the good plan next to odd code, the agent approved it 3 of 3. Plain Claude approved it once. Its two misses hedged, listing gaps in the plan instead of giving a clear go-ahead. One said there were “some gaps to close before you implement it,” which is exactly the hedge this case tests for.

The format checks didn’t count toward the score, but they passed in all 24 agent runs. Every reply put the verdict first, cited a file:line and stayed under 9,000 characters. The main session called the agent in all 24 runs too.

Length is the weak spot. The agent’s replies had a median of 1,030 words, against 668 for plain Claude, and 13 of 24 went past the 1,000-word target.

Here’s where the $34.39 went.

These are the tool’s list-price estimates. The docs say eval runs count against your plan’s usage limits or your API bill, the same as normal sessions. I’m on a flat Claude Max plan, so I didn’t pay these amounts. I explained how list price relates to a flat plan in what the $500 means if I pay a flat price.

What can’t these LLM evals tell me?

Plenty. Here’s what I’d tell anyone reading the 24 to 19 too generously.

• Too little headroom. Plain Claude already caught every planted bug. Lance’s article warns when a baseline scores about 95% or higher, because there’s no room left to see a change. My planted-bug cases sit at 100% for both arms, so they can’t show the agent getting better at finding bugs.
• Too few cases. Eight cases, three runs each. Anthropic’s guide says to prioritize volume over quality, and eight is not volume.
• One held-out case is used up. The “good plan, odd code” case was fixed after I saw its results.
• The trigger number flatters the agent. The test prompt mentions a review agent, which nudges the main session to call one. In real use, the CLAUDE.md rule does that job.
• The reports run long. 13 of 24 went over the target.
• The 1 to 10 rating is a judgment. Only the with-and-without results are measured.
• One model, one effort level. I tested Opus at high effort and nothing else.

Lance’s article also describes two Claude Code commands that automate this loop, /claude-api build-eval and /claude-api hillclimb. Build-eval runs the grader twice on the same output to see if the verdict changes, and it warns when the baseline leaves no headroom. Those two checks would have caught my two biggest problems early.

How to run LLM evals on your own subagent

1. Write “done” down first. Make it specific and measurable, with which cases must pass, how many runs, and what beating the baseline means.
2. Wrap the agent in a plugin. A folder with .claude-plugin/plugin.json, agents/<name>.md and evals/. Run claude plugin validate . to check the files.
3. Build one small fake repo with planted traps. Give each case a setup script that copies it and commits it to git. Setup scripts run as you, so only pass --scaffold for scripts you wrote.
4. Write cases with one clear answer. Include plans that are fine, so you can catch false alarms.
5. Hold some cases out. Don’t look at them while you tune the prompt.
6. Use code graders first. Use a judge model only for real judgment calls, with written PASS and FAIL conditions, and pick a judge that isn’t the model you’re testing.
7. Keep the baseline. The default with-and-without run is the only way to see what the agent adds.
8. Read the transcripts. Run with --keep-temp and read the reports behind every failure, and a few passes.
9. Suspect the test when everything fails. A case that fails every run in both arms usually has a bug in the case or the rubric.
10. Re-run only when something changes. A new prompt or a new model. Evals are not a per-plan step.

claude plugin validate .
claude plugin eval . --scaffold --trust-plugin --no-publish --judge-model sonnet --runs 3 --max-cost-usd 30

Have you run evals on your own subagents yet, and did your AI judge hold up the first time you read its transcripts?