Lesson 3.6 — Grading the grader

Your reviewer is a judge. Every judge failure mode is now yours too.

TL;DR: Rung ④'s fresh-context reviewer (Lesson 3.2) is an LLM judge — it inherits every documented judge failure mode: position, verbosity, and self-enhancement bias ¹. Make its verdicts auditable: lock and hash the rubric, stamp the judge's model ID on every verdict ³, mechanically verify every quoted finding against the source and fail closed on a mismatch ², and keep re-checking judge-vs-human agreement over time ⁴. The same treatment applies recursively to the skills and procedures you build, not just the diffs they produce ⁵.

ELI5: the judge who never shows their work

A judge who scores from memory, off an unwritten rubric that changes every session, and can't be appealed, isn't accountable — grading the grader means making the verdict show its work.

Imagine a judge who scores a case, but: the rubric they used isn't written down anywhere, nobody knows which version of "the law" they were even applying that day, and when they say "the contract violates clause 4," nobody checks whether clause 4 actually says that. You'd never accept that verdict as final. That's what an unaudited reviewer subagent's PASS looks like today — right up until you write down the rubric, stamp who's judging, and check the citations.

The reframe: your reviewer is a judge

Rung ④'s "fresh eyes" (3.2) render a probabilistic judgment, not a deterministic fact — so it inherits every documented failure mode of an LLM judge.

Lesson 3.2 already flagged this: rungs ①–③ are deterministic (a command exits 0 or it doesn't); rung ④ is a model rendering a verdict. The specific failure modes are well documented — LLM judges carry measurable position bias (favoring whichever answer is shown first/last), verbosity bias (rewarding longer output regardless of quality), and self-enhancement bias (rating output from similar models more favorably) ¹. Nothing about wrapping the judge in a fresh-context subagent and a tight correctness rubric (6.2) removes these biases — it only removes the author's blind spots. The judge's own blind spots are still there.

🧠 Test Yourself: Your fresh-context reviewer (6.2) can't see the author's reasoning, so it can't rationalize "I meant to do that." Does that make it bias-free?

Answer

No. Fresh context removes the author's specific rationalization blind spot — it does nothing about the judge's own position/verbosity/self-enhancement bias ¹. Independence and freedom from bias are two different properties.

How a verdict goes wrong

Three concrete ways a PASS/FAIL can mislead you, beyond the bias taxonomy above.

• An agreeable PASS under a long, easy rubric. A wide-open "review this" rubric invents nits on a clean diff and glosses over a real bug on a messy one — this repo's own fix for that (6.2's correctness-only, PASS-allowed rubric) is a mitigation, not a cure.
• An invented finding. The judge cites "SPEC line 14" for a requirement the spec never states, because a plausible-sounding citation is easy for a language model to produce and hard for a human skimming ten findings to catch.
• A quote that doesn't match the source. The judge's finding sounds grounded — it quotes the diff or the spec — but the quoted text isn't actually in either. This is the failure mode the mechanics below are built to catch mechanically, not by re-reading everything yourself.

Making a verdict auditable

Four mechanics, roughly in order of how easy each is to skip if you don't write it down.

flowchart TD
    R["rubric (fixed for this review cycle)"] -->|"hash it"| RH["rubric_hash"]
    D["diff + spec"] --> J["judge model<br/>(model ID recorded)"]
    RH --> J
    J --> V["verdict: PASS / FINDING + quoted evidence"]
    V --> QC{"quoted evidence found<br/>verbatim in diff/spec?"}
    QC -->|yes| LOG["logged verdict<br/>(rubric_hash + model_id + verdict)"]
    QC -->|no, mismatch| DG["downgrade to ADVISORY<br/>— never silently trusted"]
    LOG --> SAMPLE["periodic sample →<br/>human adjudicates"]
    SAMPLE --> AGREE["judge ↔ human agreement<br/>tracked over time"]
    AGREE -. "drifts low → revise rubric/judge" .-> R

Mechanic	What it buys you
Freeze the rubric, hash it, stamp the hash on every verdict	A verdict is reproducible against the exact criteria used — "the rubric changed" stops being an untraceable excuse 2
Stamp the judge's model ID on every verdict	A silent model upgrade is a silent change to your oracle — Anthropic's own model-ID docs exist because dateless IDs used to be rolling aliases; pin and record which snapshot judged this diff 3
Mechanically verify quoted evidence against the source; fail closed on mismatch	A finding whose quote doesn't check out is downgraded to advisory, not silently merged as ground truth — recent work on rubric-based LLM evaluation names this exact technique "extractive quote verification" 2
Keep a small human-adjudicated set; re-check judge↔human agreement continuously	Catches drift before it's a pattern of bad merges — an unvalidated judge "launders vibes into fake rigor" if you never check it against a human again 4

🧠 Test Yourself: Your reviewer subagent flags a finding: "Violates SPEC line 9: must reject negative values." You check SPEC line 9 — it says nothing about negative values. What should the harness do with this verdict?

Answer

Fail closed: downgrade the finding to advisory rather than treating it as ground truth. A quote that doesn't verify against the source is exactly the failure mode mechanical evidence-checking exists to catch ².

Worked example

Same finding, two harnesses — one trusts the quote, one checks it.

The reviewer returns: "FINDING: limiter uses '>' not '>=' — violates SPEC line 14 ('reject the Nth request')."

• ❌ Unaudited harness: the finding blocks the merge. Nobody checks SPEC line 14. It turns out line 14 says nothing about the Nth request — the judge paraphrased a nearby line and cited the wrong one. A correct diff sits blocked on a hallucinated citation.
• ✅ Audited harness: a mechanical check pulls SPEC line 14 and diffs it against the quoted text. They don't match. The finding is automatically downgraded to advisory — surfaced to a human, not auto-blocking — and the verdict log records which rubric hash and model ID produced it, so the next person to hit this pattern can see it's a repeat.

The recursive step: grading your skills, too

The same judge machinery works one level up — on the SKILL.md files and procedures you write, not just the diffs they produce.

If a reviewer subagent needs grading, so does the skill or subagent definition that produced the work in the first place — a skill with a vague description is a defect the same way an unvalidated judge is. Anthropic's own skill-creator skill ships exactly this as a built-in eval/benchmark mode: spawn parallel with-skill and without-skill runs on the same task, grade each against written expectations with a grader subagent, then run an analyzer pass over the aggregate pass-rate, timing, and token data to surface patterns a single number would hide ⁵. It's the grading-the-grader mechanics above, one layer up: lock the test cases, stamp what was compared, and don't trust a "the skill helped" verdict you haven't checked against the actual runs.

🧠 Test Yourself: Why does grading a skill need the same rigor as grading a diff?

Answer

A skill is itself an artifact a judge (here, a benchmark comparator) evaluates — so it inherits the same failure modes: an ungrounded "the skill helped" claim is exactly as trustworthy as an ungrounded review PASS, unless it's checked against locked test cases and real run data ⁵.

Your turn (exercise)

Take a reviewer subagent (or rubric) you already use. Add three things to its output contract:

1. A rubric_hash field (hash the rubric text; stamp it on every verdict).
2. A judge_model field (the exact model ID, not a rolling alias).
3. A rule: any finding that quotes the diff or spec must be checked verbatim before it's trusted — if the quote doesn't match, the finding becomes advisory, not a merge-blocker.

Run it on a real diff. Did the quote-check ever catch a mismatch? That's the failure mode this lesson exists to make visible instead of silent.

---

← Lesson 3.5 · Phase 3 home · → Check your understanding

---

1. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Zheng et al., NeurIPS 2023 ↩↩↩

2. From Rubrics to Reliable Scores: Evidence-Grounded Text Evaluation with LLM Judges — Hong et al., 2026 (locked task-level rubric specifications; typed evidence grounding and extractive quote verification) ↩↩↩↩

3. Model IDs and versioning — Anthropic ↩↩

4. Your AI Product Needs Evals — Hamel Husain ↩↩

5. skill-creator — Anthropic (anthropics/skills; parallel with/without-skill runs, grader + analyzer subagents, aggregate pass-rate/timing/token benchmark) ↩↩↩