Never Let the Generator Grade Its Own Work

Ask a model to check its own work and it will usually agree with itself. That is not dishonesty. The output it just produced is, by construction, the output it found most plausible, so reading it back for plausibility is the one test it is guaranteed to pass.

The most valuable single piece of the AI second brain for my enterprise architecture practice is a reviewer that exists to say no. Here is how it works and what made it trustworthy.

Separate the judge from the generator

The reviewer is its own skill with its own instructions, and its job is narrow: decide whether a change is trustworthy enough to commit, publish or build on. It does not improve the work, praise it, or rewrite it. It returns a verdict, PASS or FAIL, and every FAIL names a location, what is wrong, why it matters and the fix.

It also never fixes what it finds. That rule feels wasteful the first time, because the reviewer often knows the fix. But a judge that edits its own findings is back to grading its own work. The value is the separation.

Three calibration levers

1. Assume it is wrong. The default stance is that the artifact fails until the checks prove otherwise. No summary of what went well. Find what breaks.
2. Use a different model or a fresh session. A model forgives in review the same patterns it favored while generating. Running the reviewer in the same turn as the generator is the weakest form; a fresh session, ideally on a different model, is the strongest.
3. Run the checks. Do not read. Plausibility is exactly the surface the generator optimized for. So the reviewer executes validators, searches the cited sources, and opens the canonical page to compare a number. A check it did not actually run does not count.

What it checks

The checks depend on the artifact type, but a representative set:

• Integrity. Nothing truncated, nothing that shrank when it should only grow.
• Provenance. Every figure, disposition or recommendation carries a source, and any figure copied from elsewhere matches its canonical home.
• Duplicates. A new page is checked against the index for an existing page it should have enriched instead.
• Links and schema. New pages are linked from at least two places, every link resolves, and required fields are present.

A reviewer can fail too

The first time I ran it across the whole knowledge base, it reported 478 broken links. About 470 were false positives. The link resolver only knew one folder, split table links on the wrong character, and treated intentional forward references as errors. The knowledge base was fine; the instrument was wrong.

If a check fires on an implausible number of items, the check is wrong far more often than the thing it checks. Fix the instrument before you trust the verdict.

That lesson became a rule inside the skill. A huge alarm dump is its own failure mode, just as a rubber stamp is.

Measure the reject rate

Every review, pass or fail, appends one line to a log: date, verdict, artifact, number of defects, a short reason. The point is to make calibration visible. A healthy reviewer shows a believable mix over time. A month of nothing but PASS means it has become a rubber stamp, and it is time to tighten the stance. A wall of FAILs means the checks are not resolving doubt into specific verdicts.

Copy this: a reviewer skill

Save this as a skill or paste it into a fresh session along with the change you want checked. Replace the checks with the ones that matter for your work.

---
name: skeptical-reviewer
description: Independently verify a change before it is trusted, committed or
  published. Use when asked to review, check, fact-check or evaluate a change.
---

# Skeptical reviewer

You decide whether a change is trustworthy enough to commit. You do not
improve it, praise it or rewrite it.

## Stance
- Assume the change is WRONG until your checks prove otherwise.
- Run checks. Do not read for plausibility. A check you did not run
  does not count.
- Never fix what you find. Report it and hand it back.

## Checks (run every one that applies)
1. Integrity: nothing truncated; append-only files did not shrink.
2. Provenance: every figure, decision or recommendation has a source;
   any copied figure matches its canonical page.
3. Duplicates: a new page is not a near-copy of an existing one.
4. Links and schema: links resolve; required fields are present.

## Calibration
- If a check flags an implausible number of items, suspect the check.
  Fix the instrument and re-run before reporting.

## Output
REVIEW: PASS | FAIL - <artifact> - <date>
Checks run: <each, with pass/fail>
Defects: <file:line> - <what fails> - <why it matters> - <the fix>

Append one line per review, pass or fail, to log/reviews.md.

Try it on your own work

You do not need my setup to use the pattern. Before you trust an AI-produced change that matters:

• Hand it to a fresh session with instructions to find what is wrong, not to review it.
• Require runnable evidence for every finding: a command, a search, a comparison.
• Forbid fixes in the review pass. Collect the verdict, fix it yourself or in the generator, then review again.
• Keep a log of verdicts so you can tell a reviewer from a rubber stamp.

A reviewer that can say no is the difference between an AI that produces a lot and one you can build on.