Green Tests, Broken System: 4 Ways a Test Suite Lies

A green test suite is a claim, not a fact. It says the code satisfies the tests. It says nothing about whether the tests describe reality. Building tooling for the AI second brain for my enterprise architecture practice, I watched suites go green over broken code in four different ways. Each one had the same root cause: I trusted the test surface instead of checking what it actually asserted.

1. The fixture that encodes your assumption

I built a tool that walks relationships in a configuration database. The tests passed, 65 of 65. The fixtures described relationship types the way I assumed the upstream system stored them: a verb, which the code then combined with its reverse. The real system already stored the combined value. Every traversal in production would have failed.

Worse, the failure would have looked like correct behavior, because the code was designed to return nothing for an unrecognized verb. A total outage would have presented as a quiet, well-handled edge case.

A test fixture is an assertion about upstream shape. Give it a figure's provenance: copy real rows, record the date you read them, re-read when the contract changes.

2. The rule that can never fire

Lookup tables keyed on upstream values — a verb map, a field crosswalk, a status translation — can hold entries for things that do not exist upstream. They never error. They never misbehave. They simply never apply. In one audit, 2 of 16 hand-written mapping rules were phantoms of exactly this kind. No test caught them, because there was nothing to catch.

Two controls, in order:

1. Derive the table from the source rather than maintaining it beside the source.
2. If it must be hand-maintained, make it report what matched nothing. A table that lists its own dead entries turns an invisible class of bug into a line of output.

3. The check that stops looking

Some checks scan source code for a property: this function must call that guard, this block must include that filter. Twice in one build, a check read a fixed window of characters from a known starting point. Someone added a few lines above the target, the target slid out of the window, and a correct file went red.

That is the mirror image of the phantom rule. Not a rule that never fires, but one that quietly stops seeing its subject. The signal it gives is about file layout, not the property it claims to test. The fix is to anchor source-scanning checks on a function or block boundary, never on a byte offset.

4. The gate that defends the defect

This one is the worst of the four. A pre-existing check asserted that a certain write request, sent without a required safety header, would still be forwarded when writes were enabled. The check was correct in the narrow sense: it fired, and it asserted what the code did. What the code did was a hole. The suite was green because the hole was open, and closing the hole would have shown up as a regression.

The control is a habit of reading. When a test asserts that something permissive happens, treat it as a claim about intent and confirm that someone actually decided it. A test describing behavior nobody chose is a defect wearing a test's clothes. When you invert one, label it as inverted in the same commit, so the next reader sees a correction rather than a suspicious flip.

A fifth, for anything with a screen

Ten source-level and headless checks stayed green on a diagramming tool while a person found seven defects in a single sitting: dead keyboard shortcuts, a broken link, a properties pane that never filled, and every item wrongly reported as missing. One browser test that drove real clicks, keys and right-clicks against real data caught all seven. Source checks guard structure. They are not evidence a feature works. Any new user-facing feature now ships with at least one real-input test.

Copy this: verify-before-assert rules

I added this section to the contract for every repo an AI works in:

## Verify before you assert
A claim about an artifact must come from its contents, not its surface.
Filenames, sizes, commit counts and green badges support a hypothesis,
never a verdict. Open the thing.

- Fixtures: build test fixtures from real upstream rows. Record the date
  you read them beside the fixture. Re-read when the contract changes.
- Lookup tables: derive them from the source where possible. If one must
  be hand-maintained, make it report entries that matched nothing.
- Source-scanning checks: anchor on a function or block boundary, never
  on a character or byte offset.
- Permissive assertions: when a test asserts that something permissive
  happens, confirm someone decided that. If you invert one, say so in
  the commit message.
- UI: any new or changed user-facing feature ships with at least one
  test that drives real clicks and keys against real data.
- Before any delete or archive, confirm from contents and confirm the
  work is pushed.

The pattern underneath

All five are the same mistake: treating the surface of an artifact as evidence about its contents. Test counts, green badges, filenames and commit messages support a hypothesis. They never settle one. The discipline that fixed it for me is small:

• Build fixtures from real, dated rows. Your assumption is not a fixture.
• Make lookup tables report their phantoms. Silence is not proof of coverage.
• Anchor checks on structure, not position. Layout changes should not flip a verdict.
• Read permissive assertions as claims about intent. Confirm someone decided it.
• Drive the UI like a person would. At least once per feature.

None of this needs a new framework. It needs you to open the thing and look before you say it works.