The Test That Was Never a Test

An agent told me the tests passed. They did. The feature was still broken.

That sentence is the whole reason this post exists. Not bad code — an honest report of something that didn't happen. The agent ran a check. The check was green. And the check was guarding nothing.

I run several Claude Code agents on one codebase, for months at a time. This is the operating model I ended up with, and the failure that forced it.

The ten-second check almost nobody runs

Here it is:

Delete the feature. Run the suite.
Still green? That test was never a test.

That's it. Ten seconds. Try it on something you shipped last month.

A passing test proves one thing only: this code does not make it fail. It does not prove the test is guarding anything. Those are completely different claims, and we treat them as the same claim every day.

The worked example in my repo is a test that had existed for eleven months. Its name stated the guarantee the entire feature rested on. Delete the deduplication logic — not the key, the whole check — and the suite still reports 815 passed, 0 failed. The legacy test built its own fixture, sent once, and asserted the mailbox had one message. It never triggered a retry, so a second send had no way to appear.

A hollow guard is worse than a missing one. A missing guard gets noticed.

Better prompts don't fix this

This is the part I got wrong for a long time. I kept trying to write my way out of it — be careful, be rigorous, be objective, double-check your work.

It does very little. You cannot prompt an agent into not believing itself.

What does work is structure. Five rules, and one move underneath all of them.

The one move

Take the state out of the conversation and put it on disk.

Everything else follows from that. Context gets compacted, interrupted, restarted — anything that exists only in the chat can vanish at any moment. The chat is the driver's seat, not the database.

The five rules

1. Acceptance takes artifacts, not reports. A diff, a file, the output of a check, a page you can open. The test is whether a third party can pull it up and look at it themselves. "I finished it" is a lead, not evidence.

2. Review is independent by construction. The reviewer has no write access to what it reviews, and no channel to the author. Findings go to the hub, which transcribes them into a new work order. Telling an agent to "be objective" does almost nothing. Removing its write access and cutting the line to the author does a lot — it now has no option but to read the artifact. Independence cannot be requested. It has to be built.

3. State lives on disk. Orders, replies, decisions, and the reasoning behind them — all files. What survives a new session, a new machine, or a new person is what got written down.

4. Shifts are one-shot; permissions are per-order. Every run is a fresh process with no memory of the last one, working from the order plus what's on disk. No memory, no drift. Tools are granted for that order only, so one mistake is bounded by one order instead of the whole repository.

5. A human presses anything irreversible. Merge, deploy, publish. Agents work up to "ready." This isn't distrust — it puts the person at the single point where the information is complete and the cost is highest, instead of asking them to approve every call.

Put together, the shape is a hub and spokes — and the spokes are deliberately not connected:

Swinging knives

Rule 1 is where the real work is, and the technique is mutation testing done by hand.

Break the behaviour on purpose and see whether the test goes red. Not red means the guard is fake.

Aim at things that look correct and are not. Deleting a whole function teaches you nothing — of course that goes red. Swap the order of two operations. Set a constant to another plausible value. Change an event name to a different valid-but-wrong one. Let the code run normally with the behaviour quietly changed.

The sharpest lesson I've learned here: for the same hole, inserting a decoy before the real entry went red, while appending it after stayed green. The assertion checked presence, never only one. The shape of the knife changes the conclusion.

And two people should swing, with different knives. Reviewers tend to test "is the thing there." Somebody has to test "the thing is there, and it's wrong." Two identical knives are one knife.

What it costs

Leaving this part out would make this an advertisement.

It's slow. A mailbox round trip, a review pass, a re-check. Running a typo fix through this is absurd.

The hub is both a bottleneck and a single point of failure — if it mis-records or garbles someone's reasoning in transcription, the whole chain inherits the error. Every shift reloads its context from scratch, which is the price of having no memory. And the bookkeeping is real work: dispatch and record-keeping have to be the same action, because the moment they're two commands, one of them gets skipped. Miss once and your status board is lying while everyone makes decisions from it.

When it's worth it

One question: does this work outlive a single context window?

If not, just do the work. Don't start a process. A rule is worth what it's actually held to, not what it says — apply it where it isn't warranted and it'll get broken, and a rule that's broken quietly is worse than no rule, because everyone still believes it's there.

Long projects don't go off the rails in one step. They drift. A guarantee that quietly stopped being true. A decision nobody can reconstruct three weeks later. That's what this is for.