A test suite went green. Every assertion in it was correct. It had executed nothing.
Not "executed and passed trivially." Executed zero tests, printed a check mark, and exited zero. The filter that selected which tests to run had matched an empty set, and nothing in the pipeline was watching the count. It stayed that way for a while, because a green run is the one result nobody investigates.
That happened to us three separate times, in three unrelated gates, and none of the three was catchable by reading the assertions. This is what came out of it.
The failure has a general shape
The artifact that produces confidence and the artifact that ships are different objects, and nothing automatically asserts they correspond.
That one sentence covers a surprising amount of ground. A hermetic test certifies a helper the dispatch seam never calls. A route hand-rolls its own dictionary next to the type's serializer and never gains the two fields somebody added last month. A pattern check sweeps for a forbidden token using a pattern that can no longer match anything. In each case the verification is real, the shipping code is real, and the relationship between them is assumed rather than checked.
Calling it one defect class rather than seven unrelated bugs is what makes it tractable, because then you can ask a single question of every gate you own: which shipping artifact does this actually touch?
The ladder: prefer removing the possibility
Before building a gate, ask whether the thing needs gating. There are five rungs, strongest first, and you descend only when the rung above is genuinely unavailable and you can say why.
1. Collapse. Delete the seam. Two producers become one. Patching a duplicate leaves the divergence that produced it, so the next field to be added repeats the incident exactly.
2. Compile. Make the wrong state unrepresentable. No gate needed, because the build is the gate.
Rungs one and two remove the possibility. Below this line you are detecting an occurrence, and every detector can rot.
3. Assert. A gate that executes the shipped artifact. Live beats hermetic here by construction, because a hermetic test can inject a double and certify nothing.
4. Register. For seams that can be neither collapsed nor compiled away, a marker at the seam, in the source, naming the other artifact and the assertion that holds them together. Never a separate registry file, which drifts from the code it describes the moment somebody moves a function.
5. Observe. A committed, re-runnable instrument standing behind a measured claim. This rung is about claims rather than code, and it is the only one whose artifact is a document.
The horizontal line between rungs two and three is the whole point of the ladder. Most engineering effort around correctness goes into building better detectors, and a detector is structurally the weaker move. It has to keep working, which means it has its own failure modes, which means it needs its own verification, which is how you end up here.
When a collapse is refused for a real reason, the seam does not fall off the ladder. It drops to rung four and the refusal becomes the marker's rationale.
Six tests every gate must pass
If you are building a detector, it owes you these.
(a) Falsifiable. State the mutation that reds it before claiming it works.
(b) Green today. A standing red nobody can clear is worse than no check. Ratchet on a monotonic fact instead, so it arms itself the day it lands.
(c) No mixed worklist. Precision over recall. A gate that opens with a worklist full of false positives is one everyone learns to skip.
(d) Prefers removing the possibility to detecting the occurrence. The ladder restated deliberately, because it is the test most often skipped.
(e) Discloses its own arming. A skipped, zero-executed or vacuously satisfied run must not read as a pass. This is the one that cost us three incidents, and it is invisible to tests (a) through (d): each of those gates had correct assertions and each reported green over nothing. Read the executed count, not the check mark.
(f) A pattern gate carries a known-positive. For any gate whose verdict is the absence of matches, a zero-hit run is indistinguishable from a pattern that cannot match anything. Test (a) covers assertion gates. This is its pattern-gate form, and it is skipped most often because a green grep feels like evidence.
Test (f) deserves its own paragraph because of how ordinary the failure is. Somebody renames a directory. A forbidden-token sweep that scoped itself to the old path now searches nothing, finds nothing, and passes. The sweep is still in CI. It is still green. It has been dead for four months. The only defense is a committed fixture the check must find, verified on every run, so a check that has stopped seeing real code fails on its own test data first.
An assertion you have never reddened is not an assertion
Mutation is the admission test for the bottom three rungs. Break the shipped thing, assert the gate reddens, restore.
The important half is that the mutation gets recorded in one line beside the gate. Synthetic positives are the easy case and largely solved by committed fixtures. What otherwise goes unrecorded is source-replay knowledge, the specific fact that widening this guard reds that named test and nothing else. Without it, a gate whose mutation no longer bites is invisible, because it looks exactly like a gate that is working.
A committed fixture is strictly safer than replaying a mutation on real source, and it reaches further than it looks. A consumer's own parse-and-assert code can run against a fixture payload with no rebuild at all, which is how you check that a consumer would notice a shape change.
The restore trap, which is worse than the mutation
When source replay is unavoidable, restore by checksum from a copy you made yourself, and do not preserve the modification time.
Here is the failure. You mutate a source file, watch the gate go red, restore the original bytes, and re-run. A checksum-verified restore proves the source is back. It does not prove the binary is. If the restore preserved the original mtime, an incremental build recompiles nothing and hands the next run a binary still built from the mutant.
Now the gate is testing the broken artifact and reporting on the fixed one. That is a false green in the most dangerous possible direction, and it is this same wiring-versus-execution failure recursively eating the tool built to catch it.
Wiring and execution are different guarantees
Neither substitutes for the other, and any change claiming to close a correspondence should say which one it delivers.
That a production value is named is checkable at compile time and costs nothing. That it is executed is only provable by breaking it and watching a live gate go red, which costs a rebuild and a run.
The same split governs consumers. Feeding a consumer a fixture proves it would notice a shape change. It does not prove the serializer you kept is the one actually reached on the wire. The first is cheap and belongs in every collapse. The second is expensive and is the guarantee people assume they have when they do not.
What we refused to build
Three further instruments were proposed, measured, and turned down, each recording the evidence that would reverse the decision.
The most instructive refusal was a central registry file listing every seam and its counterpart. It reads like the obvious answer and it is a trap: the registry is a second artifact that must correspond to the code, which is the original problem with extra steps. Markers live at the seam, in the source, where the person editing the seam cannot avoid seeing them.
The second refusal worth naming is a phantom detector running by default. It found real things and it also produced a mixed worklist, which fails test (c). A gate that cries wolf trains people to ignore gates, and that cost outlives whatever the gate caught.
The one thing this does not give you
Producer reachability. Nothing here proves that the serializer surviving a collapse is the one a live request actually reaches. Every other guarantee in this document is cheaper than that one, and we have not paid for it.
We write that down rather than letting it be inferred, because a document about gates reporting green over nothing has an obligation not to do the same thing itself.