---
title: "Read the Executed Count, Not the Check Mark"
description: "A passing test suite proves nothing if the assertions never ran. Six tests every gate must pass, and the ladder that says when to build one at all."
date: 2026-08-15
author: "Ben Racicot"
tags: ["Testing", "Architecture", "Code Quality", "CI", "macOS", "Swift"]
type: "paper"
canonical: "https://modelpiper.com/blog/green-over-nothing-gate-design"
---

# Read the Executed Count, Not the Check Mark

> A passing test suite proves nothing if the assertions never ran. Six tests every gate must pass, and the ladder that says when to build one at all.

## TL;DR

The artifact that produces confidence and the artifact that ships are different objects, and nothing automatically asserts they correspond. Three of our own gates reported green over zero executed assertions while every assertion in them was correct. This paper covers the five-rung ladder that decides whether to build a gate at all, the six tests one must pass, and why an assertion you have never reddened is not an assertion.

A test suite went green. Every assertion in it was correct. It had executed nothing.

Not "executed and passed trivially." Executed zero tests, printed a check mark, and exited zero. The filter that selected which tests to run had matched an empty set, and nothing in the pipeline was watching the count. It stayed that way for a while, because a green run is the one result nobody investigates.

That happened to us three separate times, in three unrelated gates, and none of the three was catchable by reading the assertions. This is what came out of it.

## The failure has a general shape

**The artifact that produces confidence and the artifact that ships are different objects, and nothing automatically asserts they correspond.**

That one sentence covers a surprising amount of ground. A hermetic test certifies a helper the dispatch seam never calls. A route hand-rolls its own dictionary next to the type's serializer and never gains the two fields somebody added last month. A pattern check sweeps for a forbidden token using a pattern that can no longer match anything. In each case the verification is real, the shipping code is real, and the relationship between them is assumed rather than checked.

Calling it one defect class rather than seven unrelated bugs is what makes it tractable, because then you can ask a single question of every gate you own: _which shipping artifact does this actually touch?_

## The ladder: prefer removing the possibility

Before building a gate, ask whether the thing needs gating. There are five rungs, strongest first, and you descend only when the rung above is genuinely unavailable and you can say why.

**1\. Collapse.** Delete the seam. Two producers become one. Patching a duplicate leaves the divergence that produced it, so the next field to be added repeats the incident exactly.

**2\. Compile.** Make the wrong state unrepresentable. No gate needed, because the build is the gate.

_Rungs one and two remove the possibility. Below this line you are detecting an occurrence, and every detector can rot._

**3\. Assert.** A gate that executes the shipped artifact. Live beats hermetic here by construction, because a hermetic test can inject a double and certify nothing.

**4\. Register.** For seams that can be neither collapsed nor compiled away, a marker at the seam, in the source, naming the other artifact and the assertion that holds them together. Never a separate registry file, which drifts from the code it describes the moment somebody moves a function.

**5\. Observe.** A committed, re-runnable instrument standing behind a measured claim. This rung is about claims rather than code, and it is the only one whose artifact is a document.

The horizontal line between rungs two and three is the whole point of the ladder. Most engineering effort around correctness goes into building better detectors, and a detector is structurally the weaker move. It has to keep working, which means it has its own failure modes, which means it needs its own verification, which is how you end up here.

When a collapse is refused for a real reason, the seam does not fall off the ladder. It drops to rung four and the refusal becomes the marker's rationale.

## Six tests every gate must pass

If you are building a detector, it owes you these.

**(a) Falsifiable.** State the mutation that reds it before claiming it works.

**(b) Green today.** A standing red nobody can clear is worse than no check. Ratchet on a monotonic fact instead, so it arms itself the day it lands.

**(c) No mixed worklist.** Precision over recall. A gate that opens with a worklist full of false positives is one everyone learns to skip.

**(d) Prefers removing the possibility to detecting the occurrence.** The ladder restated deliberately, because it is the test most often skipped.

**(e) Discloses its own arming.** A skipped, zero-executed or vacuously satisfied run must not read as a pass. This is the one that cost us three incidents, and it is invisible to tests (a) through (d): each of those gates had correct assertions and each reported green over nothing. **Read the executed count, not the check mark.**

**(f) A pattern gate carries a known-positive.** For any gate whose verdict is the _absence_ of matches, a zero-hit run is indistinguishable from a pattern that cannot match anything. Test (a) covers assertion gates. This is its pattern-gate form, and it is skipped most often because a green grep feels like evidence.

Test (f) deserves its own paragraph because of how ordinary the failure is. Somebody renames a directory. A forbidden-token sweep that scoped itself to the old path now searches nothing, finds nothing, and passes. The sweep is still in CI. It is still green. It has been dead for four months. The only defense is a committed fixture the check must find, verified on every run, so a check that has stopped seeing real code fails on its own test data first.

## An assertion you have never reddened is not an assertion

Mutation is the admission test for the bottom three rungs. Break the shipped thing, assert the gate reddens, restore.

The important half is that the mutation gets recorded in one line beside the gate. Synthetic positives are the easy case and largely solved by committed fixtures. What otherwise goes unrecorded is source-replay knowledge, the specific fact that widening _this_ guard reds _that_ named test and nothing else. Without it, a gate whose mutation no longer bites is invisible, because it looks exactly like a gate that is working.

A committed fixture is strictly safer than replaying a mutation on real source, and it reaches further than it looks. A consumer's own parse-and-assert code can run against a fixture payload with no rebuild at all, which is how you check that a consumer would notice a shape change.

### The restore trap, which is worse than the mutation

When source replay is unavoidable, restore by checksum from a copy you made yourself, and **do not preserve the modification time.**

Here is the failure. You mutate a source file, watch the gate go red, restore the original bytes, and re-run. A checksum-verified restore proves the _source_ is back. It does not prove the _binary_ is. If the restore preserved the original mtime, an incremental build recompiles nothing and hands the next run a binary still built from the mutant.

Now the gate is testing the broken artifact and reporting on the fixed one. That is a false green in the most dangerous possible direction, and it is this same wiring-versus-execution failure recursively eating the tool built to catch it.

## Wiring and execution are different guarantees

Neither substitutes for the other, and any change claiming to close a correspondence should say which one it delivers.

That a production value is _named_ is checkable at compile time and costs nothing. That it is _executed_ is only provable by breaking it and watching a live gate go red, which costs a rebuild and a run.

The same split governs consumers. Feeding a consumer a fixture proves it would notice a shape change. It does not prove the serializer you kept is the one actually reached on the wire. The first is cheap and belongs in every collapse. The second is expensive and is the guarantee people assume they have when they do not.

## What we refused to build

Three further instruments were proposed, measured, and turned down, each recording the evidence that would reverse the decision.

The most instructive refusal was a central registry file listing every seam and its counterpart. It reads like the obvious answer and it is a trap: the registry is a second artifact that must correspond to the code, which is the original problem with extra steps. Markers live at the seam, in the source, where the person editing the seam cannot avoid seeing them.

The second refusal worth naming is a phantom detector running by default. It found real things and it also produced a mixed worklist, which fails test (c). A gate that cries wolf trains people to ignore gates, and that cost outlives whatever the gate caught.

## The one thing this does not give you

Producer reachability. Nothing here proves that the serializer surviving a collapse is the one a live request actually reaches. Every other guarantee in this document is cheaper than that one, and we have not paid for it.

We write that down rather than letting it be inferred, because a document about gates reporting green over nothing has an obligation not to do the same thing itself.

## FAQ

### What does green over nothing mean?

A gate that passes because it executed no assertions, rather than because its assertions held. A test filter that matches an empty set, a suite that skipped, or a pattern check whose pattern can no longer match anything all produce the same output as a genuine pass: a check mark and a zero exit code.

### How do you detect a gate that reports green over nothing?

Read the executed count rather than the result symbol, and make the gate fail when it cannot measure that count. For a gate whose verdict is the absence of matches, commit a fixture the pattern must find on every run, so a check that has stopped seeing real code fails on its own test data first.

### Why is a hermetic test weaker than a live one here?

Because a hermetic test can inject a double and certify nothing about the shipped path. The doubles are usually correct and the assertions usually hold. What goes unchecked is whether the object under test is the object the dispatch seam actually calls.

### Why not keep a registry file of every seam and its counterpart?

Because the registry becomes a second artifact that must correspond to the code, which is the original problem with an extra layer. A marker placed at the seam in the source is seen by the person editing that seam. A registry in another directory is not.

### What is the mutation requirement?

Break the shipped thing on purpose, confirm the gate reddens, then restore. An assertion you have never reddened is not an assertion, it is a hope. Record the specific mutation beside the gate, because a gate whose mutation no longer bites looks exactly like one that works.

### Why does modification time matter when restoring a mutated file?

A checksum-verified restore proves the source is back, not the binary. If the restore preserves the original timestamp, an incremental build may recompile nothing and hand the next run a binary still built from the mutated source. The gate then tests the broken artifact while reporting on the fixed one.