What is an eligible count and why log it next to a reject rate?

The eligible count is how many inputs a check was entitled to judge under its own filter, before any of them were accepted or rejected. A reject rate alone cannot separate a check that examined ten thousand inputs and correctly passed them all from a check whose filter is broken and examined nothing. Both report zero rejections. The eligible count separates them, and an eligible count equal to the entire population is a signal to open the filter and read it rather than a sign of full coverage.

How do you tell whether a self-improving system got better or found a flaw in its verifier?

Not from the reward, which rises the same way in both cases. Track instead what fraction of the system's recent proposals fall in regions where the gate has never rejected anything. That number is computed independently of reward and moves when the system starts operating where the gate has no record of ever firing. Legitimate new capability moves it too, so treat it as a reason to re-audit the gate rather than as an alarm.

What is a seeded-bad case?

An input constructed to be wrong on a check's own criterion, pushed through the production scoring path, with the verdict required to move. It is the cheapest available evidence that a gate can return red at all. A gate that has never rejected anything is not proven sound, it is untested, and from outside those two look identical.

How many seeded-bad cases does a check need before zero rejections means anything?

Enough that the bound is one you can live with. Zero misses in n seeded cases puts the miss rate at roughly 3/n by the rule of three, so ten seeded cases are consistent with a check that misses three in ten, and a hundred bring that to about three in a hundred. Choose the miss rate you are willing to carry and n falls out of it. Below that number the honest reading of zero rejections is unresolved rather than clean, and a gate replaced or re-elected before it clears the floor is being judged on noise.

Can a council of LLM judges that must agree unanimously be trusted without measurement?

No. Unanimity converts agreement into trust only if the judges fail independently, and LLM judges do not: their errors correlate, and published universal attack phrases transfer across judge models and pin every score at maximum, producing verdicts that are unanimous and wrong. A gate whose retries, escalation, and re-election all trigger on disagreement fails open under such an attack, because the attack removes the disagreement. Trust comes from measured human-agreement targets and seeded-bad cases pushed through the scoring path, not from the mechanism alone.

Zero Rejections and Nothing to Reject Are the Same Number

Two weeks ago I argued that a verification gate fails three ways: it can wrongly reject correct work, wrongly accept wrong work, or be bypassed by memorisation. Two readers replied with a fourth. They proposed different fourths, neither was wrong, and their answers turned out to be the same measurement approached from opposite ends.

Neither had seen the other’s comment.

The gate that cannot fail

One reader pointed out that all three of my failure modes assume a gate that discriminates and discriminates wrongly. Underneath them sits a gate that does not discriminate at all, because its scoring path is broken and every input comes back a pass.

That is not a false accept. A false accept still rejects something somewhere, so the reject rate wobbles. A broken scoring path produces a flat line, and a flat line reads as a stable suite.

Their evidence came from backtesting. A look-ahead leak took a result from +563% to +4,201% with every control silent, and their account of why is the part I keep rereading: no control objected because none of them could have.

The audit they proposed is cheap. Push an output you know is bad through the scoring path and require the number to drop. If it does not move, the gate is not passing your work. It is absent.

I had my own version of this in a Sudoku solver last spring. A pre-check was rejecting nothing across the whole corpus, which I read as evidence that no cheaper necessary condition existed, so I retired it as structurally ungateable. The reject-rate column supported that reading perfectly. What had actually happened was that the counter behind the predicate was counting a superset of what the technique filtered on, so the condition was satisfied on every step and the check admitted unconditionally. I fixed the counter and re-admitted the same predicate that afternoon at a 16.84% reject rate.

Their refinement is what makes this catchable without a known-bad case at all. A reject-rate column reports what came through. It says nothing about how many inputs the check was ever eligible to judge. A control whose eligible count equals the entire corpus is either genuinely universal or miscounting what it is allowed to look at, and those two read identically until somebody opens the filter.

Selection pressure

A second reader came at the same post from training dynamics, and proposed a different fourth: verifier error changes kind once the gate moves inside the loop.

On a leaderboard, a flawed test misprices a score and corrupts nothing downstream. The near-60% flawed-test rate that SWE-Bench ProMax cites in unsolved SWE-bench Verified instances is a measurement problem and stays one. Inside a self-improvement loop the same flaw becomes selection pressure. Every retained sample was retained for a reason, and where the reason is a gate flaw, the flaw is what the next round reinforces.

They also bounded it, which is the part I had not earned in the original post. The compounding holds only while the policy’s search can reach the flawed region. A gate that is wrong about inputs the model never proposes stays measurement noise indefinitely.

I pushed back on the word indefinitely. Reachability is a property of the current policy, and the policy is the thing training changes. A region that is unreachable early becomes reachable later, and nothing announces the crossing.

Their reply is the sharpest thing anyone said to me that week. They accepted the correction and then closed the door I had left open: the reward curve moves the same way whether the policy got better or found the flaw. So the trigger cannot be the reward.

That leaves an obvious question, and they did not answer it. If not the reward, then what?

Where the two arrive together

The first reader’s answer, written for a different problem three days earlier, is the trigger.

Track what fraction of the system’s proposals land in regions where the gate has never once rejected anything. Reward does not enter that calculation. The number moves when the policy starts operating somewhere the gate has no record of ever firing, which is what exploiting a flaw looks like from outside.

The first reached the quantity from the gate’s side, asking whether a check that never fires is doing any work. The second needed it from the policy’s side, asking whether the system has wandered somewhere the gate was never exercised. It is one number, and it answers both questions because both questions are about the same gap between what a check covers and what it has demonstrated.

What to log

Two numbers per check, and one derived from them.

Eligible count. How many inputs the check was entitled to judge under its own filter, recorded beside the result rather than inferred from it. Compute it from the filter the check actually applies, not from a proxy that looks equivalent. My solver predicate failed precisely because its counter and its filter selected different sets, and no amount of staring at the reject rate would have shown me that.

Last rejection of a seeded-bad case, and how many have run. The date the check last rejected an input built to be wrong on its own criterion, pushed through the production entry point rather than a unit-level call. A check with no such date is untested rather than sound. Record the count beside the date, because the count is what turns the date into a bound: zero misses in n seeded cases puts the miss rate at roughly 3/n by the rule of three, so ten of them are consistent with a check that misses three in ten. Choose the miss rate you are willing to carry and n falls out of it. Below that number the honest reading is unresolved rather than clean, which is this post’s title claim with an arithmetic attached.

Share of recent proposals in never-rejected regions. The derived number, and the re-audit trigger. Rising means the system is working where the gate has no track record.

Where this breaks

Legitimate new capability moves the third number as well. A policy that learns to solve a genuinely new class of problem correctly also lands where the gate has never fired. The measurement flags that you are operating outside audited territory, which is true in both cases and worth knowing in both cases, so it earns a re-audit rather than an alarm.

Clean traffic moves it too, and this is the confound that bites hardest outside a training loop. Point the same measurement at a review gate running over ordinary work and some regions never get rejected because nothing in them was ever wrong. At volume there are whole categories of boilerplate that should pass every time. The reject rate cannot separate that from a broken scoring path, and neither can the never-rejected share on its own.

The seeded stream is what separates them, and the reason is worth stating precisely: a seeded case is rejectable by construction, so its pass rate carries no cleanliness term at all. Real traffic passing tells you either that the gate works or that the work was fine. A seeded bad passing tells you one thing only. Route the seeded cases per region and a single seeded pass in a region that has never rejected anything stops being a suspicion and becomes a verdict.

A new gate has rejected nothing yet, and that is not the same condition as a gate that has been exercised for months without firing. Keep the two apart in whatever you report, or the cold-start case will train you to ignore the signal.

The hardest limit is one the first reader named themselves. Manufacturing a known-bad case is easy when the criterion is arithmetic, because you can inject a look-ahead and be certain it is wrong. For a qualitative check, the person writing the bad case reads the same specification the check was built from, so the failure both sides misread in the same way never gets written. The best partial answer either of us had was to separate the two authors: brief whoever writes the bad case on the outcome and keep the specification away from them. It is slower, it loses precision, and it still misses whatever nobody thought to be wrong about.

What I changed

The evidence method I use for migration work graded gate integrity on three questions, all of them about separation: could the author create or select the judging evidence, could it be altered afterward, did a separate person with real authority review it. A gate whose scoring path is broken passes all three. It has independent provenance, it was never relaxed, somebody competent signed off, and it returns a pass for everything it sees.

It now has a fourth question, reported like the other three with its own coverage: the date each gate last rejected a seeded-bad input, and the share of gates for which no such evidence exists. Where a criterion is qualitative and no bad case can be constructed, the gate renders as sensitivity-unevidenced rather than sound, which is the honest reading and also the uncomfortable one.

Both readers arrived at this from production systems rather than from theory. One from a backtest that reported 4,201% and meant nothing. The other from watching reward curves that look identical under improvement and under exploitation. Neither is named here because neither signed up to have a comment turned into a case study; the discussion itself is linked below and their words are their own. My contribution was a post with a gap in it, which turns out to be a reasonable way to find two people who had already thought about the thing you missed.

Update, 20 August 2026

Three days after this post went up, the third instrument arrived in production form.

A CTO published the design of a gate his team runs over millions of code and artefact review trajectories: a council of three or more LLM judges that must return the same verdict and the same score. Disagreement triggers retries. Only repeated retries escalate to a human, and repeated failures re-elect the council itself. The claim was explicit that the verdict rests on the mechanism rather than on measurement, and the falsifier was specific: show three judges agreeing unanimously and unanimously wrong.

The scenario was already in the literature. Short universal phrases, optimised against a surrogate judge, transfer to judge models the attacker never saw and pin scores at maximum regardless of the content underneath, and they work best in exactly the absolute-scoring mode a same-score requirement runs in (arXiv:2402.14016).

He conceded within the hour, and the concession named the mechanism more precisely than my comment had. The attack only inflates, so every unanimous error is a pass. Retries need disagreement to fire, and disagreement is the one signal the attack removes. The gate fails open, and unanimity is what hid it.

That is the flat gate from the top of this post, built deliberately and at scale, with agreement standing where the reject rate stood. Reward, reject rate, unanimity: three instruments, one defect. Each moves identically whether the system improved or the gate broke, so none of them can police the loop it runs. The tripwire is the one this post already logs: seeded-bad cases through the production scoring path, some of them wearing attack phrases, with the verdict required to move, and the share of traffic landing where the council has never rejected anything as the re-audit trigger.

What happened next is the part I did not expect. He priced a confound into the measurement I had just recommended: a region that has never been rejected is either a flat gate or genuinely clean traffic, and at his volume there are whole categories of boilerplate that should pass every time. Then the reader whose flat-gate comment opens this post arrived in that thread, having never met him, and answered it: a seeded case is rejectable by construction, so its pass rate carries no cleanliness term, which is what makes the two readings separable. His reply supplied the arithmetic this post’s title had been missing. Zero misses in n seeded cases bounds the miss rate at roughly 3/n, so below your chosen n a region reads unresolved rather than clean. Both refinements are folded into the two sections above rather than left here, because they belong to the instrument rather than to the news.

This post argued that two readers reached one measurement from opposite sides without having seen each other. They have now met, in a stranger’s comment section, on a gate one of them runs in production, and gone further together than either had alone.

None of the three is named here, for the reason given above. The exchange is public and their words are their own: the unanimous-council thread.

The original post and both replies are on LinkedIn: self-improving agents, grading themselves.

Related: Reading Isn’t Verifying on why gates that read artifacts miss what gates that execute catch, and this week’s dev roundup for the papers that started the argument.