The announced #1 is probably not the best model
Not because anyone cheated, mismeasured, or overfit. Because of what the word winner means when the race is close and every time is measured with error.
Every other audit on this site names something a board could fix. This one does not. It is a property of ranking itself, and it applies to a leaderboard that is perfectly run.
Why the winner is the luckiest, not the best
Each model's published score is its true ability plus a measurement error — a different test split, a flaky harness, a lucky seed. Now sort by that sum and crown the top. Sorting does not separate the two parts; it picks whoever has the largest total. Among models close together in ability, the one that rises is disproportionately the one whose error happened to be positive.
Two consequences follow, and both are measurable. The announced leader is often not the true leader. And the winning score is inflated — not by the winner's error alone, but by the expected maximum of all the errors that were competing, which grows with the number of entrants.
What it measures out to
Seven public leaderboards, each simulated 20,000 times by resampling every model's score from its own standard error and re-running the ranking. "P(wrong #1)" is how often the announced leader failed to be the true leader. "Inflation" is how many points the winning score overstates the winner's true ability.
| Leaderboard | Gap ÷ SE | P(wrong #1) | Inflation |
|---|---|---|---|
| SWE-bench Verified (180 models) | 0.00 | 72.1% | +2.41 |
| SWE-bench Multimodal (22) | 0.00 | 69.4% | +2.28 |
| SWE-bench bash-only (48) | 0.53 | 64.3% | +2.36 |
| SWE-bench Lite (84) | 0.12 | 59.6% | +2.66 |
| AlpacaEval 2 (222) | 0.19 | 53.1% | +0.97 |
| Aider polyglot (69) | 0.60 | 43.3% | +1.81 |
| SWE-bench Test (24) | 8.03 | 0.0% | +0.00 |
On five of the seven, the announced #1 is more likely wrong than right. On SWE-bench Verified there is only an 80.2% chance the true best model is anywhere in the printed top five. On SWE-bench Verified and Multimodal the top two are not merely close — the published gap is exactly zero, and the order between them was decided by whichever row the sort happened to place first.
One number predicts all of it
Look at the first column. The leader's margin, expressed in units of the leader's own standard error, orders the table almost perfectly: Spearman ρ = −0.88 against P(wrong #1), exact permutation p = 0.015 over all 5,040 orderings of these seven boards. A board separated by eight standard errors is safe. A board separated by a fifth of one is a coin flip wearing a crown.
That ratio costs nothing to compute and no leaderboard prints it.
The control that returns zero
A method that finds a curse everywhere has found nothing. SWE-bench Test was not planted as a control — it is simply the board whose leader is 8.03 standard errors clear — and the same simulator returns 0.0% and +0.00 on it. The machinery is capable of saying "this #1 is real," and on that board it does.
The simulator was also checked against closed form before it was pointed at any real board: zero noise returns exactly zero; twenty identical models at σ = 5 return +9.32 against an analytic +9.35; two identical models return +2.88 against σ/√π = 2.82. SWE-bench Multilingual was excluded because its denominator could not be pinned down, not because of what it showed.
What a leaderboard should print
Two lines, neither of which requires new data. The leader's gap in units of its own standard error — below about one, say plainly that the top is unresolved. And a shrunk winning score, discounted for the selection that produced it, so the number that gets quoted in a press release is the one the model will reproduce next month.
Until then, the honest reading of a close leaderboard is not "model X is best." It is "these five are indistinguishable, and one of them got to stand at the front."
Input: each board's published per-model scores and n (SE recovered by GRIM or read from per-instance files).
Method: 20,000 resamples per board — draw each model's score from its own SE, re-rank, count how
often the announced leader loses, and average the winner's overstatement.
Controls: closed-form validation before use; one unplanted board returning 0.0%.
Excluded: SWE-bench Multilingual — n could not be pinned.
Run it yourself:
pip install eval-integrity, thenfrom evalgate.leaderboard import selection_audit — or call check_winners_cursefrom an agent over MCP.
github.com/ipezygj/evalgate