A standard error of exactly zero

423 published records claim perfect certainty about a score that is neither 0 nor 1. A standard error can legitimately be zero only when every item agreed. These did not.

An error bar of exactly 0.0 beside a score of, say, 0.3125 is not a small error bar. It is a statement that the measurement carries no uncertainty at all — that rerunning on a different sample of items would return the identical number. For a score strictly between 0 and 1, that cannot be true.

How many, and where

Sampling the latest results file for 420 models from the leaderboard's own results repository, with no fetch failures: 107 models — 25.5% — publish at least one such record, 423 records in total. Twenty-three of those models have all seven of their MATH subtasks affected. A further 51 aggregate rows inherit the zero from the subtasks beneath them.

Every one of the 423 is a leaderboard_math exact_match figure. No other task family produces a single one.

The control is the reason this is a finding

An audit claiming a benchmark's error bars are wrong has to rule out the obvious alternative first: that the auditor reconstructed them wrongly. So the same formula — sqrt(p(1-p)/(n-1)) — was applied to every task family in the same files, and asked to reproduce the published standard errors exactly.

Thirteen thousand nine hundred and four rows reproduce to the digit, and not one of them does so under the alternative denominator sqrt(p(1-p)/n) — so the formula is pinned, not fitted. The reconstruction is right everywhere it can be checked. MATH is where it stops agreeing, and that is the finding rather than the method's failure.

It appears to have stopped

Grouping every dated result file by run month, the affected share runs between 17% and 37% from mid-2024 through January 2025, falls to 11.7% in February 2025, and reaches 0 of 23 runs in March 2025. The last affected run is dated 2025-02-06. If the rate were still around 25%, seeing zero in 23 runs would be roughly a 0.1% event — so this reads as a real change rather than thin data, though 23 runs is not many and I would treat the date as approximate.

That is good news with a tail. The fix, if it was one, does not travel backwards: the 423 published records still carry 0.0. Anything consuming those files today — a meta-analysis, a model-selection script, a paper's error bars, a filter that drops rows with high uncertainty — reads them as exact.

Reported, not just published

This was filed with the harness maintainers as lm-evaluation-harness#3966, with the widened sample and the month-by-month breakdown, before it appeared here. An audit that only ever becomes a blog post is a complaint; the useful version goes to the people who can fix it.

What to do with a zero

If you consume published leaderboard files, treat stderr == 0 on a score in the open interval as missing, not certain. It is a one-line filter and it is the difference between propagating an unknown and propagating a false precision. If you publish them, recompute rather than pass through, and emit no value at all where you cannot compute one — a blank is honest, and a zero is a claim.