Change one defensible choice, and the #1 changes

Not a bug, not a wrong number, not a disputed metric. A second reasonable way to aggregate the same released data — and across 44 such pairs, the top-ranked entry survives in only 19.

Every leaderboard makes choices before it can print a column. Which of several scoring variants to report. Whether to average repeated runs or quote one. How to combine subjects into an overall. Each choice is defensible, each has a defensible alternative, and the alternative is almost never shown.

What survives

For each pair — one board, one choice with a defensible alternative — the ranking was recomputed from the same released data under the other choice, and compared.

The pairs come from HELM classic (35), ProteinGym (4), AlpacaEval 2 (3), BigCodeBench and Open LLM v2 (one each), over fields ranging from six entries to 223.

The sharpest case changes no measurement at all

HELM runs some scenarios three times and publishes all three. Reporting one of those runs instead of their mean is not a different metric, a different model, or a different dataset — it is the same numbers, aggregated one defensible way instead of another. On two of five subjects, that alone reshuffles half the top ten.

Nothing was mismeasured. The ordering simply was not determined by the measurement in the first place.

Two corrections I had to make to my own note

An audit of other people's arithmetic earns nothing if its own goes unchecked, so both of these are stated rather than quietly fixed.

First, 30 of the 44 pairs compare six entries, not the eleven an earlier version claimed. Small fields churn more easily, so this matters.

Second, and more usefully: the same earlier note asserted those small-field pairs were inflating the top-five column. They are not. Full top-five survival is 6 of 30 among the six-entry pairs against 3 of 14 among the rest — 20% against 21%. The effect is real at every field size, and the convenient explanation for it was wrong.

What a leaderboard should print

Name the choice, and show one alternative. Not every alternative — one, chosen honestly, recomputed from the same data. If the ranking holds, that is a strong claim cheaply made. If it does not, the reader has learned the single most important thing about the table: that its order is a property of the aggregation as much as of the models.

A ranking that has never been recomputed under a different defensible choice has not been tested. It has been printed.