Your leaderboard is not one-dimensional
A ranking is a column of names, and a column of names makes exactly one promise: that better is a direction. Sometimes it isn't — and unlike most things in evaluation, this one is testable.
We rank models the way we rank runners. Rank one beats rank two, which beats rank three, and the whole comparison reduces to a single line you can walk up. That reduction does enormous work, and it is almost never tested.
The test
Take the full matrix — every model against every test item, not the averaged column everyone quotes — and ask how many independent directions of variation actually live in it. Extract the eigenvalues of the correlation structure and compare each against a parallel-analysis null: the same matrix shape, with the structure destroyed by permutation. A dimension counts only if it beats what shuffled data produces by chance.
If the answer is one, the ranking is honest: models differ by a single quantity you can reasonably call capability. If the answer is two or three, the column has been flattening real structure — somewhere in that data, model A beats model B on one kind of problem and loses on another, consistently, and the average dissolved the distinction.
What public leaderboards actually contain
| Leaderboard | Dimensions surviving the permutation null | What the single ranking hides |
|---|---|---|
| MTEB (text embeddings) | k ≈ 6 | six largely independent competences averaged into one column |
| ProteinGym (variant effect) | k ≈ 3 | a second axis with a nameable chemical mechanism |
| SWE-bench (code agents) | k ≈ 2 | models good at different classes of repository task |
The ProteinGym case is the instructive one, because the second dimension was not noise — it had a mechanism you could name. The top-ranked model handled one chemical distinction poorly and a lower-ranked model handled it well. Two models sitting several places apart in the published order were, in truth, good at different things. Anyone picking the top of that list for a task living in the second dimension chose the wrong tool, with the leaderboard's full endorsement.
Why this is not a small point
Averaging is not neutral. It presumes the thing you are averaging is one thing. When it is two, the average lands somewhere between them, describing neither and recommending the wrong one — and it does so with more apparent authority than either honest sub-score would have carried.
This is also why saturated leaderboards feel unstable. When a ranking compresses several axes, small shifts in the item mix reweight those axes and the order rotates, which reads as progress and is actually recomposition.
What a leaderboard should print
One extra line under the ranking: how many dimensions this data actually has. If it is one, say so and rank away — you have earned the column. If it is more, publish the axes, because the useful advice was never “use the winner.” It was “use this one for that kind of problem, and that one for this kind” — a more helpful sentence than a rank has ever been able to say.
The test is cheap and runs on data you have already collected. The reason nobody runs it is that we never noticed we were assuming the answer.
Input: the model × item score matrix (not the averaged leaderboard column).
Method: eigenvalue decomposition of the correlation matrix, each component gated against a
permutation (parallel-analysis) null; count only the leading contiguous survivors.
Guard: heteroscedastic benchmarks inflate the null — check it, or you will read a false k=0.
Results: MTEB k≈6 · ProteinGym k≈3 · SWE-bench k≈2.
Tooling:
github.com/ipezygj/evalgate