A fixed answer outranks most of this leaderboard

Answer D to every question. Read nothing, know nothing, run no model. That scores 0.4100 on this published table — above the board median of 0.2633, and above 22 of the 29 ranked runs.

The table is real, the arithmetic behind it is correct, and this is not a story about anyone making a mistake. It is a story about what a ranking measures when the thing being ranked is barely above the floor.

Where 0.4100 comes from

Multiple-choice answer keys are not uniform. They are written by people, and people do not distribute correct options evenly. Recovering this key item by item from the runs' own correct predictions — then confirming it against HELM's published gold references on 1,926 items with zero mismatches — gives a key in which D is correct on 41 of the 100 test items.

So the constant answerer scores 0.4100. It is not a trick or an exploit; it is what the key hands to anyone who guesses the same letter every time.

Testing each paired gap by bootstrap over items, the fixed answer beats 22 of the 29 runs by a margin whose 95% interval excludes zero. Eleven runs score below what a coin flip over four options would earn. Only the top of this table is measuring subject knowledge; the rest of the ordering is measuring something else.

The bound that makes it a finding rather than a headline

The obvious next sentence — "MMLU is broken" — is not supported, and the same measurement says so. The identical key was recovered independently on Open LLM Leaderboard v1, which runs MMLU through a different harness and publishes the gold index directly. It puts the fixed answer at 0.4100 on this subject too, agreeing to four decimals.

But there, against 152 later models with a median of 0.4600, that same fixed answer beats only 20 of them — 13%, not 76%. Across 59 Open LLM tasks it beats a median of 5.9% of the board and never more than 17.8%, and on none of them does the board median fall below it.

That is the whole result in one line. The skewed key is a property of the benchmark. Being beaten by it is a property of the roster. HELM classic's table is a snapshot of 2022-era models on a hard subject, and most of them were near the floor — which is exactly when a constant baseline stops being a joke and starts being a ranking.

What was checked before anything was questioned

An audit that cannot reproduce the numbers it criticises has no standing. Before questioning anything, every figure was rebuilt from HELM's own per-instance files: 297 runs across 8 scenarios — MMLU, BoolQ, BLiMP, CommonSense, BBQ, LegalSupport, LSAT, TruthfulQA — with median and maximum error 0.000000, and 0 of 297 exceeding 0.0005. On the second board, 614 published figures reproduced to a maximum error of 0.00000000. HELM's artifacts are sound and this audit says nothing against their arithmetic.

Three reconstruction choices could each have produced this finding on their own, so each wrong one was priced rather than assumed. Reading a metric named in the summary but absent per-instance shifts results by a median 0.753. Taking one trial instead of the published three-trial mean costs a median 0.0133 and up to 0.1467. Skipping the split filter quietly mixes 11 validation items into a 100-item test set. Any of those would have manufactured a scandal out of a bookkeeping error.

What a leaderboard should print

One row, above the models: the best constant baseline for this task. Not chance — the actual best fixed answer under the actual key. Every entry below that line is not measuring the skill the table is named after, and a reader deserves to see where the line falls before reading the ordering.

It costs one pass over the answer key, and it turns a ranking into a ranking with a floor.