Eleven public case files
The original case-file series: judges, base rates, rank stability, hidden dimensions.
Eleven audits of AI benchmarks and leaderboards from released data — ten overstated, one certified
as holding. Now with a deposited rank-stability study
(doi:10.5281/zenodo.22541065).
see the case files →
Full write-ups on the front page.
Kaggle · 653 leaderboards · two test slices
Neighbouring leaderboard ranks are a coin flip. Measured, then replicated on 653 boards.
Kaggle scores every competition on two disjoint test slices, so rank noise is observed, not modelled.
The August measurement over 19 competitions replicated on an independently built dataset of 1,157
finished boards, method file unchanged and pre-registered: pairs closer than one local movement keep
their order in 55.1% of 2.2M pairs, pairs further apart in 90.8% of 7.3M; a shuffled control stays at
50%. 58 boards invert and are kept in every figure.
replicated, 19 → 653 boards
Report, pre-registration, code and per-board results:
doi:10.5281/zenodo.23017379 · data: Kaggle Shakeup Atlas (G. Mamarin).