Check the number before you publish it
Four of the cheapest checks that catch an overstated eval claim, free and in-browser — nothing is sent anywhere. The same checks behind the case studies and the open-source evalgate.
1 · Does your subset win survive correction?
Report the subset/metric/checkpoint where a model looks best and you’re reporting the max of many noisy tests. Correct for how many you could have picked.
2 · Is the judge winning, or just biased?
Count the verdicts a side won — the longer answer, the first-listed one, the judge’s own family — and test it against chance.
3 · Does one point flip your slope?
Paste your x y points (one per line). Fits a power-law exponent on log-log axes, then refits leaving each point out — and flags if dropping one crosses your threshold.
4 · Can your test set tell the models apart?
Two models on a fixed test set. This gives the two-proportion significance of the gap and the minimum detectable effect — the smallest gap this sample could reliably see. Below the MDE, the ranking is a coin flip.
5 · What does your precision become in deployment?
A benchmark measures on a roughly balanced set. Deployment usually isn't. Precision is a property of the model and how often the target actually occurs — so give the operating point you'll ship (not an AUC, which doesn't pin down a threshold) and the prevalence you'll really face.
6 · Is your #1 the best model, or the luckiest?
Sorting selects on score, and a score is ability plus measurement error — so among close models the one that rises is disproportionately the one whose error pointed up. Paste the published scores (comma-separated, any units) and the standard error, in the same units. Nothing else is needed, so this works on someone else's board.
7 · What is the floor your benchmark is read against?
Multiple-choice answer keys are written by people, and people do not spread the correct option evenly — so the floor is not one-over-the-options. Paste the correct label for each item (letters, words, anything, separated by commas or spaces) and, if you have them, the published accuracies as 0–1.
8 · Do your published error bars survive recomputation?
A standard error is implied by the score and the number of items, so it can be checked rather than trusted. Paste one row per line as label, score, n, published stderr — score as 0–1. A stderr of exactly 0 next to a score that is neither 0 nor 1 says a rerun would return the identical number.
These run the same math as evalgate. They’re the cheap DIY version — the value of an audit is running all of them, adversarially, on a number you’re too close to.