Vendor benchmark scores average 6.4% above independent runs across 611 matched pairs — median gap just 1.2%
whatmodel.sh publishes the data and code behind a maker-vs-independent comparison that locates the gap in setup-sensitive evals.
An open-source leaderboard matched 611 maker-reported benchmark scores against independent runs of the same model on the same test and found labs average 6.4% higher — with the gap concentrated exactly where evaluation setup decides the outcome.
What shipped
The operator of whatmodel.sh, an open-source model leaderboard, published an interactive analysis at whatmodel.sh/claims comparing how labs score their own models against how independent parties score the same model on the same test. From 1,629 published benchmark results, the author matched 611 pairs covering 180 models on 103 tests. The dataset is CC BY-SA 4.0 and the code MIT, with each pair's sources listed in the table.
For each pair the author computed (lab score − independent score) / independent score × 100, equally weighted, ties and negative gaps included, and the average is +6.4%. The claims page adds context:
- The median percentage gap is +1.2%; the mean's 95% interval (resampling whole models) is 3.8%–9.5%. A heavy right tail drives the headline — the largest pair gap is +57.1 points (Molmo 7B-D, ChartQA, 84.1 vs 27.0).
- When scores differ: 375 lab higher, 216 independent higher, 20 exact ties — labs are higher 63% of the time (95% interval 59%–68%).
- The most revealing control: where a rival lab and a neutral tester both measured the same model on the same test (272 cases), the rival is lower 131 times and higher 130 times, median gap 0.0. Independent testers do not systematically disagree with each other; they disagree with makers.
- Against neutral testers specifically (leaderboards, Epoch AI, papers that don't release a model), the maker is higher 56% of 236 pairs; against rival labs' release documents, 63% of 446.
Gaps are reported in each test's own scale (percentage points), not in whatmodel's fitted IRT score, and only accuracy-type tests are included — arena-style ratings have no maker-reported twin to compare.
Why it matters
The 6.4% mean is not the interesting result — where the gap lives is. From the claims page's by-test table, the tests with the largest median gaps are exactly the ones whose scores depend on evaluation configuration:
| Test | Pairs | Median gap | % maker higher |
|---|---|---|---|
| SWE-bench Verified | 13 | +9.0 pts | 92% |
| MBPP | 13 | +9.0 pts | 77% |
| AIME 2024 | 14 | +3.8 pts | 86% |
| GSM8K | 19 | +3.3 pts | 84% |
| MMLU-Pro | 29 | +2.0 pts | 69% |
| GPQA Diamond | 51 | +1.1 pts | 67% |
| MMLU | 20 | 0.0 | 50% |
| HumanEval | 34 | 0.0 | 47% |
| ARC Challenge | 14 | −0.6 | 50% |
| τ²-bench Telecom | 18 | −1.3 | 39% |
Fixed-format tests where the harness barely matters (MMLU, HumanEval, WinoGrande, ARC Challenge) run flat or slightly negative. Agentic and reasoning evals — where reasoning effort, agent harness, few-shot prompts, and test subset/version all shift scores — run 2–9 points higher in maker hands. That matches the plausible mechanism: a wide configuration space plus the freedom to stop at the best run. One HN commenter reading the same page suggested a gentler mechanism — "a selection bias towards stopping when the score is on the higher side" — rather than deliberate inflation (HN discussion).
For anyone comparing or reproducing model numbers, the practical takeaways:
- Expect maker scores to be the favorable envelope. On coding and math evals, plan for independent replication to land several points lower; average the independent measurements when you need a baseline — the rival-vs-neutral control (median 0.0) shows they agree with each other.
- Pin the configuration, not just the model+benchmark pair. ChartQA-level gaps of 40+ points between maker and independent runs are configuration, not model quality. Publish the harness, reasoning-effort setting, and test version alongside any score you claim.
- Prefer the median. The 6.4% mean is a tail statistic; the typical pair is nearly flat. A single dramatic gap (Molmo 7B-D on ChartQA) is not evidence of systematic inflation across a lab, nor is a flat MMLU a sign a vendor is honest — the two are different questions.
The whatmodel.sh method page already hedges: maker-reported numbers are accepted like any other but carry an unverified mark, since no one else has checked them. This analysis turns that caveat into a distribution.
Caveats
- The analysis is self-published on a personal site and GitHub repository — no paper, no peer review — and we found no independent replication. Every statistic above is as reported on the claims page.
- Selection bias (author-stated): labs choose which tests to publish, so the 611 pairs are not a random sample — which is itself part of the effect being measured.
- No control for settings: the data cannot account for hardware, decoding parameters, harness version, or checkpoint version. The author explicitly concludes "gaps aren't all inflation."
- The relative-difference metric is asymmetric under multiplicative inversion — a score 1.2× the baseline registers as +20% while its inverse (0.83×) registers as −16.7% — so equally large over- and under-gaps do not cancel in the mean; pairs with a zero independent baseline are excluded entirely.
- Per-maker samples are small and volatile — Ai2 shows a +8.8-point median over 14 pairs and BigScience −11.9 over 20; the author labels these rows "rough."
- The original Reddit discussion thread blocks direct fetching, and its comment section could not be read for this writeup; the analysis itself is the primary source.