SOTA or Luck? The Winner's Curse in LLM Leaderboards
Factual evidence
What the source reports
Researchers demonstrate that top public LLM leaderboards systematically overstate performance due to statistical selection bias in noisy scores.
Inspect the evidence
- Inclusion basis
- Enterprise AI
- Publisher and source type
- arXiv cs.LG — Machine Learning · RESEARCH
- Published by source
- 18 November 2026
- Collected by OneBench
- 19 Nov 2026, 02:01 UK
- Original headline
- SOTA or Luck? The Winner's Curse in LLM Leaderboards ↗
Stored source excerpt
arXiv:2611.22227v1 Announce Type: new Abstract: Public LLM leaderboards guide model evaluation and adoption, yet the reported performance of top-ranked models can be systematically overstated. This…
Short excerpt from the collected text, not the full source. Use the source link to read it in context.
The factual summary is a OneBench synthesis, not a quotation or independent verification. Collection time is not publication time. Open the source for its full context; related reporting can share the same underlying announcement.
OneBench interpretation
Institutional assessment
So what
Vendor benchmark rankings likely reflect statistical luck rather than superior underlying capabilities, undermining public leaderboard credibility.
Do what
Review internal model selection methodology to ensure vendor claims rely on custom benchmark evaluation rather than public leaderboards.