Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias
Factual evidence
What the source reports
Researchers evaluated 21 LLM-as-a-Judge models across three benchmarks, finding exact-match metrics systematically overstate judge accuracy.
Inspect the evidence
- Inclusion basis
- Enterprise AI
- Publisher and source type
- arXiv cs.CL — Computation and Language · RESEARCH
- Published by source
- 5 November 2026
- Collected by OneBench
- 6 Nov 2026, 02:01 UK
Stored source excerpt
arXiv:2606.19544v2 Announce Type: replace Abstract: LLM-as-a-Judge has become the dominant evaluation paradigm for language models, but judge validation in practice relies on exact-match agreement, a…
Short excerpt from the collected text, not the full source. Use the source link to read it in context.
The factual summary is a OneBench synthesis, not a quotation or independent verification. Collection time is not publication time. Open the source for its full context; related reporting can share the same underlying announcement.
OneBench interpretation
Institutional assessment
So what
Automated LLM evaluation pipelines may overstate model accuracy and reliability by relying on unadjusted exact-match metrics.
Do what
Review internal model validation benchmarks with the team responsible for model risk management to account for chance agreement.