Necessary but Not Sufficient: Temperature Control and Reproducibility in LLM-as-Judge Safety Evaluations
Factual evidence
What the source reports
Research demonstrates setting temperature to zero in LLM-as-judge safety evaluators fails to guarantee deterministic results.
Inspect the evidence
- Inclusion basis
- Enterprise AI
- Publisher and source type
- arXiv cs.LG — Machine Learning · RESEARCH
- Published by source
- 28 October 2026
- Collected by OneBench
- 29 Oct 2026, 02:02 UK
Stored source excerpt
arXiv:2606.26185v2 Announce Type: replace Abstract: LLM-as-judge ("grader") components are now standard in evaluation harnesses, including safety evaluations where a pass/fail verdict may gate downstream deployment…
Short excerpt from the collected text, not the full source. Use the source link to read it in context.
The factual summary is a OneBench synthesis, not a quotation or independent verification. Collection time is not publication time. Open the source for its full context; related reporting can share the same underlying announcement.
OneBench interpretation
Institutional assessment
So what
Non-deterministic LLM-as-judge evaluations undermine automated safety gating and compliance reporting for regulated financial model deployments.
Do what
Review model risk management frameworks to ensure automated evaluation harnesses account for non-determinism in zero-temperature judge models.