Scoring & evidence
Inter-judge agreement
How consistently two or more graders assess the same outputs.
Definition
Inter-judge agreement measures whether human or model graders reach similar conclusions using the same criteria.
Why it matters
Low agreement suggests that the task, rubric or scoring process may be too ambiguous for confident decisions.
Related concepts
- Rubric
Explicit criteria describing what a good answer or deliverable must contain.
- LLM-as-judge
Using a language model to grade another system's output.
- Reproducibility
The ability to repeat an evaluation under documented conditions.