Evaluation foundations
Benchmark
A standardised test used to compare systems consistently.
Definition
A benchmark combines a defined task or dataset, instructions, scoring method and test conditions so multiple models or systems can be compared on the same basis.
Why it matters
It is useful for screening and comparison, but it rarely proves that a system is safe or effective in your organisation.
Related concepts
- Eval
A structured test of whether an AI system meets a requirement.
- Evaluation suite
A coordinated collection of evals covering different capabilities and risks.
- Evaluation harness
The software and rules used to run and score an eval.
- Leaderboard
A ranked table of systems evaluated under stated conditions.