Evaluation foundations
Evaluation harness
The software and rules used to run and score an eval.
Definition
A harness supplies prompts, tools and data to the system under test, records its trajectory and applies the benchmark's scorers.
Why it matters
Different harnesses can produce materially different results, particularly for agents and tool-using systems.
Related concepts
- Benchmark
A standardised test used to compare systems consistently.
- System under test
The complete configuration being evaluated—not only the model.
- AI agent
A system that plans and takes actions using models, tools and state.
- Reproducibility
The ability to repeat an evaluation under documented conditions.