Evaluation foundations
System under test
The complete configuration being evaluated—not only the model.
Also known as: SUT
Definition
The system under test includes the model version, prompts, retrieval, tools, data, guardrails, permissions, memory and orchestration used in the application.
Why it matters
The same model can perform very differently when any part of the surrounding system changes.
Related concepts
- Evaluation harness
The software and rules used to run and score an eval.
- Retrieval-augmented generation
Generating answers using information retrieved at request time.
- Guardrail
A technical or procedural control that constrains system behaviour.
- Reproducibility
The ability to repeat an evaluation under documented conditions.