Evaluation foundations
Held-out set
A private or untouched test set protected from optimisation.
Definition
A held-out set is deliberately withheld from model builders and evaluation participants until final testing.
Why it matters
It helps reduce benchmark gaming and accidental memorisation, although it does not eliminate contamination risk.
Related concepts
- Benchmark
A standardised test used to compare systems consistently.
- Test set
Data reserved for measuring performance after development.
- Benchmark contamination
Test material leaking into model training or optimisation.