Training & post-training
Model distillation
Training a smaller model to reproduce behaviours learned from a larger model.
Definition
Distillation uses outputs or internal signals from a teacher model to train a more compact student model.
Why it matters
It can reduce cost and latency, but the student may inherit teacher errors while losing capability on rare or complex cases.
Related concepts
- Small language model
A relatively compact language model optimised for focused or efficient deployment.
- Fine-tuning
Additional training that adapts a model to a task, domain or behaviour.
- Synthetic data
Artificially generated examples used instead of or alongside observed data.