Training & post-training
Direct preference optimisation
A method that learns from preferred answers without a separate reward-model loop.
Also known as: DPO
Definition
Direct preference optimisation adjusts a model directly using pairs of preferred and rejected responses relative to a reference model.
Why it matters
DPO can simplify post-training, but outcomes still depend on preference-data quality and the chosen optimisation settings.
Related concepts
- Post-training
Training after pretraining to improve usefulness, reasoning and behaviour.
- Reinforcement learning from human feedback
Using human preferences to train a model toward more desirable responses.
- Preference data
Examples showing which of two or more model outputs is preferred.