Training & post-training
Reinforcement learning from human feedback
Using human preferences to train a model toward more desirable responses.
Also known as: RLHF
Definition
RLHF collects human comparisons or ratings, learns a reward signal and uses reinforcement learning to optimise model responses.
Why it matters
It can improve helpfulness and safety, while inheriting annotator bias, ambiguous policy choices and reward-model limitations.
Related concepts
- Reinforcement learning from feedback
An umbrella for reinforcement learning guided by preference or quality feedback.
- Reward model
A model estimating how desirable another model's output or action is.
- Preference data
Examples showing which of two or more model outputs is preferred.