Training & post-training
Reward model
A model estimating how desirable another model's output or action is.
Definition
A reward model learns from preference data and returns a score used to guide reinforcement learning or rank candidate outputs.
Why it matters
If the reward is incomplete or gameable, the trained model may optimise the score rather than the intended business or safety outcome.
Related concepts
- Reinforcement learning from human feedback
Using human preferences to train a model toward more desirable responses.
- Preference data
Examples showing which of two or more model outputs is preferred.
- Reward hacking
Optimising a measured reward without achieving the intended objective.