Reference · Glossary
Direct Preference Optimization (DPO)
Last updated
A simpler alternative to full RLHF for aligning a model to human preferences — instead of training a separate reward model and running reinforcement learning, DPO trains directly on pairs of (preferred answer, rejected answer) using a single, more stable training step.
#When to use
Post-training a model on preference data (e.g. "responders liked A better than B") when you want most of RLHF's benefit with a simpler, cheaper training pipeline.
#When not to
Cases needing the reward model itself as a reusable component (e.g. to score arbitrary new outputs at scale) — DPO skips building that reward model, which is a tradeoff, not just a shortcut.
#Example
Feeding a model 10,000 pairs of "helpful, on-brand reply" vs. "technically correct but curt reply" and running DPO to shift its default tone — without the extra reward-model-plus-RL pipeline RLHF requires.