Reference · Glossary
RLHF
Last updated
Aligning chat models after supervised fine-tuning when you can collect preference data and run reward-model / policy training loops.
#When to use
Aligning chat models after supervised fine-tuning when you can collect preference data and run reward-model / policy training loops.
#When not to
Fixing factual freshness (use RAG), or tiny style tweaks you can get with prompting or light SFT. RLHF is expensive and easy to overfit to rater quirks.
#Mental model
1. Collect pairwise preferences on model outputs
2. Train a reward model to score answers
3. Optimize the policy to raise reward without drifting too far from the base model
#Learn next
- Lesson: `alignment-and-rlhf`
- Related: alignment, fine-tuning