Mastery: connect the pieces

1. Idea — Pretraining teaches continuation; alignment training changes which continuations are preferred—helpful, harmless, honest behaviors under a policy—without magica…
2. Mechanism — Collect comparisons or ratings of candidate answers, fit a preference/reward model, then optimize the policy (PPO, DPO, and relatives) so preferred answers beco…
3. Controls — rater guidelines, preference mixture, KL / stay-close penalty, safety refusal policy, eval harm suite, over-refusal checks
4. Tradeoff — Stronger safety can increase over-refusals. Heavier RL can reduce diversity or degrade niche skills. More rater hours improve signal and raise cost and cultural…
5. Failures — Reward hacking; Sycophancy
6. Metrics — preference win rate vs baseline, harm pass rate, over-refusal rate, capability regression on core tasks
7. Ship rule — Ship the RLHF checkpoint only if harm suite severe fails=0, over-refusal ≤ baseline+2 pts, and MMLU-like smoke does not drop >1 pt.

Neighboring layers (retrieval, serving, policy, human review) still own what alignment / RLHF cannot on the general-purpose chat model. Do not ask this chapter’s dial to replace permissions or source truth.

Mastery is transfer: take alignment / RLHF into a second scenario that is not the general-purpose chat model and rebuild the checklist without copying sentences. If you can only recite this chapter’s examples for alignment / RLHF, you have memorized a story, not a model.

Also pin one numeric memory from this alignment / RLHF chapter: If raters prefer answer A over B in 62 of 100 pairs, the preference model should rank A higher on held-out pairs; a 50/50 split means the signal is noise for that slice. That number is not decoration; it is a template for how claims about alignment / RLHF on the general-purpose chat model should look in design docs. Scoped specifically to alignment / RLHF / general-purpose chat model / mastery-connect.

People confuse alignment / RLHF with neighboring buzzwords when debugging the general-purpose chat model. Before changing prompts, ask whether the broken stage was evidence gathering, the alignment / RLHF judgment itself, validation, or the product action. Fixing the wrong stage creates folklore (“we tried alignment / RLHF and it failed”) that blocks the next team on the general-purpose chat model. Scoped specifically to alignment / RLHF / general-purpose chat model / mastery-connect.

Write a five-line artifact for this page: goal, inputs, check, owner, stop rule. Invent one fluent failure that the check would catch. Keep details specific to alignment and rlhf rather than generic AI advice.