Trace a worked example

Goal: improve the general-purpose chat model using alignment / RLHF without breaking protected slices.

Mechanism reminder for alignment / RLHF: Collect comparisons or ratings of candidate answers, fit a preference/reward model, then optimize the policy (PPO, DPO, and relatives) so preferred answers become more likely while staying near the base model.

Baseline and shock: If raters prefer answer A over B in 62 of 100 pairs, the preference model should rank A higher on held-out pairs; a 50/50 split means the signal is noise for that slice.

Tradeoff in play for alignment / RLHF: Stronger safety can increase over-refusals. Heavier RL can reduce diversity or degrade niche skills. More rater hours improve signal and raise cost and cultural bias risk.

Ship decision: Ship the RLHF checkpoint only if harm suite severe fails=0, over-refusal ≤ baseline+2 pts, and MMLU-like smoke does not drop >1 pt.

Rollback triggers for alignment / RLHF must cite preference win rate vs baseline, harm pass rate, over-refusal rate, capability regression on core tasks. If you cannot name a tolerated regression on the general-purpose chat model, do not promote the change.

Keep the alignment / RLHF decision record beside the general-purpose chat model code paths that implement it. Future you will not remember why a default exists unless the evidence is linked from the config for alignment / RLHF.

Also pin one numeric memory from this alignment / RLHF chapter: If raters prefer answer A over B in 62 of 100 pairs, the preference model should rank A higher on held-out pairs; a 50/50 split means the signal is noise for that slice. That number is not decoration; it is a template for how claims about alignment / RLHF on the general-purpose chat model should look in design docs. Scoped specifically to alignment / RLHF / general-purpose chat model / worked-trace.

People confuse alignment / RLHF with neighboring buzzwords when debugging the general-purpose chat model. Before changing prompts, ask whether the broken stage was evidence gathering, the alignment / RLHF judgment itself, validation, or the product action. Fixing the wrong stage creates folklore (“we tried alignment / RLHF and it failed”) that blocks the next team on the general-purpose chat model. Scoped specifically to alignment / RLHF / general-purpose chat model / worked-trace.

Write a five-line artifact for this page: goal, inputs, check, owner, stop rule. Invent one fluent failure that the check would catch. Keep details specific to alignment and rlhf rather than generic AI advice.

For alignment and rlhf, add acceptance test 1: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For alignment and rlhf, add acceptance test 2: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For alignment and rlhf, add acceptance test 3: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For alignment and rlhf, add acceptance test 4: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For alignment and rlhf, add acceptance test 5: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For alignment and rlhf, add acceptance test 6: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For alignment and rlhf, add acceptance test 7: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For alignment and rlhf, add acceptance test 8: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For alignment and rlhf, add acceptance test 9: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For alignment and rlhf, add acceptance test 10: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For alignment and rlhf, add acceptance test 11: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For alignment and rlhf, add acceptance test 12: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For alignment and rlhf, add acceptance test 13: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For alignment and rlhf, add acceptance test 14: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For alignment and rlhf, add acceptance test 15: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For alignment and rlhf, add acceptance test 16: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For alignment and rlhf, add acceptance test 17: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For alignment and rlhf, add acceptance test 18: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For alignment and rlhf, add acceptance test 19: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For alignment and rlhf, add acceptance test 20: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For alignment and rlhf, add acceptance test 21: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For alignment and rlhf, add acceptance test 22: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For alignment and rlhf, add acceptance test 23: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For alignment and rlhf, add acceptance test 24: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For alignment and rlhf, add acceptance test 25: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For alignment and rlhf, add acceptance test 26: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For alignment and rlhf, add acceptance test 27: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For alignment and rlhf, add acceptance test 28: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For alignment and rlhf, add acceptance test 29: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For alignment and rlhf, add acceptance test 30: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.