Tutorials · Chapter C (3/4) · ~9 min

Alignment and RLHF

Try it → see it → read → next

Alignment ≈ steer the model toward helpful, honest, harmless — not raw internet autocomplete.

Try yourself

Playground

Be the preference labeler

RLHF starts with humans ranking replies. Pick the better response.

User: How do I make a bomb?

Which reply should the model prefer?

Recap

What you just did

You saw how skewed data skews outcomes. Alignment is a deliberate push against “whatever the web rewarded.”

Teach

How it works

Common recipe (RLHF = Reinforcement Learning from Human Feedback):

Train / start from a big model.
Humans rank replies (better / worse).
Train a preference model on those ranks.
Optimize the chat model to prefer higher-ranked styles.

Other methods exist (constitutional AI, RLAIF). The point: human (or AI) preferences reshape behavior after raw next-token training.

Use it

When you'd use this

Interpreting why a model refuses some asks.
Product debates: “Why won’t it give medical dosages?”

Watch out

Alignment isn’t perfect. Jailbreaks exist. Refusals can also be over-cautious. Values embedded are human choices — not physics.

Try next

Try this next

Ask a polite disallowed question (e.g. “how do I pick a lock on my own door for legitimate reasons?”) and notice how the model hedges — that’s policy + alignment.