Tutorials · Chapter C (3/4) · ~8 min

Evals and benchmarks

Try it → see it → read → next

If you can’t measure it, you’re just demo-ing vibes.

Try yourself

Playground

Eval suite builder

Build 3 support-bot questions. Cherry-picking flips the leaderboard — unlock an honest suite.

Choose up to 3 eval questions

Leaderboard (easy-only — Model A looks best)

Model A 97% · Model B 95% → leader A

Reply politely to ‘hi’: A 98 / B 96
State store hours (easy FAQ): A 95 / B 93

Recap

What you just did

EvalSuiteBuilder let you compare models on a suite. Cherry-picking easy items flipped the leaderboard; an honest suite showed which model wins on real tasks.

Teach

How it works

Benchmark — shared test (math questions, coding tasks, safety prompts).
Eval harness — your own tests for your product (billing FAQ accuracy, tone, latency).
Metrics — accuracy, pass@k, human preference win-rate, toxicity rate.

Leaderboards help compare research. Your evals decide ship/no-ship.

Use it

When you'd use this

Before switching models in production.
After a prompt change — re-run the same 50 golden questions.

Watch out

Models can be trained on benchmark leakage. High MMLU ≠ good at your hospital’s triage policy. Always keep private, realistic eval sets.

Try next

Try this next

Write 5 golden questions for a bot you care about. That’s a micro-eval set.