Eval for everyone
Mastery: pack a personal eval kit
You can evaluate an AI feature when you can define its promise, test representative behavior, explain failures, and make an owned release decision.
1Learn the idea
Read
Faded practice: review a weak kit
A teammate submits:
“We tried five science questions. Four answers looked good, so the model is 80% accurate.”
Repair it in order:
- Name the five case IDs and why they represent the job.
- Replace “looked good” with rubric scores and reviewer evidence.
- Identify the failed case and whether it crossed a gate.
- State who approved the labels.
- Narrow “80% accurate” to “4 of 5 cases met this rubric.”
- Add untested categories and the next evaluation step.
Read
Misconceptions to catch
- “More cases automatically means better evaluation.” Coverage and label quality matter.
- “An average can absorb one safety miss.” Gates exist precisely so it cannot.
- “The model grader is objective.” It must be calibrated against qualified humans.
- “A pass proves general reliability.” It proves performance on a named set under a named configuration.
Go deeper
Before you start
Why this matters
Keep these six artifacts together:
- Promise: user, job, authoritative inputs, and prohibited behavior.
- Cases: IDs, inputs, categories, and source versions.
- Labels: required facts, forbidden claims, and expected refusal or clarification.
- Rubric: separate criteria plus non-negotiable gates.
- Comparison: baseline and candidate scores with pass-to-fail transitions.
- Decision: owners, known gaps, stop rule, and rollback.
Related lessons
Check your understanding
Page assessment
Answer from memory. Completion is saved from this evidence, not from opening the next page.
All responses are required.