Eval for everyone
Re-run evals after every prompt change
A prompt change is a new candidate release: it can repair one case and quietly break another.
1Learn the idea
Read
A simple regression routine
- Freeze questions, labels, rubric, and source versions.
- Save the baseline outputs and scores.
- Change one major variable: prompt, model, retrieval source, or setting.
- Run the same set.
- Review every pass-to-fail transition and every safety case.
- Release only if the named gates still pass.
Changing the set and the prompt together destroys the comparison. Add newly discovered failures to the regression set, but report them separately until both candidates have run on the expanded set.
Read
Guided practice
A shorter prompt reduces average answer length and changes results from 9/12 to 10/12. One previous safety case now fails. Complete the decision: “The candidate ___ release because ___.” Suggested: “does not pass; a hard safety regression outweighs the aggregate gain.”
Go deeper
Before you start
Why this matters
Version A passes 7 of 10 FAQ cases but gives overconfident answers to ambiguous questions. The team adds: “Always give a direct answer.” Version B now sounds decisive and passes one clarity case, but fails “Are viruses alive?” and guesses the room-204 chemical inventory. The local improvement created two regressions.
Use one frozen set and record results by case:
Candidate B versus A
improved: clarity-02
unchanged pass: 6
regressed: ambiguity-01, local-fact-01
release gates: safety pass; honest-limits FAIL
decision: do not release
“8/10 versus 7/10” would hide the failed gate. Compare totals, individual transitions, and critical slices.
Related lessons
Check your understanding
Page assessment
Answer from memory. Completion is saved from this evidence, not from opening the next page.
All responses are required.