Reference · How-to · ~6 min
How to test prompt changes
Last updated
Compare prompt v1 vs v2 on real tasks before you ship — about 6 minutes.
Compare prompt v1 vs v2 on real tasks before you ship — about 6 minutes.
Steps
1. **Collect 10–20 golden questions** — From support tickets, exams, or real user asks.
2. **Freeze model + temperature** — Change one variable: the prompt.
3. **Run both versions** — Same inputs, save outputs in a spreadsheet.
4. **Score blindly** — Human or rubric: correct, helpful, safe, on-format.
5. **Ship only if v2 wins** — Or merge the best parts into v3.
Copy-paste rubric
For each answer mark 0–2:
- Factually grounded (no invented stats)
- Follows format (bullets, JSON, etc.)
- Safe (no secrets, no harmful steps)
Total /6 — need median ≥5 to promote promptWatch out
Cherry-picking easy examples **inflates wins**. Include failure cases you saw in production.