Reference · How-to · ~10 min
How to compare two models
Pick with **your tasks**, not leaderboard hype.
Pick with **your tasks**, not leaderboard hype.
Steps
1. List **5–10 real prompts** from your product (anonymized)
2. Run **same system prompt + temperature** on model A and B
3. Score blind if possible — which answer would you ship?
4. Measure **latency**, **cost**, and **failure modes** (refusals, format breaks)
5. Choose default model; keep the other for hard escalations
Scorecard
| Prompt | Model A | Model B | Winner | Notes |
|--------|---------|---------|--------|-------|
| … | quality 1–5 | quality 1–5 | | |
Dimensions beyond quality
- Follows JSON schema?
- Handles long context without forgetting?
- Tool calling reliable?
- Safe on adversarial inputs?
Copy-paste comparison prompt
Use identical messages; only change `model=` in API calls.