Reference · Glossary
Benchmark
Last updated
A **standardized test suite** used to compare models on tasks like math, coding, safety, or retrieval — same questions, scored the same way.
When to use
Choosing between models, tracking regressions after prompt changes, or reading vendor leaderboards with healthy skepticism.
When not to
High benchmark scores don't guarantee your product works — real users, your docs, and your latency budget may differ completely.
Example
MMLU measures broad knowledge across subjects; a model that tops MMLU can still hallucinate your company's refund policy without RAG.