Reference · Glossary
Benchmark
Last updated
A **standardized test suite** used to compare models on tasks like math, coding, safety, or retrieval — same questions, scored the same way.
#When to use
Choosing between models, tracking regressions after prompt changes, or reading vendor leaderboards with healthy skepticism.
#When not to
High benchmark scores don't guarantee your product works — real users, your docs, and your latency budget may differ completely.
#Example
MMLU measures broad knowledge across subjects; a model that tops MMLU can still hallucinate your company's refund policy without RAG.