AI Evals Explained: How to Test LLM and Agent Quality

AI evaluations, or evals, are repeatable tests used to measure whether an AI system meets defined requirements. They can assess factuality, task completion, safety, latency, cost, retrieval, tool use, or user outcomes. A useful eval answers a product decision; it is not just a leaderboard score.

LLM output is variable and often open-ended, so ordinary unit tests are necessary but insufficient. Teams need a portfolio: deterministic checks for properties code can verify, human judgments for nuanced quality, model-based graders for scalable approximation, and production signals for behavior that offline data misses.

Use this guide alongside RAG explained, small language model selection, and AI development tools.

Start with the system, not only the model

Users experience a complete system: instructions, model, retrieval, tools, memory, application code, interface, and policy. Evaluating only the base model can hide failures introduced by chunking, permissions, tool schemas, or post-processing.

Averages can conceal serious defects. Report distributions, subgroups, scenario categories, and worst-case or high-severity failures where appropriate.

Define what good means

Translate broad goals into observable criteria. “Helpful” is too vague. For a support assistant, quality might mean:

• answers the user’s actual question;
• cites an approved source containing the claim;
• does not invent policy;
• asks for missing account information safely;
• escalates restricted or ambiguous cases;
• responds within the product’s latency target.

Prioritize criteria based on harm and product value. A factual typo and an unauthorized refund should not receive the same weight. Define hard gates for unacceptable behavior and softer metrics for optimization.

Build a representative evaluation dataset

Start with real, permissioned usage patterns: anonymized production examples, support cases, domain-expert scenarios, and known incidents. Add designed edge cases for rare but important failures. Keep provenance and ensure data handling follows privacy and retention requirements.

Divide examples into:

1. Development set: visible to builders for iteration.
2. Regression set: stable cases that catch known failures.
3. Holdout set: protected from prompt and workflow tuning.
4. Adversarial set: targeted abuse and unusual conditions.
5. Production sample: periodically reviewed real behavior.

Avoid letting test answers enter prompts, retrieval corpora, or fine-tuning data. When contamination is possible, rotate private cases and use newly collected examples.

Synthetic cases can increase coverage, but should not replace real-world evidence. A generator and the system under test may share biases. See the synthetic data workflow for provenance and validation practices.

Choose the right grading method

Deterministic graders

Code should check exact properties: JSON schema, required fields, numeric tolerance, citation existence, permissions, tool arguments, database state, and forbidden side effects. These graders are fast, reproducible, and interpretable.

They can be brittle when applied to open-ended language. Do not mark a correct answer wrong merely because it differs from one reference string.

Human evaluation

Trained reviewers remain valuable for nuance, usefulness, tone, ambiguity, and high-impact decisions. Give reviewers a concise rubric with examples, hide system variants where possible, randomize order, and measure agreement.

Model-based graders

An LLM judge can classify outputs or compare two variants at scale. Require structured reasons, use a detailed rubric, shuffle answer order to detect position bias, and calibrate against expert labels. Use a grader model distinct from the candidate when practical.

Model graders can prefer verbosity or outputs similar to their own. Treat evaluated text as quoted data and validate judge reliability continuously.

A practical seven-step eval process

1. Write the decision

Specify whether the test will choose a model, approve a release, compare retrieval settings, or investigate an incident. A test without a decision tends to accumulate attractive but irrelevant metrics.

2. Define metrics and gates

Choose primary metrics before running variants. Establish minimum sample sizes, acceptable failure thresholds, subgroup requirements, and hard safety gates. Include latency and resource use when they affect user experience.

3. Create and label cases

Document input, context, expected behavior, unacceptable behavior, difficulty, risk, and source. For agent tasks, define the allowed final state and forbidden side effects rather than one exact action sequence.

4. Establish a baseline

Run the current production system or a simple implementation. Baselines reveal whether complexity delivers real improvement and expose broken graders.

5. Run reproducibly

Record model identifier, prompt version, retrieval index, tool definitions, parameters, code revision, and timestamp. Repeat stochastic tests or use controlled settings where supported. Save traces without retaining unnecessary sensitive content.

6. Analyze errors, not only scores

Inspect false positives and negatives. Group failures by cause: missing knowledge, poor retrieval, instruction conflict, invalid tool call, unsafe action, formatting, or grader error. Improve the narrowest failing layer.

7. Promote tests to regression and monitoring

Every confirmed production incident should produce a regression case when lawful and practical. Run a compact suite in continuous integration, a broader suite before releases, and sampled monitoring after deployment.

Evaluating RAG and agents

For RAG, evaluate retrieval and answer generation separately. A grounded answer cannot recover a missing document. Measure whether the correct source was eligible, retrieved, ranked, cited, and accurately represented. Include “no answer” cases.

For agents, inspect end states and traces:

• Was the task completed?
• Were only permitted tools used?
• Were arguments valid and scoped?
• Did the agent recover from timeouts or partial failures?
• Were repeated writes prevented?
• Did it stop at limits and request approval?
• Did it expose sensitive information between tools?

Do not require one canonical trajectory when several safe paths succeed. Conversely, a correct final answer does not excuse an unauthorized intermediate action.

Eval checklist

• Each metric supports a named product or release decision.
• Cases represent common, rare, adversarial, and no-answer scenarios.
• Holdout examples are separated from development.
• Deterministic checks are used wherever possible.
• Human and model graders have explicit rubrics.
• Model judges are calibrated against expert labels.
• Results include subgroups and severity, not only averages.
• Full system versions and traces are reproducible.
• Latency, errors, and resource use are measured end to end.
• Known incidents become regression tests.
• Production monitoring detects drift after release.

Common limitations

Offline evals approximate future use; users and data change. Small score differences may be noise or grader disagreement. Public benchmarks can be contaminated or unrelated to the target workflow. Model graders may change across versions.

Optimization can also overfit the suite. Keep rotating holdouts, audit suspicious gains, and examine whether product outcomes improve. For high-impact systems, evaluations support—but do not replace—risk management, security review, domain validation, and human oversight.

FAQ

How many eval examples are enough?

There is no universal number. Use enough cases to detect changes meaningful to your decision, and increase coverage for heterogeneous or high-risk tasks. Report uncertainty rather than implying precision.

Can an LLM judge replace human reviewers?

Not completely. It can scale a calibrated rubric, while experts remain important for ambiguous, novel, and consequential cases and for auditing the judge.

What should run in CI?

Use fast deterministic checks plus a small, stable regression set. Run larger, slower, and human-reviewed suites at release checkpoints.

Is benchmark performance enough to select a model?

No. Benchmarks help shortlist candidates. Select using your complete system, representative data, operational constraints, and risk thresholds.

Bottom line

Good evals turn AI quality from an impression into a repeatable engineering decision. Define observable success, test the complete system, combine grading methods, protect holdouts, inspect failures, and connect offline results to production outcomes. Continue with AI implementation guides, model comparisons, and agentic AI fundamentals.

Sources and further reading

• NIST: AI Risk Management Framework
• NIST: Generative AI Profile
• OpenAI: Evaluation best practices
• Google Cloud: Generative AI evaluation overview
• Microsoft: Azure AI evaluation documentation
• Stanford CRFM: HELM
• MLCommons: AI Safety benchmark