Evaluation and monitoring

An evaluation should support a specific decision: approve a use, compare two versions, expand to a population, remove a review gate, or verify a fix. Without a decision, teams may collect impressive numbers that do not answer whether the system is fit for purpose.

Write an evaluation plan in this order:

1. Decision: What will the result allow, block, or change?
2. Scope: Which task, population, language, workflow, model, prompt, data source, and tool configuration are covered?
3. Failure modes: What can go wrong, including incorrect, harmful, insecure, unfair, unusable, or policy-violating behavior?
4. Test cases: Which normal, difficult, adversarial, edge, and subgroup examples represent those failures?
5. Measures: How will quality and harm be scored?
6. Thresholds: What result is required, and which failures are automatic blockers?
7. Authority: Who reviews the evidence and decides?

This order prevents a common mistake: choosing “accuracy” because it is easy to calculate, then treating it as the whole decision.

The right measures depend on the task. A retrieval assistant may need source relevance, citation support, answer correctness, refusal behavior, latency, and cost. A classifier may need precision, recall, false positives, and false negatives. A workflow agent may need task completion, unauthorized-action rate, recovery, and human override.

Severity matters alongside frequency. One disclosure of sensitive records may outweigh dozens of minor formatting errors. Establish zero-tolerance or near-zero categories where appropriate, and inspect examples rather than only averages.

Slice results by meaningful groups and conditions: language, region, device, request type, customer type, accessibility need, or other factors tied to the use. An overall score can conceal a serious pocket of failure. Choose slices based on plausible risk and lawful, proportionate data use—not merely whatever columns happen to be available.

Include human judgment carefully. Use a scoring guide, examples, trained reviewers, and disagreement checks. If experts disagree often, the task definition may be ambiguous. That uncertainty belongs in the governance decision rather than being averaged away.

Evaluation data should include ordinary production-like cases and deliberately difficult cases. Sources can include curated examples, historical cases used under appropriate controls, synthetic variations, expert-authored scenarios, and prior incidents.

Protect against test leakage and overfitting. If developers repeatedly tune against one small set, scores may improve without general ability improving. Maintain a stable regression set for known behavior and a separate holdout or refreshed set for release decisions. Record test-set versions and known gaps.

Adversarial tests ask how the system behaves under misuse or attack: prompt injection, requests for prohibited actions, sensitive data exposure, fabricated sources, excessive tool permissions, or attempts to bypass policy. The exact suite follows the system’s capabilities. A text summarizer and an agent that can issue refunds require different tests.

Before running the evaluation, define what happens. Examples:

• all critical safety and privacy cases must pass;
• answer support must exceed a threshold overall and within approved language groups;
• no subgroup may exceed an agreed error gap without a documented control;
• a candidate release must not regress more than a set amount on core tasks;
• every failed high-severity case needs remediation or formal risk acceptance;
• uncertainty outside the approved scope must route to a person.

Predefined rules reduce the temptation to lower standards after seeing a favored release. Exceptions may sometimes be justified, but they should name the owner, rationale, compensating control, expiry date, and follow-up evidence.

Production monitoring should cover four layers:

1. Inputs: volume, language, topic, missing fields, suspicious patterns, and shifts from evaluated conditions.
2. System behavior: refusals, tool calls, retrieval quality, latency, failures, and version configuration.
3. Outputs: sampled correctness, unsupported claims, policy violations, harmful content, and sensitive-data exposure.
4. Outcomes: customer complaints, corrected decisions, appeal results, financial loss, reviewer overrides, and unequal effects.

Technical uptime is not model quality. A service can return a response in 300 milliseconds while giving systematically wrong advice. Outcome signals are often delayed, noisy, or influenced by many factors, but they are essential.

Monitoring must define owners and actions. For each signal, record a baseline, threshold, review frequency, alert recipient, response time, and available action. An alert with no authority to pause or narrow the system is only a notification.

More logging is not always better. Logs can contain confidential prompts, personal data, model outputs, and reviewer notes. Collect what is necessary, restrict access, set retention periods, and consider redaction or aggregation. Governance must balance observability with privacy and security.

Feedback is also selective. People who complain may not represent everyone harmed; users may click thumbs-up for a fluent but incorrect answer; reviewers may inherit the model’s bias. Combine several signals: structured sampling, expert review, complaints, overrides, incidents, and outcome checks.

Close the loop by sending production failures into evaluation sets. When a threshold is breached, choose among investigate, restrict scope, restore human review, roll back, pause, or accept with documented rationale. A dashboard that never changes a decision is not an effective control.