Evaluate agents—or decline them

Start with real task categories, not only polished demos. Include ordinary cases, ambiguous requests, missing data, conflicting sources, tool outages, permission failures, adversarial content, and tasks that should be refused or escalated. Preserve important slices such as language, customer type, document format, and consequence level.

Each task needs expected properties. Some have a gold final answer. Others have acceptance criteria: required fields, permitted sources, forbidden actions, maximum spend, or required escalation. A debugging agent may have several valid trajectories, so do not require one exact sequence if multiple safe routes work.

Keep a stable regression set and a fresh holdout set. If prompts and tool descriptions are tuned repeatedly against the same examples, apparent improvement may be test overfitting.

Outcome measures ask whether the result is useful and safe:

• task completion and required-field coverage;
• factual correctness and grounded citations;
• policy compliance and appropriate refusal;
• side-effect correctness and absence of duplicates;
• quality of partial results and escalation.

Trajectory measures inspect how the system reached the outcome:

• valid tool selection and argument accuracy;
• repeated or unnecessary actions;
• recovery from typed failures;
• adherence to plan dependencies;
• budget compliance;
• use of untrusted content;
• accurate stop reason.

Do not score the private chain of thought. Score observable action proposals, tool observations, state transitions, short operational rationales when supplied, and final artifacts. A correct answer reached through a prohibited data source is still a failed run.

Use deterministic checks wherever possible: schema validation, citation reachability, forbidden-tool detection, call counts, exact totals, and stop-status consistency. Human reviewers or carefully validated model graders can assess semantic quality, but calibrate them against examples and disagreement.

Agents are stochastic and environment-dependent. Run important tasks multiple times. Report pass rate and the pattern of failures, not one successful trace. Break down results by task slice and consequence.

Useful reliability views include:

• full task success rate;
• safe failure rate when success is impossible;
• severe policy-violation rate;
• pass rate across repeated runs;
• tool-error recovery rate;
• p50 and p95 steps, cost, and latency.

An average can hide a dangerous tail. A mean of six calls may coexist with rare 50-call loops. A 98% success rate may be unacceptable if the remaining two percent sends unauthorized messages, while 90% may be acceptable for reversible draft suggestions.

Set confidence intervals when sample sizes matter, and inspect each severe failure. One catastrophic policy breach should not disappear inside a composite score.

Agent cost includes model calls, tool APIs, retrieval, storage, review, retries, and operational support. Measure cost per attempted task and per successful task. Cheap failed runs do not create value.

Latency should include queue time, model inference, tool execution, retries, and human approval where applicable. Report percentile distributions. Users experience the slow tail, especially when one rate-limited tool causes repeated backoff.

Budgets create trade-offs. Reducing maximum steps may lower cost but increase incomplete results. A smaller model may choose tools less accurately and cause more retries. Evaluate configurations end to end instead of assuming a lower per-token price means lower task cost.

Compare against baselines: human-only handling, one-shot calls, fixed workflows, search plus template, and the current process. The relevant question is whether the agent improves the required outcome at an acceptable total cost and risk.

Avoid an agent when:

• the correct sequence is known and stable;
• one call has all necessary context;
• deterministic rules can solve the task reliably;
• latency or cost must be tightly predictable;
• actions are high consequence and cannot be independently verified;
• available tools expose excessive privilege;
• success cannot be defined or measured;
• failures cannot be contained or recovered;
• volume makes small error rates unacceptable.

For example, calculating payroll tax from a defined formula should use tested code. Routing every payment through an open-ended agent loop adds nondeterminism to a rule-governed task. A model might still classify an unstructured note, but a workflow should validate and execute the financial operation.

Another poor fit is a “general company agent” with access to email, files, billing, and customer systems before narrow tasks and permissions are defined. Split the problem into bounded capabilities and workflows first.

Before release, define thresholds by risk: completion rate, zero tolerance for specified prohibited actions, maximum p95 cost, maximum p95 latency, and required escalation behavior. Test model updates, prompt changes, tool changes, and policy changes because any of them can alter trajectories.

Production monitoring should compare real distributions with evaluation assumptions. Track tool error rates, denied actions, step counts, costs, latency, user corrections, human overrides, and stop reasons. Sample traces for review with privacy controls. Trigger rollback or reduced autonomy when thresholds break.

Evaluation is not a one-time score. It is the evidence used to choose architecture, limits, and release scope.