Tool selection, evals, and observability

Break evaluation into stages that correspond to engineering decisions:

1. Need detection: did the system call a tool when external evidence or action was required, and avoid calls when not required?
2. Tool selection: did it choose the correct tool among available alternatives?
3. Argument quality: were fields complete, typed, grounded in user input or trusted context, and semantically valid?
4. Policy compliance: did the application block unauthorized tools and require confirmation where appropriate?
5. Execution handling: were retries, timeouts, conflicts, and duplicate risks handled correctly?
6. Result grounding: did the final answer reflect the actual normalized result without inventing success?
7. Task outcome: did the user’s goal succeed, fail safely, or receive a useful clarification?

These dimensions identify different fixes. A wrong tool may require clearer descriptions or a smaller tool set. Invalid arguments may require schema changes. An unauthorized execution is an application bug even if the model requested it. A hallucinated success after a correct failure result is a grounding problem.

Create a dataset of user requests with expected behavior. Include direct calls, no-tool questions, ambiguous requests, multi-tool dependencies, unsupported capabilities, and safety-sensitive writes.

A case can record:

{
  "case_id": "calendar_hypothetical_07",
  "user_message": "What would a meeting with Lee tomorrow look like?",
  "expected": {
    "tool_behavior": "no_write",
    "allowed_tools": ["list_calendar_events"],
    "forbidden_tools": ["create_calendar_event"],
    "must_clarify": false
  },
  "risk_tags": ["intent_boundary", "calendar_write"]
}

Include near-neighbor pairs. “Draft an email” versus “Send the email” tests action intent. “What is typical weather in Nairobi in July?” versus “What is Nairobi’s weather now?” tests knowledge versus live retrieval. “Can I refund this?” versus “Refund this” tests recommendation versus action.

Use production-like language: typos, relative dates, multiple languages, indirect requests, and users changing their minds. Add adversarial tool results that contain instructions, malformed provider responses, and calls with extra arguments. Keep sensitive production data out of eval fixtures.

Evaluate observable events: advertised tools, structured proposals, validation decisions, approvals, execution attempts, results, and final responses. You do not need private chain-of-thought to judge whether a call was appropriate.

Useful metrics include precision and recall for “tool needed,” exact or semantic match for selected tool, schema-valid argument rate, clarification rate, unnecessary call rate, blocked unauthorized-call rate, duplicate-write rate, grounded-claim rate, and end-to-end safe completion.

For arguments, exact string equality may be too strict. Equivalent timezone-aware timestamps can represent the same instant. Define canonicalization and field-specific checks. For high-risk fields such as recipient, amount, and account, require exact grounding.

Weight cases by consequence. One unauthorized payment matters more than several unnecessary weather reads. Report performance by tool, request type, language, permission group, and risk tag instead of hiding weak slices inside an average.

Production traces connect evaluation concepts to real behavior. Give each request a trace ID and each proposal a tool-call ID. Record model and prompt versions, tool definitions or schema versions, validation outcomes, policy decisions, latency, provider status, retry count, normalized result status, and final response.

{
  "trace_id": "trace_91af",
  "tool_call_id": "call_calendar_03",
  "tool_name": "create_calendar_event",
  "schema_version": "2",
  "validation": "passed",
  "authorization": "passed",
  "approval_id": "approval_8ac1",
  "attempts": 1,
  "result_status": "created",
  "latency_ms": 438
}

Do not log secrets, access tokens, full private documents, or unnecessary personal data. Hash or tokenize sensitive identifiers where analysis permits, enforce access controls, and set retention periods. Observability should not become a second uncontrolled data store.

Metrics reveal trends; traces explain individual cases. Alerts should focus on actionable signals such as authorization denials suddenly dropping to zero, timeout rate increasing, duplicate-key conflicts, tool-selection drift after a model change, or unsupported success claims.

When the wrong tool is chosen, inspect the tool set before blaming the model. Overlapping names, vague descriptions, too many irrelevant tools, and one giant multipurpose function all reduce selection quality.

Make tools available contextually. A support flow may expose order lookup and ticket creation, while an editing flow exposes no side-effect tools. Contextual allowlists improve both safety and choice.

Sometimes a deterministic router should choose the domain or tool family before the model fills arguments. Sometimes the model should ask a question because two tools are genuinely plausible. Forced selection can be worse than clarification.

Test changes to names, descriptions, schemas, prompts, model versions, and tool counts against a fixed regression set. Add new production failures as redacted cases. Avoid optimizing only to the test set; maintain holdout cases and monitor live distributions.

Run offline evals on every meaningful change, then use staged rollout, shadow mode, or read-only mode where appropriate. For write tools, start with proposals that humans review. Compare approved and edited calls to learn where the system misunderstands intent without granting premature autonomy.

Define launch gates by risk. A weather assistant may tolerate occasional clarification. A money-moving tool should have near-zero unauthorized execution and tested idempotency under network failure. “The model is 95% accurate” is not a ship criterion without a denominator, slices, error consequences, and application-control tests.

After launch, sample traces, review incidents, and monitor drift. Provider schemas change, user behavior shifts, permissions evolve, and a model upgrade can alter call patterns. Evaluation is a continuous control loop, not a one-time benchmark.