Measure whether the order-status tool caller works

Translate the claim into assertions or a tiny eval harness. The metric to protect is: tool-call validity rate; grounded status answers vs fabricated ones. Always record the denominator (how many cases) beside any rate. A percentage without a denominator is marketing, not measurement.

const parsed = OutputSchema.safeParse(raw);
if (!parsed.success) {
  throw new ContractError("invalid_output", parsed.error.issues.map(i => ({
    path: i.path.join("."), code: i.code
  })));
}
return parsed.data;

Expected evidence: A request for ord_123 produces one validated lookup and a reply based only on its result.. A passing assertion proves only the behavior it names; broader usefulness still needs the chapter’s full limits.

Be explicit: beating the baseline (assistant that answers without tools on the same prompts) on this fixture does not prove behavior under executing a hallucinated tool name, or answering status without a tool call. Label observations separately from conclusions so the next page inherits honest evidence about the order-status tool caller.

Compute tool-call validity rate; grounded status answers vs fabricated ones with the denominator written beside the rate every time. For this chapter, the evaluation set is intentionally tiny; that is allowed only if you say so in the evidence. Compare against assistant that answers without tools on the same prompts before celebrating.

Add one negative case aimed at executing a hallucinated tool name, or answering status without a tool call. A suite with only happy cases cannot protect the order-status tool caller when the characteristic failure appears in review.

If a check is expensive or flaky, shrink it until it is deterministic on orders map + tool schema for get_order_status. Flaky green builds teach the team to ignore gates. Record what this page does not prove so security-ops and mastery-ship inherit honest limits.

At the evaluation stage for function-calling-code, the job is narrower than finishing a product demo. You are creating one progressive evidence piece about orders map + tool schema for get_order_status that later pages inherit without redefining success. Keep that fixture small enough to inspect by hand, keep outputs copy-pasteable as text, and refuse to narrate this baseline as if it were a production SLA: assistant that answers without tools on the same prompts.

For this page specifically, success looks like metrics with explicit denominators and a negative case while still centering the user decision to let the model request a read-only get_order_status tool, then answer from the tool result. If you cannot point to a file, command, or assertion that proves that for the order-status tool caller, stay on this page instead of advancing.

Glossary: tool · Glossary: structured output · Cheatsheet: production ops signals

Previous · Next