Mastery: ship checklist

A feature is shippable when its contract is versioned, expected behavior is tested, failure states terminate, security boundaries are enforced, and operators can see cost and reliability without reading sensitive content. Review the implementation as a system, not merely as a successful prompt.

Roll out gradually with a small cohort or shadow traffic. Compare task success, rejection rate, p95 latency, cost per successful task, and unsafe-action rate against a baseline. Define rollback thresholds before launch so evidence, not optimism, controls expansion.

Demonstrate the complete fixture: a status question calls get_order_status once while small talk calls no tool. Then demonstrate one rejection involving unknown tool names, malformed arguments, repeated calls, and partial provider responses. Review that AI_BASE_URL, AI_API_KEY, CHAT_MODEL is documented without values, logs are scrubbed, cancellation reaches the external operation, and the terminal state is unambiguous. The release artifact should be validated tool requests and grounded assistant replies, not a raw provider object.

The rollout dashboard needs success and safe-rejection rates, p95 duration, retry/step counts, cost or work per success, and policy denials. Alert on sustained deviation, not single noisy requests. Assign owners for provider outage, schema drift, security escalation, and rollback. Ship only after another engineer can run the fixtures and explain why the implementation stops.

npm run typecheck
npm test
# Run the credentialed smoke test only in an approved environment:
RUN_LIVE_EVALS=1 npm run test:live

The code is intentionally provider-neutral at the adapter boundary. AI_BASE_URL and model names are environment placeholders, not invented services. If the topic uses MCP, install the current official SDK and verify its exported paths against its release documentation. If a provider offers an official SDK, it can replace fetch inside execute without changing the domain contract. Always inspect the real provider’s documented request and response fields before connecting credentials.

Debug from deterministic code outward. First print the parsed configuration with secret values replaced by [set]. Next run the parser against a local fixture. Then run the orchestration with a fake dependency. Only after those pass should you inspect the real boundary. Capture status, content type, request ID, elapsed time, and a redacted response shape. Do not “fix” malformed data with a permissive cast; save it as a failing fixture and decide whether the contract or adapter is wrong.

For this chapter, the fastest diagnostic is to assert that only an allowlisted function receives schema-validated arguments. When that assertion fails, stop before downstream work. Expected behavior is concrete: A request for ord_123 produces one validated lookup and a reply based only on its result. A different but plausible sentence is not enough if the structural and policy checks fail.

• Typecheck and scoped tests pass.
• Positive, negative, timeout, and cancellation fixtures pass.
• Rollout metrics and rollback thresholds have owners.
• Run npm run typecheck and npm test without provider credentials.
• Record expected terminal behavior beside the fixture so future changes remain reviewable.

A passing test is necessary but does not establish production quality. Review the fixture set for realistic distributions, stale assumptions, tenant boundaries, and expensive edge cases. When outputs are probabilistic, assert schemas, citations, chosen capabilities, stop reasons, and task rubric scores. Reserve exact string assertions for deterministic adapters and protocol framing.