Mastery: ship checklist

A feature is shippable when its contract is versioned, expected behavior is tested, failure states terminate, security boundaries are enforced, and operators can see cost and reliability without reading sensitive content. Review the implementation as a system, not merely as a successful prompt.

Roll out gradually with a small cohort or shadow traffic. Compare task success, rejection rate, p95 latency, cost per successful task, and unsafe-action rate against a baseline. Define rollback thresholds before launch so evidence, not optimism, controls expansion.

Demonstrate the complete fixture: fixtures cover direct answer, one-tool answer, budget exhaustion, and cancellation. Then demonstrate one rejection involving infinite loops, repeated tool calls, context growth, fabricated citations, and ambiguous completion. Review that AI_BASE_URL, AI_API_KEY, CHAT_MODEL, MAX_AGENT_STEPS is documented without values, logs are scrubbed, cancellation reaches the external operation, and the terminal state is unambiguous. The release artifact should be an explicit state machine with a final answer or a typed stop reason, not a raw provider object.

The rollout dashboard needs success and safe-rejection rates, p95 duration, retry/step counts, cost or work per success, and policy denials. Alert on sustained deviation, not single noisy requests. Assign owners for provider outage, schema drift, security escalation, and rollback. Ship only after another engineer can run the fixtures and explain why the implementation stops.

npm run typecheck
npm test
# Run the credentialed smoke test only in an approved environment:
RUN_LIVE_EVALS=1 npm run test:live

The code is intentionally provider-neutral at the adapter boundary. AI_BASE_URL and model names are environment placeholders, not invented services. If the topic uses MCP, install the current official SDK and verify its exported paths against its release documentation. If a provider offers an official SDK, it can replace fetch inside execute without changing the domain contract. Always inspect the real provider’s documented request and response fields before connecting credentials.

Debug from deterministic code outward. First print the parsed configuration with secret values replaced by [set]. Next run the parser against a local fixture. Then run the orchestration with a fake dependency. Only after those pass should you inspect the real boundary. Capture status, content type, request ID, elapsed time, and a redacted response shape. Do not “fix” malformed data with a permissive cast; save it as a failing fixture and decide whether the contract or adapter is wrong.

For this chapter, the fastest diagnostic is to assert that the loop has a step budget, deadline, abort signal, and deterministic terminal states. When that assertion fails, stop before downstream work. Expected behavior is concrete: The agent searches once, cites kb://returns, and stops before MAX_AGENT_STEPS. A different but plausible sentence is not enough if the structural and policy checks fail.

• Typecheck and scoped tests pass.
• Positive, negative, timeout, and cancellation fixtures pass.
• Rollout metrics and rollback thresholds have owners.
• Run npm run typecheck and npm test without provider credentials.
• Record expected terminal behavior beside the fixture so future changes remain reviewable.

A passing test is necessary but does not establish production quality. Review the fixture set for realistic distributions, stale assumptions, tenant boundaries, and expensive edge cases. When outputs are probabilistic, assert schemas, citations, chosen capabilities, stop reasons, and task rubric scores. Reserve exact string assertions for deterministic adapters and protocol framing.