Mastery: ship checklist

A feature is shippable when its contract is versioned, expected behavior is tested, failure states terminate, security boundaries are enforced, and operators can see cost and reliability without reading sensitive content. Review the implementation as a system, not merely as a successful prompt.

Roll out gradually with a small cohort or shadow traffic. Compare task success, rejection rate, p95 latency, cost per successful task, and unsafe-action rate against a baseline. Define rollback thresholds before launch so evidence, not optimism, controls expansion.

Demonstrate the complete fixture: a scripted server returns 429 then 200, while 401 fails immediately and timeout exhausts predictably. Then demonstrate one rejection involving 429 storms, Retry-After parsing, timeout races, 401 loops, invalid JSON, and retry amplification. Review that AI_BASE_URL, AI_API_KEY, REQUEST_TIMEOUT_MS, MAX_RETRIES is documented without values, logs are scrubbed, cancellation reaches the external operation, and the terminal state is unambiguous. The release artifact should be a typed success or ApiFailure carrying safe diagnostics and retry history, not a raw provider object.

The rollout dashboard needs success and safe-rejection rates, p95 duration, retry/step counts, cost or work per success, and policy denials. Alert on sustained deviation, not single noisy requests. Assign owners for provider outage, schema drift, security escalation, and rollback. Ship only after another engineer can run the fixtures and explain why the implementation stops.

npm run typecheck
npm test
# Run the credentialed smoke test only in an approved environment:
RUN_LIVE_EVALS=1 npm run test:live

The code is intentionally provider-neutral at the adapter boundary. AI_BASE_URL and model names are environment placeholders, not invented services. If the topic uses MCP, install the current official SDK and verify its exported paths against its release documentation. If a provider offers an official SDK, it can replace fetch inside execute without changing the domain contract. Always inspect the real provider’s documented request and response fields before connecting credentials.

Debug from deterministic code outward. First print the parsed configuration with secret values replaced by [set]. Next run the parser against a local fixture. Then run the orchestration with a fake dependency. Only after those pass should you inspect the real boundary. Capture status, content type, request ID, elapsed time, and a redacted response shape. Do not “fix” malformed data with a permissive cast; save it as a failing fixture and decide whether the contract or adapter is wrong.

For this chapter, the fastest diagnostic is to assert that the total attempt count and wall-clock budget are bounded and non-idempotent work is not replayed blindly. When that assertion fails, stop before downstream work. Expected behavior is concrete: A 429 with Retry-After is delayed once, then success returns with attempts=2. A different but plausible sentence is not enough if the structural and policy checks fail.

• Typecheck and scoped tests pass.
• Positive, negative, timeout, and cancellation fixtures pass.
• Rollout metrics and rollback thresholds have owners.
• Run npm run typecheck and npm test without provider credentials.
• Record expected terminal behavior beside the fixture so future changes remain reviewable.

A passing test is necessary but does not establish production quality. Review the fixture set for realistic distributions, stale assumptions, tenant boundaries, and expensive edge cases. When outputs are probabilistic, assert schemas, citations, chosen capabilities, stop reasons, and task rubric scores. Reserve exact string assertions for deterministic adapters and protocol framing.