Prove behavior with deterministic tests

Start with pure unit tests for deterministic decisions: schema checks, bucketing or routing, threshold comparisons, and redaction. Use fixed clocks and seeded identifiers. Then add contract tests against every fake dependency and one sandbox integration. Finally, run an end-to-end fixture through the public endpoint and assert the durable record and telemetry event, not just the HTTP response.

Turn the declared boundary into test cases. A valid request produces the service publishes /live, /ready, /answer, revision-tagged telemetry, an SLO, rollback controls, and a tested incident runbook. Missing required fields return a stable client error. Unknown schema versions are quarantined or rejected. Authorization failure occurs before external calls. A repeated identifier cannot duplicate effects. A timeout returns the documented degraded response. Every branch carries the revision and request ID.

For AI quality, keep a versioned JSONL golden set with input, permitted facts, forbidden claims, and expected disposition. Grade citations or fact support deterministically where possible; use a model grader only with a pinned grader prompt and periodic human calibration. Report confidence intervals and case-level failures, not one blended score. The release metric is availability, p95 latency, grounded-answer rate, saturation, error-budget burn, and recovery time from synthetic failure.

Add an adversarial set specifically for model outage, corrupt index, secret expiry, traffic spike, bad flag configuration, telemetry loss, and an unavailable on-call owner. Each fixture needs a short explanation of the production incident it prevents. This stops the suite becoming a pile of examples nobody trusts. When a real incident occurs, first add the smallest reproducing fixture, then fix the implementation.

CI should reject schema drift, unsafe configuration, failed deterministic tests, and quality regression beyond the declared tolerance. A flaky quality check is not permission to ignore failures. Quarantine it with an owner and deadline while preserving a stable blocking subset. Print revisions, seed, and dataset digest so any failure can be reproduced locally.

Create a rollout-gate function that accepts observed metrics and returns promote, hold, or rollback plus reasons. Test boundary values: exactly at threshold, one sample below minimum, stale metrics, and missing guardrails. Missing or stale evidence must return hold. This small function prevents a deployment script from silently interpreting policy differently from the runbook.

Have another person run the acceptance command without verbal guidance. They should be able to locate the failed fixture, identify its owning component, and explain whether it blocks shipment. Save a concise test report containing counts, failures, dataset digest, and environment. Tests become release evidence only when their inputs and meaning are reviewable.

Store a fixture with the local test harness so the knowledge assistant operations gate decision is reviewable rather than hidden in a mock. Pinning the revision and thresholds prevents a later environment change from silently changing what “pass” means.

capstone_ops_lab_contract:
  revision: assistant-2026.07.18
  index_snapshot: idx-2026-07-17
  flag_config: flags-184
  required_probes: [live, ready]
  max_p95_ms: 1200
  minimum_grounded_rate: 0.92

Run the same fixture unchanged and with one guardrail deliberately violated. The expected transcript makes the gate repeatable by a teammate who was not present when it was authored.

$ python3 -m pytest -q -k "capstone_ops_lab and contract"
2 passed in 0.41s
$ CONTRACT_REVISION=assistant-2026.07.18 python3 -m pytest -q -k "rejects_stale_evidence"
1 passed in 0.18s

Expected output is a stable pass count plus assistant-2026.07.18 in the report. If another revision appears, inspect fixture loading and environment precedence before changing thresholds. If stale evidence promotes instead of holding, stop: the gate is unsafe even if quality checks pass. Preserve the seed and failing fixture so the result can be reproduced without production traffic.