Chapter DCapstone ops labPage 8 of 8

Capstone ops lab

Run acceptance tests and make the ship call

Build the Personal Knowledge Assistant operated as a versioned API with retrieval, model, feature flags, probes, secrets, and an on-call owner as an operable release, not a slide-deck example.

~18 minProduction ship gate

1Try it yourself

Playground

Capstone ops gate

Before demo day — metrics, flags, and recovery must be real.

Latency + groundedness alerts wired

Before you start

Why this matters

Before changing code, write the single production outcome this chapter must prove and the signal that would stop you. For this lab, the service boundary is the service publishes /live, /ready, /answer, revision-tagged telemetry, an SLO, rollback controls, and a tested incident runbook. Record one request identifier you can follow from ingress through the final decision. If you cannot name the owner of the stop decision, the rollout is not yet controlled.

The source checklist joins SLOs, flags, probes, secrets, load tests, disaster recovery, runbooks, on-call, and postmortems into one tested gate. This chapter turns that compact lesson into implementation evidence. The running scenario is the Personal Knowledge Assistant operated as a versioned API with retrieval, model, feature flags, probes, secrets, and an on-call owner. You will keep the same scenario across all eight chapters so setup decisions, tests, telemetry, and rollback controls accumulate into one coherent system rather than eight disconnected exercises.

2Learn the idea

Read

Assemble the release candidate

Freeze a candidate by recording source SHA, dependency lock digest, configuration revision, fixture digest, and deployment artifact. Run setup from a clean checkout, then execute unit, contract, integration, quality, and failure suites in that order. Do not substitute an old green build for evidence about the frozen candidate. For this capstone, the architecture must work as a whole: API ingress, retrieval or decision layer, external dependencies, durable evidence, feature controls, telemetry, and human ownership are all included in the test.

Review the contract: the service publishes /live, /ready, /answer, revision-tagged telemetry, an SLO, rollback controls, and a tested incident runbook. Confirm one realistic request completes the happy path and can be traced end to end. Confirm malformed and unauthorized requests fail before side effects. Confirm duplicate or repeated requests preserve required consistency. Confirm dependency timeout enters the documented degraded mode.

Read

Run the acceptance plan

Execute these acceptance cases and attach evidence:

  1. Functional: the canonical fixture returns the contracted structured result and correct attribution.
  2. Quality: the versioned golden set meets the primary threshold with reviewed case-level failures.
  3. Performance: expected concurrency stays within the latency and saturation budget.
  4. Resilience: timeout, malformed response, and unavailable dependency produce bounded behavior.
  5. Security: cross-tenant or forged input is rejected with zero unauthorized side effects.
  6. Operations: an alert fires, the runbook is followed, and safe behavior is restored within the target.
  7. Rollback: disable risky flags, route to the stable revision, restore the last verified index snapshot, and invoke the human-only degraded mode.

The metric decision uses availability, p95 latency, grounded-answer rate, saturation, error-budget burn, and recovery time from synthetic failure. Require complete telemetry and the minimum observation window. Record sample counts and confidence where comparisons are statistical. A passing average cannot override a hard security, correctness, or latency guardrail.

Read

Make the ship decision

Use three outcomes. Ship means all blocking tests pass, rollback is timed, dashboards and alerts are live, ownership is accepted, and residual risks have dates. Hold means evidence is missing or a non-critical defect has an explicit repair owner. Rollback means a hard guardrail, security boundary, data-integrity invariant, or recovery target failed.

Conduct a 30-minute game day after deployment. Inject one dependency failure or risky-control breach, let the on-call responder diagnose from telemetry, execute the runbook, and verify recovery from user-visible symptoms. Capture detection time, decision time, recovery time, and any manual step.

Read

Production ship packet

The final packet contains architecture and trust-boundary diagrams, API or event contracts, configuration, test report, golden-set digest, performance result, dashboard and alert links, security review, rollback transcript, runbook, owner, and dated residual risks. The system is shipped only when another operator can use that packet to understand what is live, detect harm, stop exposure, and recover without the author present.

Checking tutor…

Continue learning · glossary & guides