Capstone ops lab
Create the runnable skeleton and contracts
Build the Personal Knowledge Assistant operated as a versioned API with retrieval, model, feature flags, probes, secrets, and an on-call owner as an operable release, not a slide-deck example.
1Try it yourself
Playground
Capstone ops gate
Before demo day — metrics, flags, and recovery must be real.
Latency + groundedness alerts wired
Before you start
Why this matters
Before changing code, write the single production outcome this chapter must prove and the signal that would stop you. For this lab, the service boundary is the service publishes /live, /ready, /answer, revision-tagged telemetry, an SLO, rollback controls, and a tested incident runbook. Record one request identifier you can follow from ingress through the final decision. If you cannot name the owner of the stop decision, the rollout is not yet controlled.
The source checklist joins SLOs, flags, probes, secrets, load tests, disaster recovery, runbooks, on-call, and postmortems into one tested gate. This chapter turns that compact lesson into implementation evidence. The running scenario is the Personal Knowledge Assistant operated as a versioned API with retrieval, model, feature flags, probes, secrets, and an on-call owner. You will keep the same scenario across all eight chapters so setup decisions, tests, telemetry, and rollback controls accumulate into one coherent system rather than eight disconnected exercises.
2Learn the idea
Read
Lay out the runnable project
Use a small repository shape: app/ for runtime code, tests/ for contract and failure tests, config/ for reviewed rollout policy, fixtures/ for sanitized events or golden questions, and ops/ for dashboards and runbooks. Pin dependencies and expose a single local command that starts the service. The goal is not framework sophistication; it is making the production boundary reproducible on a laptop and in CI.
The central interface is the service publishes /live, /ready, /answer, revision-tagged telemetry, an SLO, rollback controls, and a tested incident runbook. Represent it with typed models and reject unknown required semantics at the edge. Add a schema_version when data may be persisted or delivered asynchronously. Generate request IDs at ingress, but preserve a trusted upstream correlation ID separately. Never overload a user ID as a request ID.
Read
Add reviewed configuration
Put this starting policy under version control:
slo:
availability: 99.9
p95_latency_ms: 2000
grounded_answer_rate: 0.94
release:
strategy: canary
kill_switches: [tool_calls, index_v2]
recovery:
rto_minutes: 30
rpo_minutes: 60
Validate configuration during startup and CI. Percentages must be bounded, referenced variants must exist, durations must be positive, and production must have a safe default. A malformed file should fail closed before the process reports readiness. Runtime overrides need an audit trail containing actor, old value, new value, reason, and timestamp.
Read
Establish local dependencies
Use fakes at process boundaries, not inside business logic. A fake provider should reproduce authentication failure, timeout, rate limit, malformed response, and success. A fake store should enforce uniqueness where production does. Seed deterministic fixture IDs so retry and idempotency tests are repeatable. Never copy production tokens or customer payloads into fixtures.
Define /live as “the process event loop responds” and /ready as “this instance can safely receive normal traffic.” Keep readiness bounded; an unbounded dependency call turns a probe into an outage amplifier. Include the release SHA and configuration revision in a diagnostic endpoint protected for operators.
Read
Prove the skeleton
Your first test starts the service with fake dependencies, calls readiness, submits the smallest valid request, and asserts the contract shape. A second test starts with invalid configuration and expects startup failure. A third starts with a dependency unavailable and expects liveness to remain healthy while readiness fails. These tests establish deployment semantics before model behavior complicates diagnosis.
Document local commands and expected exit codes. A new teammate should be able to clone, install, start, test, and reset state without undocumented cloud access. That repeatability is part of the implementation, because every later incident drill depends on recreating behavior safely.
Checking tutor…