Canary deploy lab
Create the runnable skeleton and contracts
Build a containerized answer API releasing image v2 beside stable image v1 through weighted routing as an operable release, not a slide-deck example.
1Try it yourself
Playground
Canary deploy desk
Split traffic, watch metrics, rollback or promote — never big-bang without a safety net.
Route 10% traffic to v2
Before you start
Why this matters
Before changing code, write the single production outcome this chapter must prove and the signal that would stop you. For this lab, the service boundary is both revisions expose identical /answer, /live, and /ready contracts and emit revision, request_id, latency_ms, and groundedness. Record one request identifier you can follow from ingress through the final decision. If you cannot name the owner of the stop decision, the rollout is not yet controlled.
The source runbook keeps v1 as the instant rollback target and compares latency, errors, and groundedness before promotion. This chapter turns that compact lesson into implementation evidence. The running scenario is a containerized answer API releasing image v2 beside stable image v1 through weighted routing. You will keep the same scenario across all eight chapters so setup decisions, tests, telemetry, and rollback controls accumulate into one coherent system rather than eight disconnected exercises.
2Learn the idea
Read
Lay out the runnable project
Use a small repository shape: app/ for runtime code, tests/ for contract and failure tests, config/ for reviewed rollout policy, fixtures/ for sanitized events or golden questions, and ops/ for dashboards and runbooks. Pin dependencies and expose a single local command that starts the service. The goal is not framework sophistication; it is making the production boundary reproducible on a laptop and in CI.
The central interface is both revisions expose identical /answer, /live, and /ready contracts and emit revision, request_id, latency_ms, and groundedness. Represent it with typed models and reject unknown required semantics at the edge. Add a schema_version when data may be persisted or delivered asynchronously. Generate request IDs at ingress, but preserve a trusted upstream correlation ID separately. Never overload a user ID as a request ID.
Read
Add reviewed configuration
Put this starting policy under version control:
release: answer-api-v2
steps:
- {weight: 5, observe_minutes: 15}
- {weight: 25, observe_minutes: 20}
- {weight: 50, observe_minutes: 30}
abort:
error_rate: "> 1%"
p95_latency_ms: "> 1800"
grounded_rate_delta: "< -0.02"
Validate configuration during startup and CI. Percentages must be bounded, referenced variants must exist, durations must be positive, and production must have a safe default. A malformed file should fail closed before the process reports readiness. Runtime overrides need an audit trail containing actor, old value, new value, reason, and timestamp.
Read
Establish local dependencies
Use fakes at process boundaries, not inside business logic. A fake provider should reproduce authentication failure, timeout, rate limit, malformed response, and success. A fake store should enforce uniqueness where production does. Seed deterministic fixture IDs so retry and idempotency tests are repeatable. Never copy production tokens or customer payloads into fixtures.
Define /live as “the process event loop responds” and /ready as “this instance can safely receive normal traffic.” Keep readiness bounded; an unbounded dependency call turns a probe into an outage amplifier. Include the release SHA and configuration revision in a diagnostic endpoint protected for operators.
Read
Prove the skeleton
Your first test starts the service with fake dependencies, calls readiness, submits the smallest valid request, and asserts the contract shape. A second test starts with invalid configuration and expects startup failure. A third starts with a dependency unavailable and expects liveness to remain healthy while readiness fails. These tests establish deployment semantics before model behavior complicates diagnosis.
Document local commands and expected exit codes. A new teammate should be able to clone, install, start, test, and reset state without undocumented cloud access. That repeatability is part of the implementation, because every later incident drill depends on recreating behavior safely.
Checking tutor…