Canary and operate health checks

You own a Kubernetes retrieval-and-generation API at GET /livez and GET /readyz. The goal is to separate process liveness from traffic readiness so orchestration restarts dead processes without creating restart storms during dependency degradation. The measurable target is livez answers within 100 ms whenever the event loop works; readyz answers within 200 ms and requires config loaded plus a successful model credential probe cached for 30 seconds; three failed readiness probes remove the pod, while liveness waits 30 seconds and fails five times before restart. The known production tension is deep checks catch dependency faults but couple fleet availability to remote services; shallow checks preserve capacity but may route traffic to pods unable to complete expensive requests.

Before shipping Health checks, link the goal, reviewed configuration, baseline, fault drill, diagnostic timeline, automated tests, security review, and rollback procedure. The production objective is livez answers within 100 ms whenever the event loop works; readyz answers within 200 ms and requires config loaded plus a successful model credential probe cached for 30 seconds; three failed readiness probes remove the pod, while liveness waits 30 seconds and fails five times before restart. Every claim in the release note must point to a command output, telemetry query, or approved decision. Missing evidence is a release blocker, not an item to infer from confidence.

The candidate configuration is:

livenessProbe:
  httpGet: { path: /livez, port: 8080 }
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 1
  failureThreshold: 5
readinessProbe:
  httpGet: { path: /readyz, port: 8080 }
  periodSeconds: 5
  timeoutSeconds: 1
  failureThreshold: 3
  successThreshold: 2

Diff it against the running revision and identify any field with fleet-wide effect. Name the operator, approver, observation window, and rollback trigger. Confirm dashboards and alerts query the candidate's labels before sending traffic. Freeze unrelated changes during the canary so attribution remains possible.

Run the staged rollout workflow:

kubectl rollout status deploy/answer-api --timeout=180s
./scripts/check-probe-budget.sh --namespace ai --minutes 20

Start with the smallest representative slice that can reveal the known failure mode. Compare candidate and control on demand, outcomes, latency or age, saturation, cost, and the primary series probe_success{probe}, probe_duration_seconds, kube_pod_container_status_restarts_total, endpoint_ready, and request error rate by pod. Observe longer than the slowest timeout, queue cycle, probe threshold, escalation interval, or data-rebuild checkpoint relevant to this lab. Promote only on prewritten criteria; do not move a threshold after seeing inconvenient data.

Abort and roll back if the controlled risk appears: blackhole the vector database for 90 seconds, then block the application event loop; readiness must drop first without restarts, whereas the event-loop block must increment the container restart count exactly once. After rollback, prove configuration revision, traffic allocation, deferred work, and user indicators returned to baseline. Keep the evidence even when the canary succeeds so the next operator has a reference distribution.

Publish owner, escalation path, runbook, dashboard, alert meaning, safe commands, access prerequisites, and review date. Schedule the next game day and define what architectural change invalidates this procedure. Track near misses, pages, manual interventions, false positives, cost, and time to mitigation over a 28-day window. A shipped control that nobody reviews will drift as traffic and dependencies change.

Use the historical incident—a readiness endpoint reused the full user query path, timed out on the model provider, and caused all 18 pods to leave service simultaneously even though cached answers were still safe.—as a regression scenario. The enduring production tension is deep checks catch dependency faults but couple fleet availability to remote services; shallow checks preserve capacity but may route traffic to pods unable to complete expensive requests. State what the rollout chooses today and what metric would force reconsideration. Close the release only when the on-call owner accepts the handoff and can execute rollback without the implementation author.