Run the health checks baseline

You own a Kubernetes retrieval-and-generation API at GET /livez and GET /readyz. The goal is to separate process liveness from traffic readiness so orchestration restarts dead processes without creating restart storms during dependency degradation. The measurable target is livez answers within 100 ms whenever the event loop works; readyz answers within 200 ms and requires config loaded plus a successful model credential probe cached for 30 seconds; three failed readiness probes remove the pod, while liveness waits 30 seconds and fails five times before restart. The known production tension is deep checks catch dependency faults but couple fleet availability to remote services; shallow checks preserve capacity but may route traffic to pods unable to complete expensive requests.

Build the smallest normal path through GET /livez and GET /readyz for a Kubernetes retrieval-and-generation API. Use a fixed, non-sensitive fixture and assign a correlation identifier at ingress. The purpose is to establish known-good semantics before fault injection. For Health checks, the path must contribute directly to this goal: separate process liveness from traffic readiness so orchestration restarts dead processes without creating restart storms during dependency degradation. Avoid adding retries, fallback branches, or tuning until the baseline is observable.

Use the reviewed configuration:

livenessProbe:
  httpGet: { path: /livez, port: 8080 }
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 1
  failureThreshold: 5
readinessProbe:
  httpGet: { path: /readyz, port: 8080 }
  periodSeconds: 5
  timeoutSeconds: 1
  failureThreshold: 3
  successThreshold: 2

Run the baseline command:

kubectl port-forward deploy/answer-api 8080:8080
watch -n1 'curl -s localhost:8080/readyz | jq .'

Record response status or operator state, important headers or fields, elapsed time, and side effects. Follow the same correlation identifier through structured logs. If the system is asynchronous, capture enqueue time, start time, completion time, retry count, and durable checkpoint. Run at least five samples so one warm cache or connection setup does not masquerade as normal behavior.

The acceptance target is livez answers within 100 ms whenever the event loop works; readyz answers within 200 ms and requires config loaded plus a successful model credential probe cached for 30 seconds; three failed readiness probes remove the pod, while liveness waits 30 seconds and fails five times before restart. Query probe_success{probe}, probe_duration_seconds, kube_pod_container_status_restarts_total, endpoint_ready, and request error rate by pod immediately before and after the baseline. Calculate deltas rather than trusting dashboard shape. Every successful user event should map to a defined outcome counter; every duration should have a clearly named clock boundary. Confirm version and environment dimensions let an operator distinguish old and new code without adding customer-specific cardinality.

Write an expected event sequence before looking at logs. Compare it to the observed sequence and explain every extra event, especially hidden retries or duplicate processing. A successful response with unexpected retries is operational debt because it consumes capacity and predicts a sharper failure under load. Verify that logs contain reason codes and correlation IDs but not request bodies, credentials, or private document content.

Vary one legitimate input dimension—request size, tenant class, queue age, or dependency latency—and keep the rest fixed. Identify where the normal path approaches the target. Save p50, p95, and maximum for a bounded sample, plus demand and saturation. The baseline is invalid if traffic count is missing.

Use the historical failure as a warning: a readiness endpoint reused the full user query path, timed out on the model provider, and caused all 18 pods to leave service simultaneously even though cached answers were still safe. Explain which normal-path measurement would have made that incident easier to recognize. Also record the production tension: deep checks catch dependency faults but couple fleet availability to remote services; shallow checks preserve capacity but may route traffic to pods unable to complete expensive requests. The baseline page passes when another engineer can reproduce the journey and predict which telemetry changes before running it.