Triage and recover health checks

You own a Kubernetes retrieval-and-generation API at GET /livez and GET /readyz. The goal is to separate process liveness from traffic readiness so orchestration restarts dead processes without creating restart storms during dependency degradation. The measurable target is livez answers within 100 ms whenever the event loop works; readyz answers within 200 ms and requires config loaded plus a successful model credential probe cached for 30 seconds; three failed readiness probes remove the pod, while liveness waits 30 seconds and fails five times before restart. The known production tension is deep checks catch dependency faults but couple fleet availability to remote services; shallow checks preserve capacity but may route traffic to pods unable to complete expensive requests.

Assume the bounded failure is active: blackhole the vector database for 90 seconds, then block the application event loop; readiness must drop first without restarts, whereas the event-loop block must increment the container restart count exactly once. Begin with the user-visible objective—livez answers within 100 ms whenever the event loop works; readyz answers within 200 ms and requires config loaded plus a successful model credential probe cached for 30 seconds; three failed readiness probes remove the pod, while liveness waits 30 seconds and fails five times before restart.—and verify demand is present. Classify severity from measured impact, not from the apparent prestige of the dependency. Create a UTC timeline and record every observation separately from every hypothesis.

Query probe_success{probe}, probe_duration_seconds, kube_pod_container_status_restarts_total, endpoint_ready, and request error rate by pod by stable dimensions: environment, version, outcome, dependency, and instance or region where cardinality permits. Compare an affected slice with a healthy control. Narrow in this order: deployment/configuration change, tenant or workload shape, dependency response, resource saturation, then individual instance. That order reduces random restarts and preserves useful evidence.

Use the diagnostic workflow:

kubectl describe pod -l app=answer-api
kubectl get endpointslice -l kubernetes.io/service-name=answer-api -o yaml

Before each command, write what result supports the current hypothesis and what result falsifies it. A query that cannot change your decision is noise during an incident. Preserve correlation identifiers and exact query ranges. Do not paste secrets or customer content into the timeline.

The relevant operational configuration is:

livenessProbe:
  httpGet: { path: /livez, port: 8080 }
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 1
  failureThreshold: 5
readinessProbe:
  httpGet: { path: /readyz, port: 8080 }
  periodSeconds: 5
  timeoutSeconds: 1
  failureThreshold: 3
  successThreshold: 2

Prefer a reversible action scoped to the failing dimension: disable one flag, quarantine one message, reduce one concurrency pool, route one tenant, or select a verified recovery artifact. Give the mitigation an owner and expiry. Scaling, restarting, widening a timeout, suppressing an alert, or purging state may hide symptoms while increasing cost or destroying evidence; require an explicit reason before using them.

After the action, verify both harm reduction and side effects across several samples. Check deferred work and retries so the incident does not recur when a queue drains or a circuit closes. If the metric recovers but demand vanished, mitigation is unproven.

The historical case was a readiness endpoint reused the full user query path, timed out on the model provider, and caused all 18 pods to leave service simultaneously even though cached answers were still safe. Build a causal tree with at least two competing explanations, and identify the observation that eliminated each losing branch. Do not claim root cause solely because rollback correlated with recovery. Preserve build version, configuration revision, dependency status, and relevant state for later analysis.

The design tension is deep checks catch dependency faults but couple fleet availability to remote services; shallow checks preserve capacity but may route traffic to pods unable to complete expensive requests. Record whether mitigation favored availability, correctness, fairness, cost, or responder safety. This page passes when a second responder can reconstruct why the selected action was proportionate and why broader actions were rejected.