Inject a controlled health checks failure

You own a Kubernetes retrieval-and-generation API at GET /livez and GET /readyz. The goal is to separate process liveness from traffic readiness so orchestration restarts dead processes without creating restart storms during dependency degradation. The measurable target is livez answers within 100 ms whenever the event loop works; readyz answers within 200 ms and requires config loaded plus a successful model credential probe cached for 30 seconds; three failed readiness probes remove the pod, while liveness waits 30 seconds and fails five times before restart. The known production tension is deep checks catch dependency faults but couple fleet availability to remote services; shallow checks preserve capacity but may route traffic to pods unable to complete expensive requests.

This page deliberately breaks Health checks in a disposable or explicitly approved environment. The scenario is blackhole the vector database for 90 seconds, then block the application event loop; readiness must drop first without restarts, whereas the event-loop block must increment the container restart count exactly once. Write the hypothesis as an ordered signal chain: fault injection, component-level signal, user-visible indicator, protective action, and recovery. Include expected timestamps or maximum delays. “The system becomes unhealthy” is not precise enough to debug.

Capture a baseline of probe_success{probe}, probe_duration_seconds, kube_pod_container_status_restarts_total, endpoint_ready, and request error rate by pod and record the effective configuration:

livenessProbe:
  httpGet: { path: /livez, port: 8080 }
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 1
  failureThreshold: 5
readinessProbe:
  httpGet: { path: /readyz, port: 8080 }
  periodSeconds: 5
  timeoutSeconds: 1
  failureThreshold: 3
  successThreshold: 2

Define hard aborts before starting: unexpected data mutation, impact outside the test scope, unbounded cost, missing telemetry, or inability to remove the fault. Name the person who can stop the drill. Confirm the cleanup command and take a state snapshot that proves the environment can be restored.

Run exactly one fault workflow:

toxiproxy-cli toxic add vector-db -t timeout -a timeout=90000
kubectl exec deploy/answer-api -- kill -STOP 1

Mark the start time in UTC. Do not tune the system during the first observation window. Compare actual signal order with the hypothesis and note the first divergence. The target remains livez answers within 100 ms whenever the event loop works; readyz answers within 200 ms and requires config loaded plus a successful model credential probe cached for 30 seconds; three failed readiness probes remove the pod, while liveness waits 30 seconds and fails five times before restart. A protection mechanism may intentionally reject or degrade some work; count that as success only if the user contract explicitly permits it and unaffected traffic remains inside its objective.

Expected telemetry must show demand as well as outcomes. A flat error counter during a traffic outage is not resilience. Verify the injector itself worked through an independent observation such as dependency latency, worker count, policy state, or fault-proxy statistics. If no signal moves, stop and diagnose the drill rather than escalating fault intensity blindly.

Remove the injection at the planned time, preserve logs, and observe several consecutive samples. Check backlog, delayed retries, stale endpoints, cached state, and circuit or escalation state; recovery of the front-door status alone is insufficient. Compare final state to the baseline and explain any residual difference.

Use the historical incident as a reason to be exact: a readiness endpoint reused the full user query path, timed out on the model provider, and caused all 18 pods to leave service simultaneously even though cached answers were still safe. The production tradeoff is deep checks catch dependency faults but couple fleet availability to remote services; shallow checks preserve capacity but may route traffic to pods unable to complete expensive requests. State whether the current behavior contains that risk or merely moves it. The evidence bundle must contain the hypothesis, baseline, fault command, start and stop times, telemetry queries, cleanup proof, and one finding that changes a future design or operating decision.