Test health checks signals and thresholds

You own a Kubernetes retrieval-and-generation API at GET /livez and GET /readyz. The goal is to separate process liveness from traffic readiness so orchestration restarts dead processes without creating restart storms during dependency degradation. The measurable target is livez answers within 100 ms whenever the event loop works; readyz answers within 200 ms and requires config loaded plus a successful model credential probe cached for 30 seconds; three failed readiness probes remove the pod, while liveness waits 30 seconds and fails five times before restart. The known production tension is deep checks catch dependency faults but couple fleet availability to remote services; shallow checks preserve capacity but may route traffic to pods unable to complete expensive requests.

Convert the Health checks objective into tests, not screenshots. The target is livez answers within 100 ms whenever the event loop works; readyz answers within 200 ms and requires config loaded plus a successful model credential probe cached for 30 seconds; three failed readiness probes remove the pod, while liveness waits 30 seconds and fails five times before restart. Write one assertion for the exact boundary, one immediately inside it, and one immediately outside it. Include denominator behavior when traffic is zero, missing, duplicated, delayed, or partially sampled.

The telemetry contract is probe_success{probe}, probe_duration_seconds, kube_pod_container_status_restarts_total, endpoint_ready, and request error rate by pod. For every series, verify metric type, unit, label allowlist, ownership, and retention. Counters must only rise; histograms need buckets around the target; gauges require staleness handling. Confirm dashboards display demand, successful or protected outcomes, user latency or age, saturation, and version context together. A green ratio without its event count is not sufficient evidence.

Run the test workflow:

pytest -q tests/test_health_contract.py
conftest test deploy/answer-api.yaml -p policy

Store machine-readable output as a build artifact. Seed deterministic time or fixtures where possible, but keep at least one integration test against the actual collector and rule engine. Tests must fail when the threshold expression, label name, or alert receiver is intentionally broken. A test that remains green after its signal is disconnected is worse than no test because it creates false confidence.

Use the configured contract:

livenessProbe:
  httpGet: { path: /livez, port: 8080 }
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 1
  failureThreshold: 5
readinessProbe:
  httpGet: { path: /readyz, port: 8080 }
  periodSeconds: 5
  timeoutSeconds: 1
  failureThreshold: 3
  successThreshold: 2

Replay the controlled scenario—blackhole the vector database for 90 seconds, then block the application event loop; readiness must drop first without restarts, whereas the event-loop block must increment the container restart count exactly once.—and measure time from fault start to metric visibility, alert pending, alert firing, delivery, acknowledgement where relevant, mitigation, and stable recovery. Compare those times with the operational objective. Verify short benign blips do not page if the policy says they should not, while sustained or fast-burning impact does.

Check telemetry loss separately from service loss. Stop or misconfigure one scrape target and prove the system reports missing data rather than silently treating it as healthy. Verify logs and drill artifacts redact secrets and private payloads. Bound label cardinality using a representative workload and set an alert or budget for unexpected series growth.

Use the prior incident—a readiness endpoint reused the full user query path, timed out on the model provider, and caused all 18 pods to leave service simultaneously even though cached answers were still safe.—to ask which test would have detected the problem earlier. Remove or demote signals that do not lead to a decision. The production tradeoff is deep checks catch dependency faults but couple fleet availability to remote services; shallow checks preserve capacity but may route traffic to pods unable to complete expensive requests. Quantify the cost of collection and paging against the risk reduction it buys.

This page passes only when contract tests, failure tests, alert-path tests, and recovery assertions all run from a clean environment and produce retained evidence linked to the configuration revision.