Secure health checks operations

You own a Kubernetes retrieval-and-generation API at GET /livez and GET /readyz. The goal is to separate process liveness from traffic readiness so orchestration restarts dead processes without creating restart storms during dependency degradation. The measurable target is livez answers within 100 ms whenever the event loop works; readyz answers within 200 ms and requires config loaded plus a successful model credential probe cached for 30 seconds; three failed readiness probes remove the pod, while liveness waits 30 seconds and fails five times before restart. The known production tension is deep checks catch dependency faults but couple fleet availability to remote services; shallow checks preserve capacity but may route traffic to pods unable to complete expensive requests.

List every capability needed to configure, inspect, inject failure into, mitigate, and roll back Health checks for a Kubernetes retrieval-and-generation API. Separate read, change, and destructive permissions. Grant them to short-lived roles rather than permanent personal credentials. The ordinary workflow must not require administrator access, and emergency access must expire automatically and emit an audit event.

Review this operational configuration for dangerous defaults:

livenessProbe:
  httpGet: { path: /livez, port: 8080 }
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 1
  failureThreshold: 5
readinessProbe:
  httpGet: { path: /readyz, port: 8080 }
  periodSeconds: 5
  timeoutSeconds: 1
  failureThreshold: 3
  successThreshold: 2

Check fail-open versus fail-closed behavior, external endpoints, data retention, tenant isolation, command scope, and rollback authorization. Validate unknown fields and unsafe ranges. If a setting can affect all tenants, require staged application and an independent approver. Keep secrets in the platform's secret store; never print them in config dumps, command history, metrics, or drill artifacts.

Run the security and operations gate:

kubectl auth can-i get secrets --as system:serviceaccount:ai:answer-api
kubectl diff -f deploy/answer-api.yaml

Prove a read-only responder cannot perform the mutation, an authorized role can perform only the scoped change, and the audit record contains actor, UTC time, target, revision, and result. Then revoke or expire the role and repeat the denial check. For scripts, pin dependencies, quote user-controlled values, enable strict error handling, and provide a dry-run mode for high-impact operations.

The monitoring surface probe_success{probe}, probe_duration_seconds, kube_pod_container_status_restarts_total, endpoint_ready, and request error rate by pod must avoid raw prompts, document contents, credentials, email addresses, and unconstrained customer identifiers. Use reason-code enums and controlled dimensions. Restrict detailed logs by role and retention, and test redaction with deliberately planted synthetic secrets.

The system must still meet livez answers within 100 ms whenever the event loop works; readyz answers within 200 ms and requires config loaded plus a successful model credential probe cached for 30 seconds; three failed readiness probes remove the pod, while liveness waits 30 seconds and fails five times before restart. under security constraints; a control that makes mitigation impossible during an outage is incomplete. Exercise the scenario blackhole the vector database for 90 seconds, then block the application event loop; readiness must drop first without restarts, whereas the event-loop block must increment the container restart count exactly once. with the least-privilege role and confirm rollback remains possible if a dependency or identity provider is degraded. Estimate provider, compute, storage, and telemetry cost, and set a hard drill budget where applicable.

The prior incident was a readiness endpoint reused the full user query path, timed out on the model provider, and caused all 18 pods to leave service simultaneously even though cached answers were still safe. Ask whether excess access, missing approval, poor auditability, or unsafe tooling could have worsened it. The core tradeoff is deep checks catch dependency faults but couple fleet availability to remote services; shallow checks preserve capacity but may route traffic to pods unable to complete expensive requests. Document who accepts the residual risk and when it will be reviewed.

Approval requires threat model, permission matrix, redaction test, audit sample, rollback owner, cost bound, and evidence that emergency access works and expires. Do not approve on policy prose alone.