Define the production target for health checks

This lab builds Kubernetes liveness vs readiness for retrieval-and-generation API. The human in the loop is cluster operator during vector DB maintenance window. Scope is intentionally narrower than “make AI reliable”: you will prove one oracle — vector pause → readyz 503, livez 200, no restart loop; process deadlock → livez fails → restart — and one invariant — /livez only checks process; /readyz checks vector+model deps; wrong probe kills healthy pods. Record non-goals in your notes so a later change cannot silently expand authority. The incident mnemonic for the chapter is PROBE-RESTART-STORM-8; design as if that ticket is already written and you are filling evidence.

Turn the oracle into a table: input fixture, expected observable, prohibited side effect, owner, latency/cost ceiling. Separate model taste from software correctness — transport, auth, parsing, and termination must be deterministic even when generated text varies. Primary metric family: restart_count == 0 during dep outage and ready_pods ≥ 1 when deps healthy. Averages without a denominator or revision label do not gate release. Fake external dependencies in unit tests; live calls wait until fakes pass.

Before implementation, encode a failing check that would have caught readyz embedded in liveness → restart storm during dependency blip. That failure is the pedagogical north star for later pages: contracts reject it, happy path never performs it, validation asserts it, failure-handling contains it, observability detects it, security-ops prevents privilege tricks around it, and mastery replays it in a drill. Endpoint under study: GET /livez and GET /readyz.

Capacity note for planners: estimate peak demand on GET /livez and GET /readyz and the cost ceiling for a failed retry storm. Write the abort conditions — unbounded spend, cross-tenant leakage, or inability to roll back — before you enjoy the first green test. Prefer synthetic fixtures shaped like production over anonymized production dumps you cannot share in class. When you are tempted to widen scope, re-read the oracle (vector pause → readyz 503, livez 200, no restart loop; process deadlock → livez fails → restart) and cut features that do not serve it. The teaching outcome is judgment under constraints: cluster operator during vector DB maintenance window gets a trustworthy control, not a kitchen-sink framework. Keep the language of release decisions: promote, hold, or roll back — never “see if it gets better.”

Decide what will live in version control on day one: fixtures, contract markdown, and a failing test name. Write the cost ceiling as a hard number with currency and period. If the lab involves clusters, name the non-prod context you will use and forbid prod kubecontexts in scripts. Capture the baseline metric once before changing code so later gains are comparative. In this chapter the product is Kubernetes liveness vs readiness for retrieval-and-generation API, the human stakeholder is cluster operator during vector DB maintenance window, and the incident id you design against is PROBE-RESTART-STORM-8. Re-state the oracle in your notes — vector pause → readyz 503, livez 200, no restart loop; process deadlock → livez fails → restart — and keep the invariant visible: /livez only checks process; /readyz checks vector+model deps; wrong probe kills healthy pods. Track restart_count == 0 during dep outage and ready_pods ≥ 1 when deps healthy as the scoreboard. Surface under change control: GET /livez and GET /readyz.