Secrets rotation lab
Run acceptance tests and make the ship call
Build an answer API and background indexer sharing a provider credential through a versioned secrets manager as an operable release, not a slide-deck example.
1Try it yourself
Playground
Secrets rotation lab
Revoke leaked keys, rotate on schedule, audit access.
API key found in a public gist
Before you start
Why this matters
Before changing code, write the single production outcome this chapter must prove and the signal that would stop you. For this lab, the service boundary is SecretRef(name, version) is injected at runtime; health telemetry reports only credential_version and authentication result, never secret bytes. Record one request identifier you can follow from ingress through the final decision. If you cannot name the owner of the stop decision, the rollout is not yet controlled.
The source lesson requires a dual-key window: issue v2, deploy readers, confirm traffic, then revoke v1 and audit every read. This chapter turns that compact lesson into implementation evidence. The running scenario is an answer API and background indexer sharing a provider credential through a versioned secrets manager. You will keep the same scenario across all eight chapters so setup decisions, tests, telemetry, and rollback controls accumulate into one coherent system rather than eight disconnected exercises.
2Learn the idea
Read
Assemble the release candidate
Freeze a candidate by recording source SHA, dependency lock digest, configuration revision, fixture digest, and deployment artifact. Run setup from a clean checkout, then execute unit, contract, integration, quality, and failure suites in that order. Do not substitute an old green build for evidence about the frozen candidate.
Review the contract: SecretRef(name, version) is injected at runtime; health telemetry reports only credential_version and authentication result, never secret bytes. Confirm one realistic request completes the happy path and can be traced end to end. Confirm malformed and unauthorized requests fail before side effects. Confirm duplicate or repeated requests preserve required consistency. Confirm dependency timeout enters the documented degraded mode.
Read
Run the acceptance plan
Execute these acceptance cases and attach evidence:
- Functional: the canonical fixture returns the contracted structured result and correct attribution.
- Quality: the versioned golden set meets the primary threshold with reviewed case-level failures.
- Performance: expected concurrency stays within the latency and saturation budget.
- Resilience: timeout, malformed response, and unavailable dependency produce bounded behavior.
- Security: cross-tenant or forged input is rejected with zero unauthorized side effects.
- Operations: an alert fires, the runbook is followed, and safe behavior is restored within the target.
- Rollback: during the grace window restore v1 as current; after compromise, do not reactivate v1—repair the failed consumer with a newly issued v3.
The metric decision uses authentication failures, consumers by credential version, secret-read anomalies, and time from issue to old-key revocation. Require complete telemetry and the minimum observation window. Record sample counts and confidence where comparisons are statistical. A passing average cannot override a hard security, correctness, or latency guardrail.
Read
Make the ship decision
Use three outcomes. Ship means all blocking tests pass, rollback is timed, dashboards and alerts are live, ownership is accepted, and residual risks have dates. Hold means evidence is missing or a non-critical defect has an explicit repair owner. Rollback means a hard guardrail, security boundary, data-integrity invariant, or recovery target failed.
Conduct a 30-minute game day after deployment. Inject one dependency failure or risky-control breach, let the on-call responder diagnose from telemetry, execute the runbook, and verify recovery from user-visible symptoms. Capture detection time, decision time, recovery time, and any manual step.
Read
Production ship packet
The final packet contains architecture and trust-boundary diagrams, API or event contracts, configuration, test report, golden-set digest, performance result, dashboard and alert links, security review, rollback transcript, runbook, owner, and dated residual risks. The system is shipped only when another operator can use that packet to understand what is live, detect harm, stop exposure, and recover without the author present.
Checking tutor…