Chapter DSecrets rotation labPage 3 of 8

Secrets rotation lab

Implement one traceable happy path

Build an answer API and background indexer sharing a provider credential through a versioned secrets manager as an operable release, not a slide-deck example.

~14 minVertical slice

1Try it yourself

Playground

Secrets rotation lab

Revoke leaked keys, rotate on schedule, audit access.

API key found in a public gist

Before you start

Why this matters

Before changing code, write the single production outcome this chapter must prove and the signal that would stop you. For this lab, the service boundary is SecretRef(name, version) is injected at runtime; health telemetry reports only credential_version and authentication result, never secret bytes. Record one request identifier you can follow from ingress through the final decision. If you cannot name the owner of the stop decision, the rollout is not yet controlled.

The source lesson requires a dual-key window: issue v2, deploy readers, confirm traffic, then revoke v1 and audit every read. This chapter turns that compact lesson into implementation evidence. The running scenario is an answer API and background indexer sharing a provider credential through a versioned secrets manager. You will keep the same scenario across all eight chapters so setup decisions, tests, telemetry, and rollback controls accumulate into one coherent system rather than eight disconnected exercises.

2Learn the idea

Read

Build the vertical slice

Implement the smallest real flow before adding retries or optimization. Keep orchestration explicit so each decision can be tested and traced:

def provider_client() -> Client:
    versioned = secrets.get("llm/provider", stage="current")
    return Client(api_key=versioned.value,
                  telemetry={"credential_version": versioned.version})

# Rotation: create v2 -> stage v2 -> reload clients -> verify auth
# -> promote v2 current -> revoke v1 -> audit.

The handler should validate once, authorize before expensive work, and pass a structured context through each component. Record the release revision and configuration decision at the point they are made. Do not infer them later from deployment time; a request may cross a configuration update.

Read

Preserve causality

Use one correlation ID across the request, dependency calls, durable records, and outcome event. If the flow has an exposure or delivery ID, make it unique and immutable. Log a compact event with result category, duration, revision, and decision—not raw prompts, secrets, or complete customer objects. The resulting trace must answer: which code ran, which policy applied, what external facts were used, and what response class was returned?

Build one realistic fixture rather than a toy “hello world.” It should exercise the feature that justifies the release while remaining safe and deterministic. Assert the returned schema, cited or selected source identifiers, and side-effect count. Snapshot prose only when wording itself is contractual; otherwise assert structured facts so harmless model phrasing does not create brittle tests.

Read

Check operational behavior

Call the endpoint twice with the same stable identity or delivery identifier. The repeated call must preserve cohort or avoid duplicate work as required by this system. Restart the process and repeat the test; in-memory correctness is insufficient where state must survive deploys. Then inspect the trace to verify it includes the attributes needed to slice authentication failures, consumers by credential version, secret-read anomalies, and time from issue to old-key revocation.

Measure the local baseline now. Capture median and p95 duration over at least 100 fixture requests, dependency call count, and output-quality score on the golden set. This is not a capacity claim; it is a regression baseline. Save the command, machine assumptions, and revisions with the result.

Read

Keep scope disciplined

Do not add a dashboard, elaborate retry library, or generalized plugin system to make the happy path look production-ready. The chapter is complete when one valid request traverses real orchestration, emits attributable evidence, and produces the contracted result. The next chapters intentionally attack this slice. A narrow, visible implementation is easier to harden than a broad abstraction whose failure points are hidden.

Checking tutor…

Continue learning · glossary & guides