Feature flags lab
Define the service boundary and ship target
Build a FastAPI RAG endpoint whose new index can be dark-launched, exposed to 5%, or disabled without a deploy as an operable release, not a slide-deck example.
1Try it yourself
Playground
Feature flag lab
Flags control who sees what — without redeploying code.
New RAG index for 5% of traffic
Before you start
Why this matters
Before changing code, write the single production outcome this chapter must prove and the signal that would stop you. For this lab, the service boundary is FlagContext(flag_key, user_id, tenant_id, environment) returning FlagDecision(enabled, variant, reason, config_version). Record one request identifier you can follow from ingress through the final decision. If you cannot name the owner of the stop decision, the rollout is not yet controlled.
The source lesson separates release from exposure and requires every trace to carry the evaluated variant. This chapter turns that compact lesson into implementation evidence. The running scenario is a FastAPI RAG endpoint whose new index can be dark-launched, exposed to 5%, or disabled without a deploy. You will keep the same scenario across all eight chapters so setup decisions, tests, telemetry, and rollback controls accumulate into one coherent system rather than eight disconnected exercises.
2Learn the idea
Read
Frame the release
Start with a one-page release brief. Name the current behavior, proposed behavior, affected cohort, operator, and explicit non-goals. The primary decision signal is decision latency, evaluation-error rate, cohort error rate, and grounded-answer rate by variant. A release cannot be called successful merely because requests return 200. Specify the observation window, minimum useful traffic, and the exact threshold that means hold, promote, or stop. State what happens when telemetry is missing: missing evidence must produce hold, never promotion.
Write an invariant beside each boundary. Inputs must be authenticated and schema-valid. Decisions must be deterministic where user consistency matters. Outputs must identify the code or configuration revision that produced them. Side effects must be idempotent or reversible. The operating team must be able to execute this rollback without editing application code: activate the server-side kill switch to serve control, verify decision logs, then revert the configuration revision. These invariants become acceptance tests later.
Read
Define the evidence packet
Create release-brief.md in your lab notes with a hypothesis, baseline, target, guardrails, and owner. Use a table in the artifact if useful, but make every threshold executable. “Latency should be reasonable” cannot drive automation; “p95 under 1,800 ms for two consecutive 10-minute windows” can. Include the expensive failure modes—non-sticky bucketing, stale SDK caches, missing defaults, mutually inconsistent flags, and a control path that has rotted—because an apparently positive average can hide cohort harm or operational instability.
The minimum evidence packet contains a configuration revision, test output, dashboard link, rollout timestamps, and rollback confirmation. For AI behavior, add a versioned golden dataset and grader version. A model score without the prompt, retriever, dataset, and grader revisions is not reproducible. For every metric, identify numerator, denominator, excluded traffic, and aggregation window.
Read
Make the stop rule operational
Assign one release commander. That person may promote only after the named gates pass and may rollback immediately when a hard guardrail breaches. Separate reversible exposure changes from irreversible data changes. If a migration cannot be reversed safely, move it outside this release or use an expand-and-contract sequence.
Run a paper drill: the primary metric improves, but the guardrail degrades. The correct decision is hold or rollback, not “wait for the average to recover.” Then remove the primary dashboard for five minutes. The decision must again be hold. Record those decisions in the brief so an operator under pressure does not improvise policy.
Read
Deliverable
Your deliverable is a signed release contract: one system boundary, one primary outcome, at least two guardrails, a minimum observation requirement, an owner, and a command-level rollback. It is complete only when a second person can distinguish promote, hold, and rollback from the written thresholds alone.
Checking tutor…