A/B test lab
Create the runnable skeleton and contracts
Types and env contracts make **support-answer service comparing retriever_v1 vs reranking retriever_v2** fail closed before any provider or cluster call.
1Try it yourself
Decision drill
A/B test lab
Split traffic and measure quality — not every change needs an experiment, but behavior shifts do.
1/3New system prompt — 50/50 traffic.
2Learn the idea
Read
Define trusted borders
Implement types or schemas around POST /v1/support/answer so illegal states are unrepresentable at the boundary. For support-answer service comparing retriever_v1 vs reranking retriever_v2, trusted inputs come from sessions, signatures, pinned digests, or workload identity — not from free-form model text. experiment owner measuring groundedness lift with SRM checks should be able to read the contract and know which fields are optional, which are enumerated, and which abort the request. Constructors and boot paths must not perform provider side effects; injection points keep tests honest.
Read
Environment and secrets contract
Document required env vars in .env.example with placeholders only. Rotation story belongs later, but the contract already forbids printing secrets and forbids defaulting to fail-open when a dependency is missing. Invariant to encode in types/tests: assignment sticky by user_id; analysis gated on sample ratio mismatch (SRM) p>0.001 fail. If a config number lacks units, fix the name (timeout_ms, rpm_hard) before writing logic.
Read
Implementation artifact
def assign(user_id: str, exp=EXP) -> str:
return "v2" if stable_hash(user_id, exp["key"]) % 100 < 50 else "v1"
Read
Fixture kit
Create fixtures for the golden path and for AB-SRM-FAIL-27. Name files after the behavior (429-retry-after.json, cross-tenant.json, empty-citation.json) rather than test1. Each fixture carries expected status/code. This kit is the shared language for validation and failure pages.
Read
Stage depth
Compatibility promise: additive fields may appear only if readers ignore unknowns safely; breaking changes bump a version visible on the wire. For AI payloads, size limits arrive before JSON parse when hostile blobs are a risk. Document how clock skew, idempotency keys, and tracing headers travel through support-answer service comparing retriever_v1 vs reranking retriever_v2. If you use feature flags later, the contract already states that flags are not authorization. Link each config knob to a unit and a failure mode (“0 means disabled” vs “0 means divide-by-zero”). A peer reviewing the PR should find AB-SRM-FAIL-27 named in a comment on the adversarial fixture.
Read
Field notes for `ab-test-lab` / `setup-and-contract`
Generate OpenAPI or a typed client only after the hand schema is stable for one fixture round-trip. Record how errors look on the wire — problem+json, envelope, or bare status — and stick to one. Clock sources must be injectable for skew tests. If webhooks appear later, document signature header names now even as TODOs. Keep sample payloads UTF-8 and free of real emails. Add a makefile or npm script that validates schemas without network. In this chapter the product is support-answer service comparing retriever_v1 vs reranking retriever_v2, the human stakeholder is experiment owner measuring groundedness lift with SRM checks, and the incident id you design against is AB-SRM-FAIL-27. Re-state the oracle in your notes — after 20k sessions, v2 groundedness +2.1pp, p95 latency +80ms within budget — and keep the invariant visible: assignment sticky by user_id; analysis gated on sample ratio mismatch (SRM) p>0.001 fail. Track groundedness_lift_pp and srm_pvalue as the scoreboard. Surface under change control: POST /v1/support/answer. If you only have forty minutes, finish the fixture for reassignment every request → users flicker; metrics uninterpretable before polishing UI. Promotion language stays ternary: promote, hold, or roll back based on evidence, not hope.
Go deeper
Before you start
Why this matters
For A/B test lab, sketch the request and response shapes that cross POST /v1/support/answer without naming a framework. Mark which fields are trusted (session, signatures, digests) versus untrusted (user text, model JSON, webhook bodies). If a field can change authorization, it does not belong in model output. Predict one 422/401 you will assert before coding adapters for support-answer service comparing retriever_v1 vs reranking retriever_v2.
Related lessons
Check your understanding
Page assessment
Answer from memory. Completion is saved from this evidence, not from opening the next page.
All responses are required.