How to run an A/B model test

Compare two models or prompts on live traffic safely.

1. Define one primary metric (groundedness, CSAT proxy, task success)

2. Split 50/50 or 90/10 canary — log variant on every request

3. Run until sample size hits power target (not arbitrary hours)

4. Check quality **and** latency/cost per variant

5. Ship winner; archive loser config in experiment log

**Try the lesson:** `canary-deploy-lab` in Lane D