Eval gates in code
Ship and explain the CI eval release gate
Page 8 packages proved vs unproved evidence so another engineer can run, trust, or reject the release eval gate in CI.
1Learn the idea
Read
Assemble the ship record
A shippable lab artifact includes: how to run it, the metric result (pass/fail exit code; metric deltas vs baseline; seed recorded), the failure you can still reproduce (flaky eval that flips without seed control, or soft-fail that never blocks merge), the security gate for editing golden expected answers to greenwash a bad candidate, and a rollback note. The user decision it supports remains: fail a candidate release when golden-task metrics regress past a pinned threshold.
Read
Freeze the evidence
release: candidate
baseline: stable
gates:
primary_metric: "weighted pass rate"
required: "0.90 overall and 1.00 for critical policy cases"
security_regression: pass
staging_probe: pass
telemetry_signal: "eval_gate_pass_rate"
rollout:
canary_percent: 5
rollback_on: "critical failure or sustained SLO breach"
owner: on-call-ai-platform
Expected evidence: 0.90 overall and 1.00 for critical policy cases. Store this beside the fixture version so scores remain meaningful after content changes in eval-in-code.
Read
Explain limits without apology
State operating limits for the CI eval release gate in plain language: fixture size, offline vs live dependencies, and what would require a new eval set. Shipping eval-in-code is honest scoping, not maximal confidence language.
Read
Lab notebook: proved vs unproved
Fill this table in your notes for the CI eval release gate:
- Proved on
golden JSONL + threshold.yaml: … - Unproved beyond the fixture: …
- Metric that blocks release: pass/fail exit code; metric deltas vs baseline; seed recorded
- Failure still reproducible: flaky eval that flips without seed control, or soft-fail that never blocks merge
- Security gate: editing golden expected answers to greenwash a bad candidate
- Rollback: …
Ship the narrative only when the unproved list is honest. Reviewers trust narrow claims that support fail a candidate release when golden-task metrics regress past a pinned threshold more than maximal language that collapses under the first production oddity.
Read
Worked judgment
Hand your ship note to a peer and ask them to recreate a proved/unproved ship note with rollback without watching you type. If they cannot, your evidence is still tribal knowledge. Tighten the run command and the metric line until a stranger can validate the CI eval release gate against golden JSONL + threshold.yaml.
Read
Why this stage matters for the CI eval release gate
At the mastery and shipping stage for eval-in-code, the job is narrower than finishing a product demo. You are creating one progressive evidence piece about golden JSONL + threshold.yaml that later pages inherit without redefining success. Keep that fixture small enough to inspect by hand, keep outputs copy-pasteable as text, and refuse to narrate this baseline as if it were a production SLA: last known good release metrics.
For this page specifically, success looks like a proved/unproved ship note with rollback while still centering the user decision to fail a candidate release when golden-task metrics regress past a pinned threshold. If you cannot point to a file, command, or assertion that proves that for the CI eval release gate, stay on this page instead of advancing.
Go deeper
Before you start
Why this matters
List two things this chapter proved on the fixture and two things it did not prove about the CI eval release gate. If you cannot name the gaps, you are not ready to ship the narrative—even if the code runs.
In the wild
See how this idea shows up as a product and a company — then come back to the lesson. Skills transfer across vendors.
Related lessons
Check your understanding
Page assessment
Answer from memory. Completion is saved from this evidence, not from opening the next page.
All responses are required.