Set release boundaries for the CI eval release gate

Operational risks for the release eval gate in CI center on editing golden expected answers to greenwash a bad candidate, plus the earlier failure mode (flaky eval that flips without seed control, or soft-fail that never blocks merge). Safety lives in executable gates, allowlists, redaction, and a named owner—not in a warning paragraph under an unsafe function.

gold_hash='sha256:fixture' assert gold_hash.startswith('sha256:') print({'gold_mutable_in_ci':False,'gold_hash':gold_hash})

Expected evidence: **gold immutable**. A failed assertion means stop, investigate, and do not publish the CI eval release gate.

Name who can disable the feature, what data is retained, and how to roll back to the last known good artifact. Pin the reviewed configuration (versions, thresholds, allowlists) so “what shipped” is reconstructable for eval-in-code.

Write the release blocker as a predicate, not a feeling: “Do not ship the CI eval release gate if editing golden expected answers to greenwash a bad candidate.” Pair it with a passing control that shows the reviewed configuration still works for golden JSONL + threshold.yaml. Name an owner and a rollback handle (git tag, docs_version, previous image).

Security pages must not paste the happy-path demo. If your gate code looks like the implementation page, replace it with a deny/allow check aimed at editing golden expected answers to greenwash a bad candidate.

State the data retention rule in one line (what is stored, for how long, who can read it). Then state the kill switch (env flag, config pin, or feature owner). The CI eval release gate is not shippable without both, even when pass/fail exit code; metric deltas vs baseline; seed recorded looks healthy.

At the safety and operations stage for eval-in-code, the job is narrower than finishing a product demo. You are creating one progressive evidence piece about golden JSONL + threshold.yaml that later pages inherit without redefining success. Keep that fixture small enough to inspect by hand, keep outputs copy-pasteable as text, and refuse to narrate this baseline as if it were a production SLA: last known good release metrics.

For this page specifically, success looks like an executable deny gate for the lab-specific threat while still centering the user decision to fail a candidate release when golden-task metrics regress past a pinned threshold. If you cannot point to a file, command, or assertion that proves that for the CI eval release gate, stay on this page instead of advancing.

How-to: ship agent with eval gate · Glossary: eval set

Previous · Next