Frame the CI eval release gate experiment

Success is not “I followed the tutorial.” Success is producing evidence that: gate exits non-zero on regression and zero on the pinned baseline. The accepted input is narrow on purpose: golden cases, metric functions, threshold config, candidate outputs. That narrowness is what lets you inspect every field and prevents a toy demo from being narrated as a production system.

Record the baseline you must beat: last known good release metrics. If the finished artifact cannot beat that baseline on the fixture below, stop and revise the claim before writing more code.

{"id":"refund-window","question":"Can I return headphones after 20 days?","must_include":["30 days","receipt"],"must_not_include":["no returns"],"severity":"critical"}

Expected evidence: 0.90 overall and 1.00 for critical policy cases. Treat the printout as a claim about this fixture, not as proof that the toolchain merely started.

For the release eval gate in CI, a decorative win often looks like a clean run that never checks pass/fail exit code; metric deltas vs baseline; seed recorded. Write the metric down now so later pages cannot redefine success after the fact. Also note the operational threat you will eventually gate on: editing golden expected answers to greenwash a bad candidate.

For eval-in-code, write the claim on a sticky note in this exact shape: “Given golden cases, metric functions, threshold config, candidate outputs, the CI eval release gate will …”. Fill the ellipsis with the observable part of: gate exits non-zero on regression and zero on the pinned baseline. Tape the baseline beside it: last known good release metrics. If someone later replaces your metric with a vibe check, the sticky note is how you push back.

Also sketch the one-sentence user story: a person uses this output to fail a candidate release when golden-task metrics regress past a pinned threshold. If that sentence needs a dashboard, a model zoo, or five services, the lab scope is too wide—shrink the fixture (golden JSONL + threshold.yaml) until the story fits on one screen.

Decide now whether live network calls are allowed on page 1. For this lab they usually are not; inventory and contracts should run offline against golden JSONL + threshold.yaml. Note the metric you will eventually require (pass/fail exit code; metric deltas vs baseline; seed recorded) so page 4 cannot invent a softer target. The characteristic failure to keep in mind is flaky eval that flips without seed control, or soft-fail that never blocks merge.

At the experiment brief stage for eval-in-code, the job is narrower than finishing a product demo. You are creating one progressive evidence piece about golden JSONL + threshold.yaml that later pages inherit without redefining success. Keep that fixture small enough to inspect by hand, keep outputs copy-pasteable as text, and refuse to narrate this baseline as if it were a production SLA: last known good release metrics.

For this page specifically, success looks like a falsifiable claim and baseline written before coding while still centering the user decision to fail a candidate release when golden-task metrics regress past a pinned threshold. If you cannot point to a file, command, or assertion that proves that for the CI eval release gate, stay on this page instead of advancing.

How-to: ship agent with eval gate · Glossary: eval set

Next