Build the first working CI eval release gate

Build only what the claim requires: gate exits non-zero on regression and zero on the pinned baseline. Prefer boring, deterministic code over frameworks you cannot yet explain. Run the path twice; identical output on this fixture is a feature, not a lack of creativity.

def run_case(case, *, release, trace_id):
    # Adapters are injected in production; the orchestration owns policy.
    answer = staging_agent(case["question"], request_id=trace_id)
    reasons = evaluate(answer if "answer" in locals() else locals().get("result", locals().get("decision", locals().get("job", locals().get("choice")))))
    return {"passed": not reasons, "reasons": reasons,
            "release": release, "trace_id": trace_id}

result = run_case({"id":"refund-window","question":"Can I return headphones after 20 days?","must_include":["30 days","receipt"],"must_not_include":["no returns"],"severity":"critical"}, release="candidate", trace_id="trace-42")
assert result["trace_id"] == "trace-42"

Expected evidence: the new prompt omits the 30-day refund window while producing fluent prose. Read each printed intermediate as part of the argument that the path works—not as decoration.

Narrate the journey from raw input to result for a single example from golden JSONL + threshold.yaml. If you cannot name an intermediate, the implementation is still too opaque for this lab. Only after this path is solid should you generalize data sources or UI.

While implementing the CI eval release gate, print or log at least three intermediates that map to the claim (gate exits non-zero on regression and zero on the pinned baseline). Good intermediates are values a teammate could recompute with a calculator or diff. Bad intermediates are framework traces you cannot explain.

Re-run with golden JSONL + threshold.yaml twice. If the second run differs, either the path is nondeterministic (document the seed) or you have hidden global state—both are lab bugs until named.

Stop adding features once the path supports fail a candidate release when golden-task metrics regress past a pinned threshold. Extra UI, extra tools, or extra models belong in later chapters. The mastery bar for this page is simply: a deterministic end-to-end path with intermediates.

At the implementation stage for eval-in-code, the job is narrower than finishing a product demo. You are creating one progressive evidence piece about golden JSONL + threshold.yaml that later pages inherit without redefining success. Keep that fixture small enough to inspect by hand, keep outputs copy-pasteable as text, and refuse to narrate this baseline as if it were a production SLA: last known good release metrics.

For this page specifically, success looks like a deterministic path with printed intermediates while still centering the user decision to fail a candidate release when golden-task metrics regress past a pinned threshold. If you cannot point to a file, command, or assertion that proves that for the CI eval release gate, stay on this page instead of advancing.

How-to: ship agent with eval gate · Glossary: eval set

Previous · Next