Define the CI eval release gate input contract

The accepted input remains: golden cases, metric functions, threshold config, candidate outputs. Keep parsing and normalization in functions that do not score, train, or call a model. That split lets a test fail the boundary without blaming the core logic. The user-facing decision stays: fail a candidate release when golden-task metrics regress past a pinned threshold.

from dataclasses import dataclass
@dataclass(frozen=True)
class EvalGate:
    metric: str
    threshold: float
    seed: int
print(EvalGate('task_pass_rate', 0.9, 7))

Expected evidence: eval gate contract. If the contract is silent on a bad value, later debugging will look like an algorithm bug when it is really a data bug.

Write one assertion for a neighboring valid input to the CI eval release gate so tightening the boundary does not over-reject. Document field names and types the way a teammate would need them on day two of eval-in-code—not as comments you plan to delete.

List every field in golden JSONL + threshold.yaml and mark each as required, optional, or forbidden. Required fields must fail loudly when missing; optional fields need defaults you can quote in a test; forbidden fields (secrets, raw PII, path escapes) must never be accepted silently. This list is the contract for the CI eval release gate.

Add one sentence about encoding, units, or timezones if relevant to golden cases, metric functions, threshold config, candidate outputs. Contracts that ignore units create “correct” programs that still ship wrong decisions when someone tries to fail a candidate release when golden-task metrics regress past a pinned threshold.

Write the error string you want for the most likely bad input. Prefer ValueError('threshold out of range')-style messages over generic invalid input. The contract’s job is to make flaky eval that flips without seed control, or soft-fail that never blocks merge harder to confuse with a model or algorithm bug later.

At the data contract stage for eval-in-code, the job is narrower than finishing a product demo. You are creating one progressive evidence piece about golden JSONL + threshold.yaml that later pages inherit without redefining success. Keep that fixture small enough to inspect by hand, keep outputs copy-pasteable as text, and refuse to narrate this baseline as if it were a production SLA: last known good release metrics.

For this page specifically, success looks like malformed inputs rejected with field-named errors while still centering the user decision to fail a candidate release when golden-task metrics regress past a pinned threshold. If you cannot point to a file, command, or assertion that proves that for the CI eval release gate, stay on this page instead of advancing.

How-to: ship agent with eval gate · Glossary: eval set

Previous · Next