Debug flaky eval that flips without seed control in the CI eval release gate

Do not start with a speculative fix for the CI eval release gate. Force the failure on purpose, save the before output, then change one cause at a time. Retries are allowed only for transient conditions—not for bad input that will fail forever on eval-in-code.

def execute_with_policy(operation, *, attempts=3):
    for attempt in range(attempts):
        try:
            return operation()
        except (TimeoutError, ConnectionError):
            if attempt == attempts - 1:
                raise
        except (PermissionError, ValueError):
            raise  # deterministic or unsafe: never retry

def test_known_failure_is_contained():
    case = {"id":"refund-window","question":"Can I return headphones after 20 days?","must_include":["30 days","receipt"],"must_not_include":["no returns"],"severity":"critical"}
    answer = staging_agent(case["question"], request_id=trace_id)
    assert "30 days" in answer and "no returns" not in answer

Expected evidence: the new prompt omits the 30-day refund window while producing fluent prose. If you cannot reproduce on demand, you do not yet control the failure mode for eval-in-code.

After repair, rerun the exact reproduction command. Keep the failing fixture as a regression seed for the observability page. For the release eval gate in CI, remember the claim you are restoring: gate exits non-zero on regression and zero on the pinned baseline.

Store a one-command reproduction for: flaky eval that flips without seed control, or soft-fail that never blocks merge. The command should use golden JSONL + threshold.yaml or a minimal mutant of it. Paste the failing output into notes/failure-before.txt (or your shell scrollback as copied text). After the fix, paste notes/failure-after.txt and keep both.

Retries belong only on transient faults. If the failure is bad input, a bad allowlist, or a logic bug in the CI eval release gate, retrying will amplify cost without repairing trust around fail a candidate release when golden-task metrics regress past a pinned threshold.

Classify the failure as prevent, detect, contain, or recover—using this lab’s language, not a generic poster. For eval-in-code, the first fix should usually be detect+prevent at the boundary, because flaky eval that flips without seed control, or soft-fail that never blocks merge is cheaper to stop early than to explain in production prose.

At the debugging stage for eval-in-code, the job is narrower than finishing a product demo. You are creating one progressive evidence piece about golden JSONL + threshold.yaml that later pages inherit without redefining success. Keep that fixture small enough to inspect by hand, keep outputs copy-pasteable as text, and refuse to narrate this baseline as if it were a production SLA: last known good release metrics.

For this page specifically, success looks like before/after evidence for the characteristic failure while still centering the user decision to fail a candidate release when golden-task metrics regress past a pinned threshold. If you cannot point to a file, command, or assertion that proves that for the CI eval release gate, stay on this page instead of advancing.

How-to: ship agent with eval gate · Glossary: eval set

Previous · Next