Reference · How-to · ~6 min
How to run golden-task eval locally
Last updated
Score a prompt change before opening a PR.
Score a prompt change before opening a PR.
Steps
1. Create `eval/golden.json` with 10 tasks
2. Script calls staging model with fixed prompt template
3. Score with rules: must_include, must_not_include, optional LLM judge
4. Print pass rate; exit 1 if below threshold
5. Commit golden file — treat changes like code review