Define the RAG quality audit report input contract

The accepted input remains: golden questions, expected evidence IDs, candidate answers with citations. Keep parsing and normalization in functions that do not score, train, or call a model. That split lets a test fail the boundary without blaming the core logic. The user-facing decision stays: turn golden questions, expected evidence IDs, and citation checks into a release artifact.

required={'q','evidence_id'}
row={'q':'When open?','evidence_id':'hours'}
assert required<=set(row)
print('gold row contract ok')

Expected evidence: gold row contract ok. If the contract is silent on a bad value, later debugging will look like an algorithm bug when it is really a data bug.

Write one assertion for a neighboring valid input to the RAG quality audit report so tightening the boundary does not over-reject. Document field names and types the way a teammate would need them on day two of rag-quality-audit—not as comments you plan to delete.

List every field in gold set with expected evidence IDs + candidate run output and mark each as required, optional, or forbidden. Required fields must fail loudly when missing; optional fields need defaults you can quote in a test; forbidden fields (secrets, raw PII, path escapes) must never be accepted silently. This list is the contract for the RAG quality audit report.

Add one sentence about encoding, units, or timezones if relevant to golden questions, expected evidence IDs, candidate answers with citations. Contracts that ignore units create “correct” programs that still ship wrong decisions when someone tries to turn golden questions, expected evidence IDs, and citation checks into a release artifact.

Write the error string you want for the most likely bad input. Prefer ValueError('threshold out of range')-style messages over generic invalid input. The contract’s job is to make scoring fluency instead of citation support, or mixing docs versions in one report harder to confuse with a model or algorithm bug later.

At the data contract stage for rag-quality-audit, the job is narrower than finishing a product demo. You are creating one progressive evidence piece about gold set with expected evidence IDs + candidate run output that later pages inherit without redefining success. Keep that fixture small enough to inspect by hand, keep outputs copy-pasteable as text, and refuse to narrate this baseline as if it were a production SLA: previous docs_version audit score.

For this page specifically, success looks like malformed inputs rejected with field-named errors while still centering the user decision to turn golden questions, expected evidence IDs, and citation checks into a release artifact. If you cannot point to a file, command, or assertion that proves that for the RAG quality audit report, stay on this page instead of advancing.

Glossary: faithfulness · Glossary: recall@k · Cheatsheet: RAG quality · How-to: evaluate RAG quality

Previous · Next