Measure whether the RAG quality audit report works

Translate the claim into assertions or a tiny eval harness. The metric to protect is: citation precision, unsupported answer rate, abstention correctness. Always record the denominator (how many cases) beside any rate. A percentage without a denominator is marketing, not measurement.

report={'citation_precision':0.5,'unsupported':0,'abstain_correct':1,'docs_version':'faq-2026-07-18'}
assert report['docs_version']
print(report)

Expected evidence: audit metrics with version. A passing assertion proves only the behavior it names; broader usefulness still needs the chapter’s full limits.

Be explicit: beating the baseline (previous docs_version audit score) on this fixture does not prove behavior under scoring fluency instead of citation support, or mixing docs versions in one report. Label observations separately from conclusions so the next page inherits honest evidence about the RAG quality audit report.

Compute citation precision, unsupported answer rate, abstention correctness with the denominator written beside the rate every time. For this chapter, the evaluation set is intentionally tiny; that is allowed only if you say so in the evidence. Compare against previous docs_version audit score before celebrating.

Add one negative case aimed at scoring fluency instead of citation support, or mixing docs versions in one report. A suite with only happy cases cannot protect the RAG quality audit report when the characteristic failure appears in review.

If a check is expensive or flaky, shrink it until it is deterministic on gold set with expected evidence IDs + candidate run output. Flaky green builds teach the team to ignore gates. Record what this page does not prove so security-ops and mastery-ship inherit honest limits.

At the evaluation stage for rag-quality-audit, the job is narrower than finishing a product demo. You are creating one progressive evidence piece about gold set with expected evidence IDs + candidate run output that later pages inherit without redefining success. Keep that fixture small enough to inspect by hand, keep outputs copy-pasteable as text, and refuse to narrate this baseline as if it were a production SLA: previous docs_version audit score.

For this page specifically, success looks like metrics with explicit denominators and a negative case while still centering the user decision to turn golden questions, expected evidence IDs, and citation checks into a release artifact. If you cannot point to a file, command, or assertion that proves that for the RAG quality audit report, stay on this page instead of advancing.

Glossary: faithfulness · Glossary: recall@k · Cheatsheet: RAG quality · How-to: evaluate RAG quality

Previous · Next