Set release boundaries for the RAG quality audit report

Operational risks for the versioned RAG quality audit report center on publishing golden questions that contain real customer PII, plus the earlier failure mode (scoring fluency instead of citation support, or mixing docs versions in one report). Safety lives in executable gates, allowlists, redaction, and a named owner—not in a warning paragraph under an unsafe function.

gold_q='When is the cafe open?'
assert '@' not in gold_q and 'ssn' not in gold_q.lower()
print({'gold_contains_pii_marker':False})

Expected evidence: gold_contains_pii_marker False. A failed assertion means stop, investigate, and do not publish the RAG quality audit report.

Name who can disable the feature, what data is retained, and how to roll back to the last known good artifact. Pin the reviewed configuration (versions, thresholds, allowlists) so “what shipped” is reconstructable for rag-quality-audit.

Write the release blocker as a predicate, not a feeling: “Do not ship the RAG quality audit report if publishing golden questions that contain real customer PII.” Pair it with a passing control that shows the reviewed configuration still works for gold set with expected evidence IDs + candidate run output. Name an owner and a rollback handle (git tag, docs_version, previous image).

Security pages must not paste the happy-path demo. If your gate code looks like the implementation page, replace it with a deny/allow check aimed at publishing golden questions that contain real customer PII.

State the data retention rule in one line (what is stored, for how long, who can read it). Then state the kill switch (env flag, config pin, or feature owner). The RAG quality audit report is not shippable without both, even when citation precision, unsupported answer rate, abstention correctness looks healthy.

At the safety and operations stage for rag-quality-audit, the job is narrower than finishing a product demo. You are creating one progressive evidence piece about gold set with expected evidence IDs + candidate run output that later pages inherit without redefining success. Keep that fixture small enough to inspect by hand, keep outputs copy-pasteable as text, and refuse to narrate this baseline as if it were a production SLA: previous docs_version audit score.

For this page specifically, success looks like an executable deny gate for the lab-specific threat while still centering the user decision to turn golden questions, expected evidence IDs, and citation checks into a release artifact. If you cannot point to a file, command, or assertion that proves that for the RAG quality audit report, stay on this page instead of advancing.

Glossary: faithfulness · Glossary: recall@k · Cheatsheet: RAG quality · How-to: evaluate RAG quality

Previous · Next