RAG quality audit
Instrument the RAG quality audit report
Page 6 adds signals that distinguish bad input from component failure in the versioned RAG quality audit report.
1Learn the idea
Read
Emit stage signals
Instrument the versioned RAG quality audit report so a run records enough structure to debug offline: counts, latency if relevant, pass/fail of citation precision, unsupported answer rate, abstention correctness, and a stable stage name. Redact secrets and raw credentials from every event.
Read
Emit and assert
import json
print(json.dumps({'docs_version':'faq-2026-07-18','per_q':[{'q':'Wifi?','want':'wifi','got':'parking','ok':False}]}))
Expected evidence: per-question audit trace. Prefer JSON or structured text you can grep in CI over prose logs for rag-quality-audit.
Read
Lock signals with a regression test
Turn one historical failure—especially scoring fluency instead of citation support—into a test that fails if the signal disappears for the RAG quality audit report. Observability without a failing test is optional decoration; observability with a test is part of the rag-quality-audit artifact.
Read
Lab notebook: signal schema
Draft a three-field event for the RAG quality audit report: stage, ok, and one domain field derived from citation precision, unsupported answer rate, abstention correctness. Add fixture_id or docs_version when content can change. Explicitly list fields that must never appear (tokens, passwords, raw prompts) because publishing golden questions that contain real customer PII is in scope for this lab.
Wire one assertion that fails if the RAG quality audit report event is missing after a run. Observability that cannot fail a test will not survive contact with a busy rag-quality-audit repository.
Read
Worked judgment
Imagine a teammate opens only your event stream after a bad deploy. Could they tell whether gold set with expected evidence IDs + candidate run output was wrong, whether scoring fluency instead of citation support, or mixing docs versions in one report returned, or whether publishing golden questions that contain real customer PII slipped through? If not, rename fields until those three stories are distinguishable.
Read
Why this stage matters for the RAG quality audit report
At the testing and observability stage for rag-quality-audit, the job is narrower than finishing a product demo. You are creating one progressive evidence piece about gold set with expected evidence IDs + candidate run output that later pages inherit without redefining success. Keep that fixture small enough to inspect by hand, keep outputs copy-pasteable as text, and refuse to narrate this baseline as if it were a production SLA: previous docs_version audit score.
For this page specifically, success looks like a structured event schema locked by a test while still centering the user decision to turn golden questions, expected evidence IDs, and citation checks into a release artifact. If you cannot point to a file, command, or assertion that proves that for the RAG quality audit report, stay on this page instead of advancing.
Glossary: faithfulness · Glossary: recall@k · Cheatsheet: RAG quality · How-to: evaluate RAG quality
Go deeper
Before you start
Why this matters
Write the single log line or metric event that would tell you whether a bad result came from input vs implementation for the RAG quality audit report. If your line could not tell them apart, redesign it before coding.
In the wild
See how this idea shows up as a product and a company — then come back to the lesson. Skills transfer across vendors.
Related lessons
Check your understanding
Page assessment
Answer from memory. Completion is saved from this evidence, not from opening the next page.
All responses are required.