Debug scoring fluency instead of citation support in the RAG quality audit report

Do not start with a speculative fix for the RAG quality audit report. Force the failure on purpose, save the before output, then change one cause at a time. Retries are allowed only for transient conditions—not for bad input that will fail forever on rag-quality-audit.

fluency_score=0.99; citation_precision=0.5
if fluency_score>0.9 and citation_precision<0.8:
    print('FAIL: fluent but weakly cited')

Expected evidence: FAIL: fluent but weakly cited. If you cannot reproduce on demand, you do not yet control the failure mode for rag-quality-audit.

After repair, rerun the exact reproduction command. Keep the failing fixture as a regression seed for the observability page. For the versioned RAG quality audit report, remember the claim you are restoring: audit records hit/miss per question with docs_version and fails release under threshold.

Store a one-command reproduction for: scoring fluency instead of citation support, or mixing docs versions in one report. The command should use gold set with expected evidence IDs + candidate run output or a minimal mutant of it. Paste the failing output into notes/failure-before.txt (or your shell scrollback as copied text). After the fix, paste notes/failure-after.txt and keep both.

Retries belong only on transient faults. If the failure is bad input, a bad allowlist, or a logic bug in the RAG quality audit report, retrying will amplify cost without repairing trust around turn golden questions, expected evidence IDs, and citation checks into a release artifact.

Classify the failure as prevent, detect, contain, or recover—using this lab’s language, not a generic poster. For rag-quality-audit, the first fix should usually be detect+prevent at the boundary, because scoring fluency instead of citation support, or mixing docs versions in one report is cheaper to stop early than to explain in production prose.

At the debugging stage for rag-quality-audit, the job is narrower than finishing a product demo. You are creating one progressive evidence piece about gold set with expected evidence IDs + candidate run output that later pages inherit without redefining success. Keep that fixture small enough to inspect by hand, keep outputs copy-pasteable as text, and refuse to narrate this baseline as if it were a production SLA: previous docs_version audit score.

For this page specifically, success looks like before/after evidence for the characteristic failure while still centering the user decision to turn golden questions, expected evidence IDs, and citation checks into a release artifact. If you cannot point to a file, command, or assertion that proves that for the RAG quality audit report, stay on this page instead of advancing.

Glossary: faithfulness · Glossary: recall@k · Cheatsheet: RAG quality · How-to: evaluate RAG quality

Previous · Next