Evaluate retrieval and answers separately

Start with representative user questions, not synthetic trivia alone. Include common questions, important rare cases, ambiguous wording, exact codes, multi-part requests, comparisons, questions with no answer, and adversarial requests. Sample real production patterns only with appropriate privacy controls.

For each item, record more than a preferred final answer. Useful labels include:

• the authoritative source and required chunk or passage;
• acceptable alternative evidence;
• filters such as region, audience, and date;
• key facts a correct answer must include;
• facts or inferences it must not claim;
• whether it should clarify, answer partly, or refuse;
• expected citation support.

Source owners or domain experts should review high-impact labels. If the gold evidence is wrong or outdated, the evaluation rewards the wrong behavior.

Split a stable regression set from a development set used during tuning. Otherwise repeated tuning can overfit the exact examples that report progress.

Retrieval evaluation ignores answer prose and asks whether the right evidence was selected.

Recall at k measures how often at least one required relevant item appears in the first k results. High recall matters because generation cannot use evidence it never sees. Precision at k measures how much of the selected set is relevant. High precision reduces noise and conflicting distractors.

Also track rank-sensitive measures such as mean reciprocal rank when the first relevant result should appear early. For questions requiring several pieces of evidence, measure whether all required pieces are covered, not merely whether one appeared.

Slice results by query type, source, language, region, recency, document format, and access group. An overall 90% recall can hide severe failure on tables or a specific country policy.

Inspect failures qualitatively. Scores tell you where to look; ranked chunks reveal whether the cause is parsing, chunking, filters, embeddings, keyword matching, top-k, or re-ranking.

Answer evaluation should score distinct properties:

• correctness: are claims accurate relative to authoritative sources?
• groundedness: are factual claims supported by supplied evidence?
• completeness: are required parts of the question answered?
• citation support: does each citation justify its nearby claim?
• scope: are region, audience, product, and date preserved?
• abstention: does the system refuse or clarify when evidence is insufficient?
• usefulness: is the answer understandable and actionable without unnecessary text?

Exact string matching is often too rigid because several phrasings can be correct. Use deterministic checks for labels, links, required terms, and structured fields; use expert review or carefully validated model-based grading for semantic judgments. Calibrate automated graders against human decisions and periodically recheck them.

A fluent answer should not earn a high score if it is unsupported. Weight safety-critical properties accordingly.

Change one major lever at a time and record the configuration. Retrieval levers include parser, chunk size and overlap, metadata, embedding model, keyword weighting, candidate k, filters, query rewriting, and re-ranking. Generation levers include evidence ordering, prompt instructions, context budget, model choice, decoding settings, citation format, and output validation.

If retrieval recall is low, changing the answer model is unlikely to help. If retrieval is strong but citations do not support claims, focus on prompting, source framing, model behavior, or validation. If both offline stages look good but users complain, the evaluation set may not represent real requests or the interface may obscure caveats.

Compare a candidate against a baseline on the same frozen set. Report quality, latency, token usage, and cost together. A re-ranker that gains one point but triples latency may not be the right product tradeoff.

Do not choose k from intuition alone. Evaluate recall and precision across values such as 3, 5, 10, and 20. Retrieval recall often rises as k grows, while answer quality may eventually fall because extra context introduces noise.

Remember that candidate k and final context k are different. Retrieving 40 candidates for a fast re-ranker does not mean all 40 should enter the prompt. A useful experiment holds the candidate pool steady while varying final k, then separately tests candidate coverage.

Measure context duplication and source diversity as well. Ten chunks from one repeated footer are not ten useful pieces of evidence.

An answerable-only dataset encourages the model to answer everything. Include unanswerable questions, insufficient regional context, restricted sources, obsolete policies, and equally authoritative conflicts. Score both false refusals and unsafe answers.

For prompt injection tests, place hostile instructions inside retrieved test documents and verify that the system treats them as data. If the application has tools, test that no retrieved text can expand permissions or trigger actions.

For freshness tests, replace a source version and confirm that old chunks disappear, the new evidence retrieves, citations point to the new locator, and regression answers change where expected.

Offline evaluation is necessary but cannot model every query. In production, monitor retrieval-empty rate, score distributions, source mix, stale-source age, refusal rate, citation validation failures, latency, errors, and carefully sampled quality reviews.

User feedback is a signal, not ground truth. A user may dislike a correct policy answer or approve a pleasantly worded wrong one. Connect feedback to traces, have experts review important reports, and categorize the failed stage.

Roll out major index or prompt changes gradually when possible. Keep a rollback path and compare against a stable baseline. Add confirmed incidents and new query patterns to the regression set so the system learns operationally rather than repeating failures.

Write a release decision

A release gate should name minimum performance for critical slices, not only an average. For example: required-policy recall at 5 above a target for every region, zero cross-role access leaks in the security set, citation support above a target, and correct abstention on all high-risk unanswerable cases.

Thresholds depend on impact and should be agreed with owners. The important practice is explicit evidence: what improved, what regressed, which risks remain, and who accepts them.