Trace a worked example

You operate a technical support search assistant. A teammate proposes a change that sounds beneficial, but you require a trace connecting configuration to evidence. Here is the observed run:

For “router XG-410 red status after update,” lexical search ranks the exact model manual first but misses the paraphrased symptom article; dense search finds the article but ranks an XG-400 page above it. RRF merges ranks, then a cross-encoder puts the XG-410 firmware article first. recall@5 rises from 72% and 81% alone to 93% hybrid.

First identify the input and scope. Which user, segment, document, image, query, hardware profile, or task was involved? Next record the exact configuration: model or checkpoint, prompt, index, context policy, sampler, thresholds, and tool versions that matter for hybrid search for RAG. Then preserve the intermediate artifact that explains the result. Finally attach the user-visible output and measured consequence.

Write the trace as a sequence rather than a conclusion:

request + configuration
  -> intermediate evidence
  -> model or policy decision
  -> validation / fusion / routing
  -> user-visible action
  -> measured outcome

This format prevents hindsight from collapsing several stages into “AI error.” It also exposes where a deterministic check could have stopped propagation.

Use absolute counts alongside percentages. If success falls from 78 of 100 to 62 of 100, that is a 16 percentage-point decrease, not merely “16% worse.” If cost rises from $0.006 to $0.018 for one million requests, variable spend rises from $6,000 to $18,000. If a sample contains only ten cases from a critical language, one miss moves its rate by ten points; collect more evidence before claiming stability.

Build judged query-document pairs covering exact codes, paraphrases, mixed queries, and no-answer cases. Compare recall@k, MRR or nDCG, duplicate rate, filter correctness, latency, and end-to-end citation support. Ablate lexical-only, dense-only, fusion, and reranking. Pick one primary metric and list gates separately. Do not average a privacy breach, severe unsafe action, or failed authorization with stylistic quality.

Generate at least three explanations: input mix changed; a component configuration changed; or measurement changed. Then propose a discriminating test for each. Replay the same cases on old and new configurations, compare intermediate artifacts, and rescore both with the same rubric. This controls more variables than debating outputs by eye.

Combining raw BM25 and cosine scores gives one engine accidental dominance; filters differ between paths; duplicate chunks fill top-k; query expansion deletes product codes; stale indexes disagree; and a reranker trained on web passages mishandles logs or tables. The likely failure should match the earliest divergent artifact. If it does not, revise the hypothesis.

Choose among keep, roll back, canary, route, or collect more data. State the owner and deadline. A rollback restores safety but does not explain root cause; preserve the failed configuration for offline reproduction. A successful fix adds the case to a regression set and updates the runbook.

The expert habit is modest: claim only what the trace supports. One run can demonstrate a mechanism, not a universal advantage. A coherent sequence with inspectable evidence teaches more than a polished before-and-after screenshot.