Retrieval, query flow, and top-k

The request reaching retrieval may need more than raw user text. Conversation follow-ups such as “What about contractors?” are ambiguous without the prior subject. A query-rewrite step can produce a standalone search query such as “Do contractors receive volunteer leave?” This rewrite should preserve intent, not invent details.

Applications may also extract filters from trusted context: the user’s organization, region, product, language, or document permissions. Keep trusted filters separate from free-form model output. A user should not gain access to executive documents by typing “department: executive.”

Some questions should be decomposed. “Compare parental leave in Canada and France” may work better as two scoped retrievals followed by comparison. Query expansion can add known synonyms, but excessive expansion introduces unrelated material. Log the original query and every transformed query so failures remain explainable.

Vector retrieval embeds the query and looks for nearby chunk vectors. It works well when wording differs: “carry unused days” can match “leave does not roll over.” Keyword retrieval works well for exact strings such as error codes, policy numbers, legal citations, and product names.

Hybrid search combines both. One path retrieves semantic neighbors, another retrieves lexical matches, and the system merges their candidates. A re-ranker may then examine each query–chunk pair more carefully and produce a better final order.

Choose the simplest method that passes real tests. A small, consistently worded help center may work with keyword search. A mixed technical library often benefits from hybrid retrieval. Adding a re-ranker can improve ordering but also increases latency and cost.

Vector search often returns a similarity or distance score, but the number is not a universal probability of relevance. A score of 0.78 may be strong for one model and corpus but weak for another. Score distributions also change with query type.

Use labeled examples to set any threshold. If no candidate clears a validated minimum, the system should avoid pretending it found evidence. Thresholds can support abstention, but a single global cutoff may be too crude for a varied collection.

Metadata filtering and ranking solve different problems. Filtering decides what is eligible, such as “current Canadian HR policies the user may access.” Ranking decides which eligible chunks best match the question. Do not rely on similarity to enforce scope or security.

In top-k retrieval, k is the number of highest-ranked items selected at a stage. A system might retrieve 30 inexpensive candidates, re-rank them, and pass the best 5 to the model. Both numbers are top-k choices at different stages.

If k is too small, the retriever may omit an important exception or second document. This lowers recall. If k is too large, the prompt fills with duplicates, weak matches, and conflicting details. This can lower precision, raise cost, slow responses, and distract the model.

More context is not automatically better. The goal is enough relevant evidence, not the maximum number of tokens. Tune k with representative questions and measure whether required evidence appears in the selected set.

For a simple factual answer, three focused chunks may be enough. A comparison across policies may require more. Some systems choose k dynamically based on question type, but that added logic must also be evaluated.

Nearest-neighbor results often contain overlapping chunks from the same document. Five nearly identical passages create the illusion of strong evidence while crowding out alternatives. Deduplicate by stable chunk or source IDs, and consider diversity rules that limit repeated material from one section.

At the same time, preserve context needed to interpret a hit. If a chunk says “This benefit is unavailable during probation,” the parent heading should identify which benefit. A retriever can add neighboring chunks or a parent section after finding a precise child chunk. This is sometimes called parent-child or small-to-big retrieval.

Be careful: automatically expanding every hit can bring large amounts of noise. Expansion should be bounded and tested against prompt limits.

A robust retrieval result contains the expected evidence, correct scope, source metadata, and no unauthorized content. It should also signal uncertainty. Useful signals include low scores, disagreement between retrieval methods, missing required document types, or strong results from conflicting versions.

The application can respond in several ways:

• proceed with the selected evidence;
• run a second retrieval with a rewritten query;
• ask the user a clarifying question;
• route to a specialized index or tool;
• refuse because no adequate evidence was found.

Repeated retrieval is not always progress. Limit retries and record which strategy changed. Otherwise the system may spend time searching the same weak corpus in slightly different ways.

For each test question, display the ranked chunks, scores, filters, source versions, and final selected set. Ask: Is the answerable evidence present? Is it high enough in the ranking? Are irrelevant results entering because of chunking, embedding, filters, or query rewriting?

If the right chunk ranks at position 18 and only 5 chunks reach the model, the problem is retrieval. Rewriting the answer prompt cannot repair missing evidence. This separation is one of the most valuable habits in RAG engineering.