Chunks, documents, and queries

A document embedding represents a whole item: an article, product description, support ticket, or report. It is useful when documents are short and focused or when the task is broad classification and recommendation. A movie synopsis can be represented as one unit for “find films like this.” A short product card can work for catalog discovery.

Long, multi-topic documents create dilution. A single fixed-length vector compresses all subjects into one point. A rare but important clause may contribute too little to make the document a near neighbor for a precise query. The source might rank reasonably for “employee policies” but poorly for “parental leave eligibility after six months.”

Document vectors can still complement chunks. First retrieve likely documents, then search passages within them. This hierarchical pattern narrows work and preserves document-level context, but it can miss an answer if the first stage excludes the correct document. Evaluate both stages.

A chunk is a retrieval unit cut from a larger source. Good chunks are coherent, self-contained enough to judge, and small enough to focus on one idea. Boundaries should follow structure when possible: headings, paragraphs, list items, code blocks, or transcript turns.

Fixed token windows are easy to implement, but arbitrary cuts can split a definition from its exception. Structure-aware splitting usually produces more understandable results. A chunk should include inherited context such as the document title and section heading. “It lasts 30 days” is weak alone; “Refund window — unopened products: It lasts 30 days from delivery” is usable.

There is no universal best chunk size. Smaller chunks sharpen matching and reduce prompt cost, but may lose context. Larger chunks preserve context but blend topics and consume more generation tokens. Test several settings on representative queries. Measure retrieval and final task quality, not merely the number of chunks.

Chunk overlap repeats some text between adjacent windows. If chunks contain 300 tokens with 50-token overlap, a statement crossing one boundary may remain intact in at least one chunk. Overlap can improve recall for prose without clear structure.

Overlap has costs. It increases embedding volume, index size, and near-duplicate results. The top five may contain five versions of the same paragraph, reducing evidence diversity. Deduplicate by source and position, merge adjacent retrieved chunks, or cap results per document.

More overlap is not always safer. If every chunk repeats a long preamble, the repeated language can dominate similarity. Prefer meaningful boundaries first, then use modest overlap for boundary insurance. For tables, code, and policy clauses, custom parsers may be more important than a generic overlap percentage.

Track chunk IDs and character or token offsets. They allow the interface to link back to exact source locations, reconstruct neighboring context, update changed regions, and remove deleted content reliably.

A query embedding represents what the user is seeking, not necessarily a passage written in the same style. Queries are often short, misspelled, conversational, or underspecified. Documents may be formal and technical. Retrieval models trained on query–passage pairs are designed to bridge this asymmetry.

Follow model-specific instructions. Some models expose separate query and document encoders; others require prefixes such as query: and passage:. Using passage mode for both because the vectors have the same shape can harm ranking.

Query expansion can add synonyms, resolve abbreviations, or incorporate safe context. “PTO carryover?” might become “paid time off carryover policy.” But expansion must not silently decide an ambiguous meaning. In an engineering knowledge base, PTO might mean “power take-off.” Search can issue multiple interpretations or ask a clarifying question.

Keep metadata filters separate from semantic meaning. If a user selects “Canada” and “2026 policy,” use structured filters rather than hoping those words steer the vector strongly enough. Filters enforce exact constraints; embeddings rank semantically relevant candidates inside them.

One item can have several useful representations. A product may have a vector for its title and summary, vectors for detailed sections, and perhaps an image vector in a multimodal space. A support article can have a synthetic question vector plus passage vectors. Search can query one or more fields and combine results.

Synthetic questions can help when users phrase needs differently from documentation. For a passage about thermal throttling, generated questions might include “Why does assistance stop uphill?” The technique adds maintenance and quality risks: poor generated questions can bias retrieval or promise content the passage does not contain. Preserve provenance and evaluate against raw passage embeddings.

Late-interaction models retain multiple token-level vectors rather than compressing a passage to one vector. They can match details more precisely but require more storage and specialized indexes. This illustrates a broader tradeoff: richer representations can improve quality while raising cost and complexity.

Define an explicit contract for every indexed unit:

• source identity, version, and permissions;
• parsing and chunking rules;
• text actually sent to the model;
• embedding model, mode, and version;
• vector dimensions and normalization;
• metadata and index namespace;
• deletion and refresh behavior.

Version the contract. If you change the heading prefix, chunk size, or model, vectors may no longer be comparable with old ones. Rebuild into a new collection and evaluate before switching traffic.

Inspect retrieved chunks directly. Ask whether each can be understood, cited, and connected to its source. A metric may say the correct document was retrieved while the returned chunk omits the sentence needed to answer. That is a chunking failure, not necessarily a model failure.