From documents to a searchable index

Ingestion starts by choosing authoritative sources, not by scraping everything available. For every source, record an owner, stable identifier, version or update time, access rules, and deletion policy. These details become part of the retrieval system’s correctness.

Connectors may read a document drive, help center, wiki, object store, or database. Each format needs parsing. A PDF parser must recover headings, paragraphs, lists, and tables in sensible reading order. A web parser should remove menus, cookie notices, and repeated footers. Optical character recognition may be needed for scans, but its output should be checked because a misread number can become a confident wrong answer.

Keep the original source and a normalized text representation. Normalization can standardize whitespace and headings, but it should not silently rewrite meaning. Log parse failures rather than indexing empty or scrambled text.

Embedding an entire handbook as one item would make every question match the same huge block. Splitting every sentence separately would lose the surrounding conditions that make policy language meaningful. Chunking aims for a middle unit: small enough to retrieve precisely, large enough to stand on its own.

A practical starting range is a few hundred tokens, often around 300–800, but there is no universal best number. Respect natural boundaries such as headings, paragraphs, procedures, and table rows. Keep a short heading path with each chunk, for example Benefits > Leave > Volunteer days. Use limited overlap when a thought crosses a boundary, while remembering that too much overlap creates duplicate results and wastes prompt space.

Special content deserves special handling. Keep a short definition with the term it defines. Preserve steps in one procedure when later steps depend on earlier ones. Convert a table so row and column labels remain understandable. Do not separate an exception from the rule it limits.

The text is only part of a chunk. Useful metadata may include:

• source ID, title, and canonical URL;
• section heading, page, or paragraph locator;
• document version and effective date;
• department, product, region, or language;
• access-control labels;
• content type and ingestion timestamp.

Metadata supports filtering before similarity search and enables citations afterward. If a user is asking about the Canadian policy, a region filter can prevent a semantically similar United States policy from ranking first. If the application cannot reconstruct a source link from a retrieved chunk, citations become guesswork.

Never treat metadata as optional bookkeeping. It carries scope, freshness, ownership, and permissions that the chunk text alone may not express.

An embedding is a list of numbers representing patterns in a piece of text. Texts with related meanings tend to receive nearby representations, which makes semantic search possible. The system sends each chunk to an embedding model and stores the returned vector beside the chunk and metadata.

The query must later use a compatible embedding model and preprocessing path. Vectors from unrelated model spaces cannot be compared meaningfully. If you change embedding models, dimensions, or normalization assumptions, plan to rebuild the index rather than mixing old and new vectors.

Embeddings are useful, but not magical. Exact identifiers, acronyms, dates, and rare names may work better with keyword search. Many production systems retain both lexical and vector representations so they can combine them later.

An index is the structure used to find likely chunks quickly. A vector database or vector-capable search engine stores chunk vectors and supports nearest-neighbor search. The same system may store text and metadata, or it may keep vectors linked to records elsewhere.

Indexing should be repeatable. Give each chunk a stable ID based on the source and location. When a document changes, replace affected chunks, remove obsolete ones, and avoid leaving both versions searchable. When a source is deleted or access is revoked, propagate that change promptly.

A safe update flow builds or updates data, checks counts and sample queries, and then exposes the new version. For larger changes, an alias can switch traffic from an old index to a validated new index. Keep enough lineage to answer, “Which source version produced this chunk?”

“Indexed successfully once” is not a finish line. Define a freshness target: minutes for incident procedures, perhaps a day for a slowly changing archive. Monitor connector delays, parse errors, embedding failures, stale documents, and deletion lag.

Apply authorization before returning chunks to the model. Post-filtering after retrieval can reduce quality and may expose restricted text to logs or prompts. A common pattern is to attach access metadata during ingestion and enforce it as a retrieval filter for every request.

Also assume ingested documents are untrusted input. A document can contain text such as “ignore previous instructions.” Store it as content, but later prompts must clearly separate source data from system instructions.

Before adding generation, inspect the index directly. Choose representative questions and verify that expected chunks exist, are readable, carry correct metadata, and can be found. Check awkward formats: tables, scanned pages, duplicated headers, nested lists, and documents with similar names.

This stage is easier to debug without an LLM. If the correct leave-policy paragraph never entered the index, no prompt can retrieve it. If an expired policy remains active, a better model will merely explain the expired policy more fluently.