Mastery: connect the pieces

A complete explanation of chunking for RAG now has four connected claims. Chunking decides which pieces of a document can be found and shown to the model. Imagine cutting a map into cards: tiny cards lose landmarks and huge cards bury the street you need. Good cuts preserve a complete answerable idea. Parse document structure, split at semantic boundaries, attach titles and metadata, optionally overlap neighboring text, embed each chunk, and retrieve chunks for a query. Parent-child retrieval can search small child passages while returning a larger parent section for generation. Chunking creates retrievable units; embedding maps units to vectors; retrieval selects candidates; reranking reorders them; generation writes the answer. Increasing top-k cannot reliably repair a chunk that never contained a coherent fact.

Turn those claims into a design for an employee handbook assistant. State the user job, data boundary, uncertain model contribution, deterministic controls, evaluation set, release gate, production signal, and failure response. If any item is missing, the concept is not yet operational.

Use this spoken diagram:

authorized input -> scoped evidence -> learned operation
                 -> deterministic validation -> bounded action
                 -> outcome + trace -> evaluation and improvement

At every arrow ask: what representation crosses, who owns it, what can be lost, and how is it versioned? Tune chunk unit, target token size, overlap, heading inheritance, table and code handling, metadata, parent-child ratio, top-k, and reranking. Different content needs different boundaries: policies by clause, API docs by endpoint, transcripts by speaker turn, and tables as coherent units. The controls should be few enough to understand and complete enough to constrain the severe failures.

Small chunks improve retrieval precision but lose context and increase index entries. Large chunks preserve context but consume the prompt and may dilute similarity. Overlap prevents boundary loss but duplicates evidence, storage, and citations; semantic splitting costs more than fixed windows. Choose one tradeoff and defend it quantitatively. Name a hard constraint, a primary metric, and the cost you accept. Then name evidence that would reverse your decision. This last step protects the design from becoming identity or vendor loyalty.

A defensible statement sounds like: “We choose configuration B because it passes the privacy and severe-error gates, improves task success on the target slice, and stays within the p95 latency budget. We will reconsider if traffic or review cost crosses the recorded threshold.”

Fixed character cuts split sentences or tables, headers become detached, boilerplate dominates embeddings, OCR artifacts create nonsense chunks, and overlap returns near-duplicates. Updating only part of a document can leave stale chunks. One global size rarely serves prose, code, and tables equally. Pick the most consequential failure and walk through trigger, earliest signal, containment, owner, recovery, and prevention. Measure retrieval recall@k against evidence spans, precision or nDCG, answer groundedness, citation correctness, duplicate rate, average retrieved tokens, and latency. Build queries that require facts near boundaries and compare chunking variants while holding the embedder and generator fixed. Monitoring should reuse the evaluation construct where possible, while acknowledging that production labels may arrive late.

Answer without notes:

1. What does this concept change: evidence, learned behavior, runtime state, coordination, or measurement?
2. Which neighboring concept is commonly confused with it?
3. Which intermediate artifact would you inspect first?
4. Which knob has the largest quality/resource interaction?
5. What hard gate cannot be traded for average quality?
6. What baseline could disprove the need for the complex design?
7. How would you detect harm hidden by an aggregate metric?
8. What is the safe state during uncertainty?

Now explain the worked evidence: A leave-policy answer spans the final sentence of “Eligibility” and first list under “Exceptions.” With 400-token non-overlapping windows, recall@5 is 71%. Adding 60-token overlap raises it to 86% but duplicates 28% of results. Heading-aware 250-token clauses with parent expansion reach 91% recall and only 7% duplicates. If you can identify the causal chain, calculate the consequential change, propose an alternative hypothesis, and choose a reversible response, you have moved from vocabulary to engineering judgment.

End with one page containing: context, alternatives, assumptions, case-set version, configuration IDs, metric table, gates, selected option, rejected options, owner, rollout, rollback, and review date. This artifact makes future disagreement productive because teammates can challenge evidence or weights instead of reconstructing hidden reasoning.