RAG is an open-book system

Retrieval means searching a collection for passages likely to help with the user’s question. The collection might contain policy pages, product manuals, support articles, research papers, or database records converted into readable text.

Augmented means adding those passages to the model’s input. The retrieved material joins the user’s question and the system’s instructions inside the prompt.

Generation means producing a useful response from that combined input. The model can summarize several passages, explain them in plain English, or state that the evidence is insufficient.

The usual flow is therefore:

1. receive a question;
2. retrieve relevant passages;
3. place them in a structured prompt;
4. generate an answer;
5. return citations or source links.

RAG is a system pattern, not a special kind of model. The same model may answer with or without retrieval depending on how the application is built.

A language model learns broad patterns during training. That background knowledge helps it understand language and connect ideas, but it has limits. Training data has a cutoff, may not contain your private documents, and does not provide a dependable record of which sentence supports a particular claim. Model weights are also not a convenient document store: you cannot reliably edit one policy fact in them and prove every old version disappeared.

RAG supplies working evidence at request time. If the vacation policy changes this morning and the search index is refreshed, the next answer can use the new policy without retraining the model. If the documents are access-controlled, retrieval can be limited to sources the current user is allowed to see.

This does not erase the model’s background knowledge. Instructions must tell it how to treat retrieved evidence, and evaluation must check whether it actually follows those instructions.

RAG is a good starting point when answers depend on information that is:

• private, such as internal procedures or customer-specific records;
• frequently updated, such as prices, schedules, inventory, or regulations;
• too large to paste into every prompt;
• spread across many documents;
• expected to come with inspectable sources.

Examples include an assistant for equipment manuals, a help center that answers from approved articles, or a research tool that summarizes a selected paper library. RAG is especially useful when users ask varied natural-language questions but the authoritative facts already exist in maintained sources.

The key test is not “Would search be impressive?” Ask: Does the task require finding evidence before composing the answer? If yes, retrieval probably belongs in the design.

Do not add retrieval just because an application uses an LLM. A fixed transformation such as “rewrite this paragraph more clearly” already has all necessary information in the request. A calculator should use arithmetic code, not retrieve prose about multiplication. A small, stable set of instructions may fit directly in a system prompt. A transactional question such as “What is my current balance?” often needs a live database query or tool call, because a periodically indexed document copy could be stale.

RAG also differs from fine-tuning. Fine-tuning changes model behavior by training on examples; it can help with tone, format, or repeated task patterns. It is not a reliable way to load a changing handbook into model weights. A system may eventually use both: retrieval for current facts and fine-tuning for consistent behavior.

Sometimes ordinary keyword search is enough. If users need to locate and read an exact document rather than receive a synthesized answer, generation adds cost and another opportunity for error.

RAG can make answers more grounded, but it does not make them automatically true. The search step may miss the right passage. The index may be stale. Two documents may conflict. The model may combine facts that the sources never connect, or ignore a limitation buried in a retrieved paragraph.

Think of RAG as an evidence pipeline with several testable stages, not as an anti-hallucination switch. A trustworthy design records what was retrieved, shows where claims came from, refuses unsupported questions, and evaluates retrieval separately from writing quality.

An answer can sound excellent while using the wrong policy. Conversely, retrieval can find the exact policy while a poor prompt produces a confusing answer. Those are different failures and need different fixes.

Before building, write down:

1. the authoritative source of truth;
2. how quickly source changes must appear;
3. who may access each source;
4. what a supported answer must cite;
5. what the system should do when evidence is missing;
6. whether a deterministic tool is required for live or calculated facts.

If no one owns the source material, retrieval cannot create authority. If the source changes faster than the index refreshes, freshness becomes a product promise you cannot keep. If no safe “I don’t know” behavior exists, the model will be pressured to fill gaps.