Fine-tuning vs RAG
Anticipate failure modes
Design for predictable breakage: explain fine-tuning versus RAG by connecting a concrete decision to observable evidence.
Before you start
Why this matters
Imagine you own a legal research assistant and must explain one decision to a teammate who knows basic AI vocabulary but has never operated this feature. Write two sentences: what problem does fine-tuning versus RAG solve, and what evidence would show it is solving that problem? Do not name a vendor or model yet. This separates the enduring idea from one implementation.
1Learn the idea
Read
Failures have shapes
See it
Fine-tune
Teach voice / format into weights
RAG
Fetch fresh docs at ask time
Fine-tune = bake in style · RAG = look things up when answering
Teams fine-tune on changing policies and get stale answers, expect RAG to teach a stable output schema, train on synthetic mistakes, retrieve unauthorized documents, or evaluate only fluent answers. A tuned model may memorize sensitive examples; RAG may cite an irrelevant but similar passage. These are not all “hallucinations.” Give each failure a stage, trigger, observable symptom, consequence, and response. Precise names produce precise tests.
Use the chain trigger → earliest evidence → user impact → containment → prevention. The earliest evidence is especially valuable. If a user complaint is the first signal, detection arrived late. For a legal research assistant, inspect intermediate artifacts so a bad input, retrieval, model judgment, validation, or action can be distinguished.
Read
Failure register
| Failure class | Early signal | Immediate containment | Longer-term repair | |---|---|---|---| | Input or data shift | Segment distribution changes | Narrow scope | Improve parser/data contract | | Model or scoring error | Offline slice regresses | Fallback or review | Prompt/model/data change | | Resource exhaustion | Queue, memory, or p95 spike | Shed load | Capacity/control redesign | | Policy or privacy breach | Forbidden field/action appears | Stop and revoke | Minimize access and retest | | Coordination defect | Duplicate/stale artifacts | Freeze action | Version state and ownership |
Populate the rows with topic-specific signals, not generic red/yellow labels. A threshold should include a time window, minimum sample, segment, and owner.
Read
Why plausible systems fail silently
Prompting supplies instructions now, fine-tuning changes weighted behavior, RAG supplies external evidence now, and tools perform actions or deterministic lookups. Fine-tuning is not a database, and retrieval is not learning. Confusing neighboring concepts leads teams to repair the wrong layer. A polished answer can conceal missing evidence; a valid schema can contain false values; a successful tool call can perform the wrong authorized action. Validate content, structure, and consequence separately.
Correlated failures deserve special attention. Model-based judges may favor the same style as the model they score. Several agents may share one false assumption. Overlapping chunks may look like independent evidence. Repetition is not corroboration unless sources and failure paths are independent.
Read
Design the response before the incident
For each severe failure, define a stop condition and safe state. Rollback works only if old prompts, indexes, weights, and schemas remain available and compatible. A fallback must be tested under load. Human review needs queue capacity and enough context to decide; “send to a human” is not a complete control.
Practice one tabletop scenario: inject a realistic defect, verify detection, identify the owner, execute containment, and measure recovery time. Preserve the trace for a blameless review. The goal is not to claim failures are impossible but to reduce their frequency, blast radius, and time to recovery.
Checking tutor…