How to Fine-Tune an LLM (And When RAG Is the Better Choice)
Learn how to fine-tune an LLM for style, format, and stable tasks—and when retrieval or prompting is cheaper, safer, and easier to update.
Fine-tuning an LLM updates model weights so the model becomes better at a narrow distribution: a house style, a classification scheme, a tool-call format, or a language you under-represent in the base model. It is not the default way to add new facts. Facts that change should usually live in a database or a RAG index.
This guide covers when to fine-tune, how to prepare data, how LoRA-style training fits, and how to evaluate without fooling yourself. Practice the decision in Fine-tuning vs RAG. Related: RAG vs fine-tuning, prompting vs RAG vs fine-tuning, how to build a RAG chatbot, open-source LLMs, and AI evals.
Fine-tuning vs RAG vs prompts
| Need | Start with | Fine-tune if |
|---|---|---|
| New or changing facts | RAG or tools | Never as the first choice |
| Tone and house style | Prompts + examples | Volume is high and prompts fail |
| Strict output schema | Prompts + validators | The model still drifts after checks |
| Domain jargon | Glossary in prompt/RAG | The jargon is stable and frequent |
| Classification / routing | Small classifier or SLM | You already know labels |
| Brand-new capability | Wrong tool | Fine-tuning cannot add true knowledge it never saw |
If you are about to fine-tune so the model “knows our 2026 pricing,” stop. Put pricing in a doc and retrieve it.
When fine-tuning is justified
Fine-tune when you can say all of the following:
- The behavior is stable for months.
- You have hundreds to thousands of representative, permissioned examples.
- You can grade success automatically or with a tight rubric.
- Prompting and retrieval already failed a fair test.
- You can retrain when the base model or the task changes.
Examples that often qualify: support ticket routing, extracting a fixed schema from emails, matching a regulated report format, or adapting a small model to a language.
Choose a base model you can afford to own
Smaller instruction-tuned models are easier to fine-tune and host. See small language models and open-source LLMs.
Hosted fine-tuning APIs reduce ops and can lock you to a vendor. Open-weight LoRA on your GPUs costs engineering and gives you portability. Pick based on data residency and how often you will retrain.
Step-by-step fine-tune
1. Freeze the task definition
Write the input, the allowed outputs, and disallowed behavior. Example: “Given a support email, return JSON {intent, product, urgency} from closed lists. If product is unclear, product=unknown.”
2. Build a dataset with provenance
Each row needs:
- input (and optional system prompt);
- target output;
- source and permission to use;
- split tag: train / validation / holdout.
Remove secrets. Deduplicate. Balance rare classes or the model will ignore them.
Quality beats clever loss functions. One hundred clean rows often beat five thousand scraped ones.
3. Prefer parameter-efficient methods
LoRA and related adapters train a small set of weights on top of a frozen base. That is cheaper, stores cleanly, and lets you swap adapters. Full-parameter fine-tuning is for teams that already know they need it.
4. Train with boring, logged settings
Record base model id, adapter config, learning rate, epochs, max length, and dataset hash. Early-stop on validation. If training loss crashes while validation worsens, you are memorizing.
5. Evaluate the holdout and the production shape
Run:
- exact-match or schema validity for structured tasks;
- side-by-side human review for style;
- safety and refusal cases;
- latency and cost versus the base model.
A model that writes prettier paragraphs but breaks JSON is a failed train.
6. Deploy behind the same guards as the base model
Fine-tuning does not remove prompt injection risk or hallucinations. Keep validators, retrieval for facts, and monitoring.
Data recipes that work
Style transfer: paired examples of “generic draft → house style,” not just finished essays.
Extraction: real documents with human-labeled JSON, including messy and empty cases.
Routing: historical tickets with corrected labels, not the raw first-touch guess.
Don’t: dump your wiki into completion format and hope the model “absorbs” it. That is an expensive, stale RAG.
Cost and maintenance
Budget for:
- labeling and legal review;
- GPU or vendor train jobs;
- eval each base-model upgrade;
- rollback if a train regresses a safety case.
If you cannot afford to re-eval when the vendor ships a new base model, do not fine-tune on that vendor.
Common failure modes
| Symptom | Cause | Fix |
|---|---|---|
| Perfect train, dumb prod | Leakage or non-representative data | New holdout from production |
| Lost general skill | Over-narrow data | Mix general instruction data; fewer epochs |
| Invented facts | You trained for “sound sure” | RAG + refusals |
| License/privacy incident | Unpermitted data | Delete, retrain, notify per policy |
FAQ
How much data do I need to fine-tune an LLM?
For a narrow classification or format task, a few hundred excellent examples can work. For open-ended style, you typically need more, plus a holdout. There is no magic number—plot quality versus dataset size.
Can I fine-tune ChatGPT?
OpenAI and other hosts offer fine-tuning or customization products on specific models. Features and data-use terms vary. A custom GPT is not a fine-tune; it is instructions plus optional files.
Is LoRA as good as full fine-tuning?
For most product tasks, yes enough—and far cheaper. Use full fine-tuning only when adapter methods fail a measured test.
Should I fine-tune or use RAG?
If the knowledge changes or must be cited, RAG. If the behavior is stable and prompting failed, fine-tune. Many systems do both: retrieve facts, generate in a fine-tuned style.
Bottom line
Fine-tune an LLM only after prompts and retrieval lose a fair bake-off, and only for stable behavior you can grade. Prepare clean, permitted data, prefer LoRA-style adapters, and keep facts in RAG. Practice in Fine-tuning vs RAG, then continue with RAG vs fine-tuning, eval tools, and guides.