Compare · Building · Research notes
Local LLM vs API — scored
Local is not automatically cheaper or safer. Write the constraint, then measure.
Last reviewed: 2026-08-15. Re-run the protocol after major model or pricing changes.
How do you choose local vs API?
- Write privacy, latency, and quality constraints.
- Estimate VRAM for one 7B Q4 model.
- Run the same three prompts local and hosted.
- Price tokens vs electricity/engineering time.
What is the scoring rubric?
| Criterion | Weight | How to score |
|---|---|---|
| Data cannot leave | Gate | If yes, local or a signed VPC — not a consumer chat. |
| Quality on the job | 1–5 | Frontier APIs still win many hard tasks. |
| Ops cost | 1–5 | Include updates, GPUs, and on-call. |
| Fallback plan | Pass/Fail | What happens when the box is down? |
Which default should I start with?
| Tool | Best fit | When | Watch-out | Note |
|---|---|---|---|---|
| Hosted API | Quality / speed to value | Frontier models | Data leaves the box | Default unless privacy forbids it. |
| Local stack | Privacy / offline | Ollama / MLC / LocalAI | VRAM and quality caps | Use the VRAM estimator first. |
Learn the craft, not just the tool
Practice the transferable habit in Local LLMs explained, then VRAM estimator, First local LLM guide.
FAQ
Is a local LLM cheaper than an API?
Only after you count GPUs, electricity, and engineering. Low volume often favors APIs.
Is local always more private?
It can be, if the model and logs stay on-device. Browser demos and synced folders can still leak.