Mastery: connect the pieces

A complete explanation of local LLMs now has four connected claims. A local LLM runs under infrastructure you control—on a laptop, workstation, edge device, or private server. “Local” changes data flow and operations, not intelligence: the model still needs memory, compute, updates, evaluation, and security boundaries. Model weights are loaded into CPU/GPU memory; a runtime tokenizes prompts, executes transformer layers, maintains a key-value cache, and samples tokens. Quantization stores weights at fewer bits to reduce memory. A server adds batching, concurrency, authentication, and observability. Local means inference runs on controlled hardware; open-weight means weights are obtainable under a license; open source implies more than downloadable weights. Quantization changes numerical representation, not parameter count, and privacy still depends on logs, plugins, and network paths.

Turn those claims into a design for an offline clinical note assistant. State the user job, data boundary, uncertain model contribution, deterministic controls, evaluation set, release gate, production signal, and failure response. If any item is missing, the concept is not yet operational.

Use this spoken diagram:

authorized input -> scoped evidence -> learned operation
                 -> deterministic validation -> bounded action
                 -> outcome + trace -> evaluation and improvement

At every arrow ask: what representation crosses, who owns it, what can be lost, and how is it versioned? Choose model and license, parameter count, quantization, context size, CPU/GPU layers, thread count, batch size, concurrency, cache policy, sampler, and serving runtime. Pin model hashes and templates because the wrong chat format can destroy apparent quality. The controls should be few enough to understand and complete enough to constrain the severe failures.

Local deployment offers data control, offline use, and predictable marginal cost, but transfers patching, capacity, and incident ownership to you. Quantization and smaller models fit cheaper hardware while possibly reducing quality. Long contexts consume substantial KV-cache memory. Choose one tradeoff and defend it quantitatively. Name a hard constraint, a primary metric, and the cost you accept. Then name evidence that would reverse your decision. This last step protects the design from becoming identity or vendor loyalty.

A defensible statement sounds like: “We choose configuration B because it passes the privacy and severe-error gates, improves task success on the target slice, and stays within the p95 latency budget. We will reconsider if traffic or review cost crosses the recorded threshold.”

Weights fit but KV cache causes out-of-memory under concurrency; thermal throttling slows laptops; permissive endpoints expose the model; an incompatible prompt template causes nonsense; licenses prohibit the intended use; and “no cloud” claims fail when telemetry or embeddings leave the device. Pick the most consequential failure and walk through trigger, earliest signal, containment, owner, recovery, and prevention. Test task quality at the exact quantization, tokens per second, time to first token, peak memory, concurrent throughput, energy, startup time, and failure recovery. Verify network egress, license terms, model provenance, and behavior on target hardware—not a vendor GPU. Monitoring should reuse the evaluation construct where possible, while acknowledging that production labels may arrive late.

Answer without notes:

1. What does this concept change: evidence, learned behavior, runtime state, coordination, or measurement?
2. Which neighboring concept is commonly confused with it?
3. Which intermediate artifact would you inspect first?
4. Which knob has the largest quality/resource interaction?
5. What hard gate cannot be traded for average quality?
6. What baseline could disprove the need for the complex design?
7. How would you detect harm hidden by an aggregate metric?
8. What is the safe state during uncertainty?

Now explain the worked evidence: A 7B model at 4-bit needs roughly 3.5 GB just for raw weights, plus runtime overhead and KV cache. It runs one 4k-token chat in 6 GB, but four concurrent 16k chats trigger OOM. Limiting concurrency to two and context to 8k stabilizes service; a queue makes the capacity limit explicit. If you can identify the causal chain, calculate the consequential change, propose an alternative hypothesis, and choose a reversible response, you have moved from vocabulary to engineering judgment.

End with one page containing: context, alternatives, assumptions, case-set version, configuration IDs, metric table, gates, selected option, rejected options, owner, rollout, rollback, and review date. This artifact makes future disagreement productive because teammates can challenge evidence or weights instead of reconstructing hidden reasoning.