Weigh the tradeoffs

Local deployment offers data control, offline use, and predictable marginal cost, but transfers patching, capacity, and incident ownership to you. Quantization and smaller models fit cheaper hardware while possibly reducing quality. Long contexts consume substantial KV-cache memory. This is why “best” must always finish the sentence: best for which users, traffic, risk, hardware, budget, and deadline?

Start with constraints, not preferences. A hard privacy rule, an accessibility requirement, or a two-second interaction budget eliminates designs before a weighted score is useful. Among feasible choices, compare expected utility. A simple decision model is:

utility = task_value - error_cost - inference_cost - delay_cost - operations_cost

The terms need not share natural units; agreed weights make assumptions visible. Run sensitivity analysis. If a small change in the error-cost weight flips the winner, the decision is fragile and needs better evidence or a reversible rollout.

For an offline clinical note assistant, compare at least a simple baseline, a moderate design, and a maximal design. Rate each on quality, severe failures, latency, variable cost, privacy, debuggability, and team burden. Do not let one average score compensate for a prohibited failure. Apply gates first.

A 7B model at 4-bit needs roughly 3.5 GB just for raw weights, plus runtime overhead and KV cache. It runs one 4k-token chat in 6 GB, but four concurrent 16k chats trigger OOM. Limiting concurrency to two and context to 8k stabilizes service; a queue makes the capacity limit explicit. The lesson is not the final setting; it is the sequence of evidence and the willingness to choose a less impressive configuration when it better satisfies the whole system.

Count retries, fallbacks, duplicated tokens, review labor, index updates, GPU idle capacity, incident response, and evaluation maintenance. Per-token price alone often reverses the wrong decision. Likewise, local operation is not free after hardware purchase, and automation is not free when humans must repair low-quality cases.

Opportunity cost matters too. A complex architecture may gain two quality points while delaying the feedback loop by a month. A simpler version with a clean trace and rollback can teach more. Choose the smallest design that tests the riskiest assumption.

Prefer reversible choices under uncertainty: canary traffic, versioned indexes, expiring memory, adapters rather than irreversible data changes, and feature flags around orchestration. Restrict early exposure to cases where failure is recoverable. Consequences—not model size—determine the required approval level.

Finally, state who bears each cost. A system can improve an aggregate metric while shifting work to reviewers, slowing users on poor connections, or degrading one language. Segment results and ask whether the people receiving benefits are also absorbing the errors. That question turns an abstract tradeoff into an accountable product decision.