Weigh the tradeoffs

Larger models often improve difficult reasoning but cost more and respond slower. Hosted APIs reduce operations work but add dependency and data-governance concerns; local models offer control but require capacity planning. Long context is useful only if the model can reliably use it. This is why “best” must always finish the sentence: best for which users, traffic, risk, hardware, budget, and deadline?

Start with constraints, not preferences. A hard privacy rule, an accessibility requirement, or a two-second interaction budget eliminates designs before a weighted score is useful. Among feasible choices, compare expected utility. A simple decision model is:

utility = task_value - error_cost - inference_cost - delay_cost - operations_cost

The terms need not share natural units; agreed weights make assumptions visible. Run sensitivity analysis. If a small change in the error-cost weight flips the winner, the decision is fragile and needs better evidence or a reversible rollout.

For an invoice-extraction service, compare at least a simple baseline, a moderate design, and a maximal design. Rate each on quality, severe failures, latency, variable cost, privacy, debuggability, and team burden. Do not let one average score compensate for a prohibited failure. Apply gates first.

Three invoice models score 94%, 96%, and 97% field accuracy. The 97% model costs $0.018/request and misses VAT IDs; the 96% model costs $0.006 and passes every critical-field gate. At one million requests, the difference is $12,000 monthly. Choose the middle model and route low-confidence handwritten invoices to the larger one. The lesson is not the final setting; it is the sequence of evidence and the willingness to choose a less impressive configuration when it better satisfies the whole system.

Count retries, fallbacks, duplicated tokens, review labor, index updates, GPU idle capacity, incident response, and evaluation maintenance. Per-token price alone often reverses the wrong decision. Likewise, local operation is not free after hardware purchase, and automation is not free when humans must repair low-quality cases.

Opportunity cost matters too. A complex architecture may gain two quality points while delaying the feedback loop by a month. A simpler version with a clean trace and rollback can teach more. Choose the smallest design that tests the riskiest assumption.

Prefer reversible choices under uncertainty: canary traffic, versioned indexes, expiring memory, adapters rather than irreversible data changes, and feature flags around orchestration. Restrict early exposure to cases where failure is recoverable. Consequences—not model size—determine the required approval level.

Finally, state who bears each cost. A system can improve an aggregate metric while shifting work to reviewers, slowing users on poor connections, or degrading one language. Segment results and ask whether the people receiving benefits are also absorbing the errors. That question turns an abstract tradeoff into an accountable product decision.