Weigh the tradeoffs

More context can reduce retrieval misses but increases cost, latency, and distraction. Summaries save tokens but discard detail and can fossilize errors. Sliding windows preserve recent turns while forgetting early commitments; long-context models may cost more than targeted retrieval. This is why “best” must always finish the sentence: best for which users, traffic, risk, hardware, budget, and deadline?

Start with constraints, not preferences. A hard privacy rule, an accessibility requirement, or a two-second interaction budget eliminates designs before a weighted score is useful. Among feasible choices, compare expected utility. A simple decision model is:

utility = task_value - error_cost - inference_cost - delay_cost - operations_cost

The terms need not share natural units; agreed weights make assumptions visible. Run sensitivity analysis. If a small change in the error-cost weight flips the winner, the decision is fragile and needs better evidence or a reversible rollout.

For a contract-review assistant, compare at least a simple baseline, a moderate design, and a maximal design. Rate each on quality, severe failures, latency, variable cost, privacy, debuggability, and team burden. Do not let one average score compensate for a prohibited failure. Apply gates first.

A model has a 16,384-token limit. System and schema use 1,200, history 5,600, retrieved clauses 7,900, and requested output 2,000: total 16,700, so it cannot fit. Reducing top-k removes the governing clause. Instead compress stale history to 1,400 tokens, leaving 3,884 tokens of headroom and retaining all cited clauses. The lesson is not the final setting; it is the sequence of evidence and the willingness to choose a less impressive configuration when it better satisfies the whole system.

Count retries, fallbacks, duplicated tokens, review labor, index updates, GPU idle capacity, incident response, and evaluation maintenance. Per-token price alone often reverses the wrong decision. Likewise, local operation is not free after hardware purchase, and automation is not free when humans must repair low-quality cases.

Opportunity cost matters too. A complex architecture may gain two quality points while delaying the feedback loop by a month. A simpler version with a clean trace and rollback can teach more. Choose the smallest design that tests the riskiest assumption.

Prefer reversible choices under uncertainty: canary traffic, versioned indexes, expiring memory, adapters rather than irreversible data changes, and feature flags around orchestration. Restrict early exposure to cases where failure is recoverable. Consequences—not model size—determine the required approval level.

Finally, state who bears each cost. A system can improve an aggregate metric while shifting work to reviewers, slowing users on poor connections, or degrading one language. Segment results and ask whether the people receiving benefits are also absorbing the errors. That question turns an abstract tradeoff into an accountable product decision.