Learn the controls and knobs

Choose model and license, parameter count, quantization, context size, CPU/GPU layers, thread count, batch size, concurrency, cache policy, sampler, and serving runtime. Pin model hashes and templates because the wrong chat format can destroy apparent quality. A knob is useful only when you can predict its directional effect, observe that effect, and reverse it. Record defaults and units; “increase context” is ambiguous, while “raise retrieved top-k from 4 to 8” is testable.

Separate controls into three layers. Quality controls change evidence or model behavior. Resource controls bound tokens, compute, latency, storage, and concurrency. Risk controls restrict data, tools, actions, or release exposure. One setting can affect several layers, so never assume that a quality gain is free.

For an offline clinical note assistant, pick a baseline and change one variable at a time on a fixed case set. If you change model, prompt, retrieval, and threshold together, the result may improve but you have learned nothing reusable. Log the configuration with every output. Seeds help where supported, but repeated runs are still needed for stochastic systems.

Create a control card:

| Control | Expected gain | Expected cost | Guardrail | |---|---|---|---| | Primary topic knob | A named task metric | Latency, money, or diversity | Hard limit or rollback | | Evidence scope | Better relevant coverage | More distraction or storage | Access filter | | Confidence threshold | Fewer unsafe automatic actions | More escalation | Human queue capacity | | Release exposure | Faster learning | User impact | Canary and kill switch |

Local deployment offers data control, offline use, and predictable marginal cost, but transfers patching, capacity, and incident ownership to you. Quantization and smaller models fit cheaper hardware while possibly reducing quality. Long contexts consume substantial KV-cache memory. Write the expected direction before running the test. A surprising result then becomes information rather than an excuse to cherry-pick another setting.

A minimal experiment record can be plain text:

configuration_id: topic-baseline-v1
case_set: representative-v3
changed_control: one_named_setting
primary_metric: task_success
hard_gates: [severe_error_rate, privacy, p95_latency]
rollback_if: any_hard_gate_fails

The names are illustrative; the discipline is not. The case set and gates must be chosen before seeing results. For high-consequence actions, thresholds route uncertain cases to review rather than pretending uncertainty disappeared.

Some controls multiply each other. Longer inputs raise token cost and may reduce concurrency. More candidates can improve recall while burdening a reranker. Stronger preference or guidance can improve adherence but reduce diversity. More memory can improve continuity and worsen privacy. Test important pairs after single-variable tests, especially near hard limits.

A mature system exposes only safe controls to end users. Infrastructure limits, policy thresholds, and private prompts remain server-owned. User-facing controls should have understandable effects and bounded ranges.