Learn the controls and knobs

Tune lexical and dense candidate counts, fusion weights or RRF constant, field boosts, metadata filters, query rewriting, reranker depth, final top-k, and duplicate removal. Preserve exact identifiers and quoted phrases rather than rewriting them away. A knob is useful only when you can predict its directional effect, observe that effect, and reverse it. Record defaults and units; “increase context” is ambiguous, while “raise retrieved top-k from 4 to 8” is testable.

Separate controls into three layers. Quality controls change evidence or model behavior. Resource controls bound tokens, compute, latency, storage, and concurrency. Risk controls restrict data, tools, actions, or release exposure. One setting can affect several layers, so never assume that a quality gain is free.

For a technical support search assistant, pick a baseline and change one variable at a time on a fixed case set. If you change model, prompt, retrieval, and threshold together, the result may improve but you have learned nothing reusable. Log the configuration with every output. Seeds help where supported, but repeated runs are still needed for stochastic systems.

Create a control card:

| Control | Expected gain | Expected cost | Guardrail | |---|---|---|---| | Primary topic knob | A named task metric | Latency, money, or diversity | Hard limit or rollback | | Evidence scope | Better relevant coverage | More distraction or storage | Access filter | | Confidence threshold | Fewer unsafe automatic actions | More escalation | Human queue capacity | | Release exposure | Faster learning | User impact | Canary and kill switch |

Hybrid search improves coverage but runs two retrieval paths and complicates tuning. Dense search handles synonyms but may miss rare codes; lexical search nails codes but misses paraphrases. Reranking improves precision at added latency and cost. Write the expected direction before running the test. A surprising result then becomes information rather than an excuse to cherry-pick another setting.

A minimal experiment record can be plain text:

configuration_id: topic-baseline-v1
case_set: representative-v3
changed_control: one_named_setting
primary_metric: task_success
hard_gates: [severe_error_rate, privacy, p95_latency]
rollback_if: any_hard_gate_fails

The names are illustrative; the discipline is not. The case set and gates must be chosen before seeing results. For high-consequence actions, thresholds route uncertain cases to review rather than pretending uncertainty disappeared.

Some controls multiply each other. Longer inputs raise token cost and may reduce concurrency. More candidates can improve recall while burdening a reranker. Stronger preference or guidance can improve adherence but reduce diversity. More memory can improve continuity and worsen privacy. Test important pairs after single-variable tests, especially near hard limits.

A mature system exposes only safe controls to end users. Infrastructure limits, policy thresholds, and private prompts remain server-owned. User-facing controls should have understandable effects and bounded ranges.