Learn the controls and knobs

Control case mix, sample size, rubric, pass threshold, judge prompt, reference answers, random seeds, retries, contamination checks, slice definitions, and confidence intervals. Version datasets and prompts. Blind reviewers to candidate identity where possible. A knob is useful only when you can predict its directional effect, observe that effect, and reverse it. Record defaults and units; “increase context” is ambiguous, while “raise retrieved top-k from 4 to 8” is testable.

Separate controls into three layers. Quality controls change evidence or model behavior. Resource controls bound tokens, compute, latency, storage, and concurrency. Risk controls restrict data, tools, actions, or release exposure. One setting can affect several layers, so never assume that a quality gain is free.

For a coding copilot, pick a baseline and change one variable at a time on a fixed case set. If you change model, prompt, retrieval, and threshold together, the result may improve but you have learned nothing reusable. Log the configuration with every output. Seeds help where supported, but repeated runs are still needed for stochastic systems.

Create a control card:

| Control | Expected gain | Expected cost | Guardrail | |---|---|---|---| | Primary topic knob | A named task metric | Latency, money, or diversity | Hard limit or rollback | | Evidence scope | Better relevant coverage | More distraction or storage | Access filter | | Confidence threshold | Fewer unsafe automatic actions | More escalation | Human queue capacity | | Release exposure | Faster learning | User impact | Canary and kill switch |

Exact-match scoring is reproducible but rejects valid variation; human review handles nuance but is costly and inconsistent; model judges scale but can favor their own style. Public benchmarks aid comparison while encouraging overfitting and often underrepresent your deployment. Write the expected direction before running the test. A surprising result then becomes information rather than an excuse to cherry-pick another setting.

A minimal experiment record can be plain text:

configuration_id: topic-baseline-v1
case_set: representative-v3
changed_control: one_named_setting
primary_metric: task_success
hard_gates: [severe_error_rate, privacy, p95_latency]
rollback_if: any_hard_gate_fails

The names are illustrative; the discipline is not. The case set and gates must be chosen before seeing results. For high-consequence actions, thresholds route uncertain cases to review rather than pretending uncertainty disappeared.

Some controls multiply each other. Longer inputs raise token cost and may reduce concurrency. More candidates can improve recall while burdening a reranker. Stronger preference or guidance can improve adherence but reduce diversity. More memory can improve continuity and worsen privacy. Test important pairs after single-variable tests, especially near hard limits.

A mature system exposes only safe controls to end users. Infrastructure limits, policy thresholds, and private prompts remain server-owned. User-facing controls should have understandable effects and bounded ranges.