Learn the controls and knobs

Choose sampling by risk tier, alert windows, baseline segments, quality judges, cost and latency budgets, redaction rules, canary size, and rollback thresholds. Version prompts, indexes, tools, and providers independently so an alert can identify the changed component. A knob is useful only when you can predict its directional effect, observe that effect, and reverse it. Record defaults and units; “increase context” is ambiguous, while “raise retrieved top-k from 4 to 8” is testable.

Separate controls into three layers. Quality controls change evidence or model behavior. Resource controls bound tokens, compute, latency, storage, and concurrency. Risk controls restrict data, tools, actions, or release exposure. One setting can affect several layers, so never assume that a quality gain is free.

For a customer-support assistant, pick a baseline and change one variable at a time on a fixed case set. If you change model, prompt, retrieval, and threshold together, the result may improve but you have learned nothing reusable. Log the configuration with every output. Seeds help where supported, but repeated runs are still needed for stochastic systems.

Create a control card:

| Control | Expected gain | Expected cost | Guardrail | |---|---|---|---| | Primary topic knob | A named task metric | Latency, money, or diversity | Hard limit or rollback | | Evidence scope | Better relevant coverage | More distraction or storage | Access filter | | Confidence threshold | Fewer unsafe automatic actions | More escalation | Human queue capacity | | Release exposure | Faster learning | User impact | Canary and kill switch |

Detailed traces improve diagnosis but raise storage, privacy, and review costs. Fast proxy judges provide coverage but can share model biases; slow human labels are credible but sparse. Global thresholds are simple, while segmented thresholds catch damage hidden in averages. Write the expected direction before running the test. A surprising result then becomes information rather than an excuse to cherry-pick another setting.

A minimal experiment record can be plain text:

configuration_id: topic-baseline-v1
case_set: representative-v3
changed_control: one_named_setting
primary_metric: task_success
hard_gates: [severe_error_rate, privacy, p95_latency]
rollback_if: any_hard_gate_fails

The names are illustrative; the discipline is not. The case set and gates must be chosen before seeing results. For high-consequence actions, thresholds route uncertain cases to review rather than pretending uncertainty disappeared.

Some controls multiply each other. Longer inputs raise token cost and may reduce concurrency. More candidates can improve recall while burdening a reranker. Stronger preference or guidance can improve adherence but reduce diversity. More memory can improve continuity and worsen privacy. Test important pairs after single-variable tests, especially near hard limits.

A mature system exposes only safe controls to end users. Infrastructure limits, policy thresholds, and private prompts remain server-owned. User-facing controls should have understandable effects and bounded ranges.