Configure the load testing contract

You own a streaming chat completion gateway at POST /v1/chat/stream. The goal is to measure the saturation knee with a workload that preserves prompt size, streaming duration, tenant mix, and cache-hit distributions seen in production. The measurable target is hold 120 virtual users for 15 minutes with p95 time-to-first-token below 900 ms, p99 full response below 8 s, non-429 errors below 0.5%, and queue depth below 80; identify the first bottleneck rather than merely reporting requests per second. The known production tension is closed-loop virtual users hide coordinated omission while open arrival rates can overload a test environment unrealistically; production-shaped data improves validity but requires strict redaction and spend controls.

The contract for Load testing must survive a clean checkout. Pin tool and image versions, provide sample inputs, and fail startup on malformed configuration. The service boundary is POST /v1/chat/stream on a streaming chat completion gateway. Before starting dependencies, list required ports, identities, secrets, storage, and cleanup steps. Use synthetic credentials and redacted fixtures; production tokens do not make a staging lab more realistic.

Create the configuration as a reviewed artifact:

export const options = {
  scenarios: { ramp: {
    executor: 'ramping-arrival-rate', startRate: 20, timeUnit: '1s',
    stages: [{target: 60,duration:'5m'},{target:120,duration:'10m'},{target:180,duration:'5m'}],
    preAllocatedVUs: 200, maxVUs: 400 }},
  thresholds: { http_req_failed: ['rate<0.005'],
    'http_req_duration{phase:ttft}': ['p(95)<900'] }
};

For every field, document type, valid range, default, reload behavior, and failure behavior. Numbers are incomplete without units. A timeout of 40 could mean milliseconds or seconds; a ratio of 0.005 could be a fraction or percentage. Reject unknown keys so a typo cannot silently select a permissive default. If live reload is supported, expose a configuration revision in telemetry and keep the previous valid revision for rollback.

Run the setup workflow:

docker compose up -d gateway model-stub prometheus
curl -fsS localhost:8080/healthz && curl -fsS localhost:9090/-/ready

The expected result is not merely exit code zero. Confirm the process identity, dependency reachability, effective configuration, and one health or introspection response. Then restart the component and prove durable state remains correct. Run the setup twice to expose non-idempotent creation, conflicting ports, or duplicate policy entries. Capture the exact versions in the evidence bundle.

The production objective remains hold 120 virtual users for 15 minutes with p95 time-to-first-token below 900 ms, p99 full response below 8 s, non-429 errors below 0.5%, and queue depth below 80; identify the first bottleneck rather than merely reporting requests per second. Therefore setup must expose http_req_duration, time_to_first_token_ms, generation_queue_depth, active_streams, tokens_generated_total, provider_errors_total, and cost_usd_total before meaningful traffic arrives. Query each metric by its stable labels and verify an idle system is distinguishable from a broken scraper. Generate one event and check that the relevant counter changes by exactly one. Reject high-cardinality labels and ensure sensitive configuration values are redacted from startup logs.

Model a bad configuration: remove one required field, set one threshold just outside its range, and reference an unavailable dependency. Each case should fail predictably, explain the field, and avoid partial state. Next, apply the previous good revision and time recovery. A rollback that requires undocumented shell history is not a rollback plan.

The central design tension is closed-loop virtual users hide coordinated omission while open arrival rates can overload a test environment unrealistically; production-shaped data improves validity but requires strict redaction and spend controls. Record the chosen default, its operational owner, and a date for reevaluation. Complete setup only after a teammate can run it from the repository instructions, produce the same effective configuration, and cleanly tear it down without affecting shared environments.