Run the load testing baseline

You own a streaming chat completion gateway at POST /v1/chat/stream. The goal is to measure the saturation knee with a workload that preserves prompt size, streaming duration, tenant mix, and cache-hit distributions seen in production. The measurable target is hold 120 virtual users for 15 minutes with p95 time-to-first-token below 900 ms, p99 full response below 8 s, non-429 errors below 0.5%, and queue depth below 80; identify the first bottleneck rather than merely reporting requests per second. The known production tension is closed-loop virtual users hide coordinated omission while open arrival rates can overload a test environment unrealistically; production-shaped data improves validity but requires strict redaction and spend controls.

Build the smallest normal path through POST /v1/chat/stream for a streaming chat completion gateway. Use a fixed, non-sensitive fixture and assign a correlation identifier at ingress. The purpose is to establish known-good semantics before fault injection. For Load testing, the path must contribute directly to this goal: measure the saturation knee with a workload that preserves prompt size, streaming duration, tenant mix, and cache-hit distributions seen in production. Avoid adding retries, fallback branches, or tuning until the baseline is observable.

Use the reviewed configuration:

export const options = {
  scenarios: { ramp: {
    executor: 'ramping-arrival-rate', startRate: 20, timeUnit: '1s',
    stages: [{target: 60,duration:'5m'},{target:120,duration:'10m'},{target:180,duration:'5m'}],
    preAllocatedVUs: 200, maxVUs: 400 }},
  thresholds: { http_req_failed: ['rate<0.005'],
    'http_req_duration{phase:ttft}': ['p(95)<900'] }
};

Run the baseline command:

k6 run -e BASE_URL=http://localhost:8080 -e MAX_COST_USD=25 scenarios/chat-stream.js

Record response status or operator state, important headers or fields, elapsed time, and side effects. Follow the same correlation identifier through structured logs. If the system is asynchronous, capture enqueue time, start time, completion time, retry count, and durable checkpoint. Run at least five samples so one warm cache or connection setup does not masquerade as normal behavior.

The acceptance target is hold 120 virtual users for 15 minutes with p95 time-to-first-token below 900 ms, p99 full response below 8 s, non-429 errors below 0.5%, and queue depth below 80; identify the first bottleneck rather than merely reporting requests per second. Query http_req_duration, time_to_first_token_ms, generation_queue_depth, active_streams, tokens_generated_total, provider_errors_total, and cost_usd_total immediately before and after the baseline. Calculate deltas rather than trusting dashboard shape. Every successful user event should map to a defined outcome counter; every duration should have a clearly named clock boundary. Confirm version and environment dimensions let an operator distinguish old and new code without adding customer-specific cardinality.

Write an expected event sequence before looking at logs. Compare it to the observed sequence and explain every extra event, especially hidden retries or duplicate processing. A successful response with unexpected retries is operational debt because it consumes capacity and predicts a sharper failure under load. Verify that logs contain reason codes and correlation IDs but not request bodies, credentials, or private document content.

Vary one legitimate input dimension—request size, tenant class, queue age, or dependency latency—and keep the rest fixed. Identify where the normal path approaches the target. Save p50, p95, and maximum for a bounded sample, plus demand and saturation. The baseline is invalid if traffic count is missing.

Use the historical failure as a warning: a launch test used tiny cached prompts, declared 300 RPS safe, then production traffic with 1,800-token prompts saturated all model workers at 62 RPS. Explain which normal-path measurement would have made that incident easier to recognize. Also record the production tension: closed-loop virtual users hide coordinated omission while open arrival rates can overload a test environment unrealistically; production-shaped data improves validity but requires strict redaction and spend controls. The baseline page passes when another engineer can reproduce the journey and predict which telemetry changes before running it.