Triage and recover load testing

You own a streaming chat completion gateway at POST /v1/chat/stream. The goal is to measure the saturation knee with a workload that preserves prompt size, streaming duration, tenant mix, and cache-hit distributions seen in production. The measurable target is hold 120 virtual users for 15 minutes with p95 time-to-first-token below 900 ms, p99 full response below 8 s, non-429 errors below 0.5%, and queue depth below 80; identify the first bottleneck rather than merely reporting requests per second. The known production tension is closed-loop virtual users hide coordinated omission while open arrival rates can overload a test environment unrealistically; production-shaped data improves validity but requires strict redaction and spend controls.

Assume the bounded failure is active: step from 40 to 240 virtual users while forcing cache misses and limiting the model worker pool to eight; observe queue growth before latency collapse and abort if provider spend reaches $25. Begin with the user-visible objective—hold 120 virtual users for 15 minutes with p95 time-to-first-token below 900 ms, p99 full response below 8 s, non-429 errors below 0.5%, and queue depth below 80; identify the first bottleneck rather than merely reporting requests per second.—and verify demand is present. Classify severity from measured impact, not from the apparent prestige of the dependency. Create a UTC timeline and record every observation separately from every hypothesis.

Query http_req_duration, time_to_first_token_ms, generation_queue_depth, active_streams, tokens_generated_total, provider_errors_total, and cost_usd_total by stable dimensions: environment, version, outcome, dependency, and instance or region where cardinality permits. Compare an affected slice with a healthy control. Narrow in this order: deployment/configuration change, tenant or workload shape, dependency response, resource saturation, then individual instance. That order reduces random restarts and preserves useful evidence.

Use the diagnostic workflow:

curl -s 'localhost:9090/api/v1/query?query=generation_queue_depth' | jq .
pidstat -p $(pgrep gateway) 1 10

Before each command, write what result supports the current hypothesis and what result falsifies it. A query that cannot change your decision is noise during an incident. Preserve correlation identifiers and exact query ranges. Do not paste secrets or customer content into the timeline.

The relevant operational configuration is:

export const options = {
  scenarios: { ramp: {
    executor: 'ramping-arrival-rate', startRate: 20, timeUnit: '1s',
    stages: [{target: 60,duration:'5m'},{target:120,duration:'10m'},{target:180,duration:'5m'}],
    preAllocatedVUs: 200, maxVUs: 400 }},
  thresholds: { http_req_failed: ['rate<0.005'],
    'http_req_duration{phase:ttft}': ['p(95)<900'] }
};

Prefer a reversible action scoped to the failing dimension: disable one flag, quarantine one message, reduce one concurrency pool, route one tenant, or select a verified recovery artifact. Give the mitigation an owner and expiry. Scaling, restarting, widening a timeout, suppressing an alert, or purging state may hide symptoms while increasing cost or destroying evidence; require an explicit reason before using them.

After the action, verify both harm reduction and side effects across several samples. Check deferred work and retries so the incident does not recur when a queue drains or a circuit closes. If the metric recovers but demand vanished, mitigation is unproven.

The historical case was a launch test used tiny cached prompts, declared 300 RPS safe, then production traffic with 1,800-token prompts saturated all model workers at 62 RPS. Build a causal tree with at least two competing explanations, and identify the observation that eliminated each losing branch. Do not claim root cause solely because rollback correlated with recovery. Preserve build version, configuration revision, dependency status, and relevant state for later analysis.

The design tension is closed-loop virtual users hide coordinated omission while open arrival rates can overload a test environment unrealistically; production-shaped data improves validity but requires strict redaction and spend controls. Record whether mitigation favored availability, correctness, fairness, cost, or responder safety. This page passes when a second responder can reconstruct why the selected action was proportionate and why broader actions were rejected.