Test load testing signals and thresholds

You own a streaming chat completion gateway at POST /v1/chat/stream. The goal is to measure the saturation knee with a workload that preserves prompt size, streaming duration, tenant mix, and cache-hit distributions seen in production. The measurable target is hold 120 virtual users for 15 minutes with p95 time-to-first-token below 900 ms, p99 full response below 8 s, non-429 errors below 0.5%, and queue depth below 80; identify the first bottleneck rather than merely reporting requests per second. The known production tension is closed-loop virtual users hide coordinated omission while open arrival rates can overload a test environment unrealistically; production-shaped data improves validity but requires strict redaction and spend controls.

Convert the Load testing objective into tests, not screenshots. The target is hold 120 virtual users for 15 minutes with p95 time-to-first-token below 900 ms, p99 full response below 8 s, non-429 errors below 0.5%, and queue depth below 80; identify the first bottleneck rather than merely reporting requests per second. Write one assertion for the exact boundary, one immediately inside it, and one immediately outside it. Include denominator behavior when traffic is zero, missing, duplicated, delayed, or partially sampled.

The telemetry contract is http_req_duration, time_to_first_token_ms, generation_queue_depth, active_streams, tokens_generated_total, provider_errors_total, and cost_usd_total. For every series, verify metric type, unit, label allowlist, ownership, and retention. Counters must only rise; histograms need buckets around the target; gauges require staleness handling. Confirm dashboards display demand, successful or protected outcomes, user latency or age, saturation, and version context together. A green ratio without its event count is not sufficient evidence.

Run the test workflow:

k6 run --summary-export artifacts/load-summary.json scenarios/chat-stream.js
python scripts/assert-load-budget.py artifacts/load-summary.json

Store machine-readable output as a build artifact. Seed deterministic time or fixtures where possible, but keep at least one integration test against the actual collector and rule engine. Tests must fail when the threshold expression, label name, or alert receiver is intentionally broken. A test that remains green after its signal is disconnected is worse than no test because it creates false confidence.

Use the configured contract:

export const options = {
  scenarios: { ramp: {
    executor: 'ramping-arrival-rate', startRate: 20, timeUnit: '1s',
    stages: [{target: 60,duration:'5m'},{target:120,duration:'10m'},{target:180,duration:'5m'}],
    preAllocatedVUs: 200, maxVUs: 400 }},
  thresholds: { http_req_failed: ['rate<0.005'],
    'http_req_duration{phase:ttft}': ['p(95)<900'] }
};

Replay the controlled scenario—step from 40 to 240 virtual users while forcing cache misses and limiting the model worker pool to eight; observe queue growth before latency collapse and abort if provider spend reaches $25.—and measure time from fault start to metric visibility, alert pending, alert firing, delivery, acknowledgement where relevant, mitigation, and stable recovery. Compare those times with the operational objective. Verify short benign blips do not page if the policy says they should not, while sustained or fast-burning impact does.

Check telemetry loss separately from service loss. Stop or misconfigure one scrape target and prove the system reports missing data rather than silently treating it as healthy. Verify logs and drill artifacts redact secrets and private payloads. Bound label cardinality using a representative workload and set an alert or budget for unexpected series growth.

Use the prior incident—a launch test used tiny cached prompts, declared 300 RPS safe, then production traffic with 1,800-token prompts saturated all model workers at 62 RPS.—to ask which test would have detected the problem earlier. Remove or demote signals that do not lead to a decision. The production tradeoff is closed-loop virtual users hide coordinated omission while open arrival rates can overload a test environment unrealistically; production-shaped data improves validity but requires strict redaction and spend controls. Quantify the cost of collection and paging against the risk reduction it buys.

This page passes only when contract tests, failure tests, alert-path tests, and recovery assertions all run from a clean environment and produce retained evidence linked to the configuration revision.