Secure load testing operations

You own a streaming chat completion gateway at POST /v1/chat/stream. The goal is to measure the saturation knee with a workload that preserves prompt size, streaming duration, tenant mix, and cache-hit distributions seen in production. The measurable target is hold 120 virtual users for 15 minutes with p95 time-to-first-token below 900 ms, p99 full response below 8 s, non-429 errors below 0.5%, and queue depth below 80; identify the first bottleneck rather than merely reporting requests per second. The known production tension is closed-loop virtual users hide coordinated omission while open arrival rates can overload a test environment unrealistically; production-shaped data improves validity but requires strict redaction and spend controls.

List every capability needed to configure, inspect, inject failure into, mitigate, and roll back Load testing for a streaming chat completion gateway. Separate read, change, and destructive permissions. Grant them to short-lived roles rather than permanent personal credentials. The ordinary workflow must not require administrator access, and emergency access must expire automatically and emit an audit event.

Review this operational configuration for dangerous defaults:

export const options = {
  scenarios: { ramp: {
    executor: 'ramping-arrival-rate', startRate: 20, timeUnit: '1s',
    stages: [{target: 60,duration:'5m'},{target:120,duration:'10m'},{target:180,duration:'5m'}],
    preAllocatedVUs: 200, maxVUs: 400 }},
  thresholds: { http_req_failed: ['rate<0.005'],
    'http_req_duration{phase:ttft}': ['p(95)<900'] }
};

Check fail-open versus fail-closed behavior, external endpoints, data retention, tenant isolation, command scope, and rollback authorization. Validate unknown fields and unsafe ranges. If a setting can affect all tenants, require staged application and an independent approver. Keep secrets in the platform's secret store; never print them in config dumps, command history, metrics, or drill artifacts.

Run the security and operations gate:

trufflehog filesystem fixtures/requests.json --fail
./scripts/estimate-load-cost.sh --vus 120 --minutes 15

Prove a read-only responder cannot perform the mutation, an authorized role can perform only the scoped change, and the audit record contains actor, UTC time, target, revision, and result. Then revoke or expire the role and repeat the denial check. For scripts, pin dependencies, quote user-controlled values, enable strict error handling, and provide a dry-run mode for high-impact operations.

The monitoring surface http_req_duration, time_to_first_token_ms, generation_queue_depth, active_streams, tokens_generated_total, provider_errors_total, and cost_usd_total must avoid raw prompts, document contents, credentials, email addresses, and unconstrained customer identifiers. Use reason-code enums and controlled dimensions. Restrict detailed logs by role and retention, and test redaction with deliberately planted synthetic secrets.

The system must still meet hold 120 virtual users for 15 minutes with p95 time-to-first-token below 900 ms, p99 full response below 8 s, non-429 errors below 0.5%, and queue depth below 80; identify the first bottleneck rather than merely reporting requests per second. under security constraints; a control that makes mitigation impossible during an outage is incomplete. Exercise the scenario step from 40 to 240 virtual users while forcing cache misses and limiting the model worker pool to eight; observe queue growth before latency collapse and abort if provider spend reaches $25. with the least-privilege role and confirm rollback remains possible if a dependency or identity provider is degraded. Estimate provider, compute, storage, and telemetry cost, and set a hard drill budget where applicable.

The prior incident was a launch test used tiny cached prompts, declared 300 RPS safe, then production traffic with 1,800-token prompts saturated all model workers at 62 RPS. Ask whether excess access, missing approval, poor auditability, or unsafe tooling could have worsened it. The core tradeoff is closed-loop virtual users hide coordinated omission while open arrival rates can overload a test environment unrealistically; production-shaped data improves validity but requires strict redaction and spend controls. Document who accepts the residual risk and when it will be reviewed.

Approval requires threat model, permission matrix, redaction test, audit sample, rollback owner, cost bound, and evidence that emergency access works and expires. Do not approve on policy prose alone.