Constrain access and rehearse rollback

Mark user input, application runtime, configuration service, external provider, durable store, and operator console. At each crossing, specify authentication, authorization, validation, encryption, and audit evidence. The system’s concrete policy is: verify HMAC over the exact raw body with constant-time comparison, reject stale timestamps, limit body size, and rotate the webhook secret. Test that policy with a negative request, not only a configuration screenshot.

Secrets enter at runtime through a secrets manager or workload identity. They never appear in source, fixtures, process arguments, telemetry, or exception bodies. Scope credentials to the smallest resource and operation. Production configuration writes require role-based access, review for high-risk changes, and an immutable record with actor and reason.

Set body and field-size limits before parsing expensive input. Normalize only after signature or identity checks where raw bytes are security-relevant. Treat retrieved documents, webhook payloads, and tool responses as untrusted data. Apply output schemas and explicit allowlists before any side effect. For customer data, minimize collection, define retention, and verify deletion reaches logs or indexes where promised.

Build abuse tests for replay, cross-tenant identifiers, prompt injection, forged configuration, stale credentials, and denial-of-service through expensive requests. Assert both response and absence of side effects. A 403 that still performed a provider call is a security and cost bug.

Execute: pause workers while continuing durable inbox writes, disable the new event handler by type, and replay quarantined events after repair. Use a two-person checklist for production-like drills: announce start, snapshot current revision and metrics, apply the control, verify safe behavior, reconcile in-flight work, and announce completion. Ensure rollback permissions are available to on-call staff without granting broad routine admin access.

Distinguish rollback from evidence destruction. Keep the failed artifact, configuration digest, audit events, and redacted traces under retention policy. If compromise is possible, rotate credentials and isolate the revision before analysis. Never reactivate a known-compromised secret merely because it restores service.

The runbook must name owner, escalation path, safe-mode behavior, exact control, verification query, and conditions for re-enabling. Run it with someone who did not write the code. Measure completion time and note any undocumented access. The chapter passes only when least privilege, negative tests, audited changes, and a timed rollback drill all produce evidence.

Separate runtime and operator permissions. The application receives one capability, while the operator role owns the audited recovery action. Security invariant: Stale, replayed, oversized, or incorrectly signed payloads reach no queue.

roles:
  runtime:
    principal: webhook-receiver
    allow:
      - queue:publish/batch-events
    deny:
      - admin:*
  break_glass_operator:
    allow:
      - queue:publish/batch-events
    conditions:
      reason_required: true
      approval_count: 2
      ttl_minutes: 30

Run a negative probe before the positive path. The exact status may be 401 or 403 by contract, but no downstream call or durable mutation may occur. Audit output names the decision, never credentials or payload contents.

$ curl -si -H 'Authorization: Bearer invalid-fixture' http://localhost:8000/webhooks/batch-completed
HTTP/1.1 401 Unauthorized
$ printf '%s\n' '{"decision":"deny","principal":"unknown","side_effects":0}'
{"decision":"deny","principal":"unknown","side_effects":0}

If side_effects is nonzero, move authorization ahead of retrieval, queue publication, or provider calls; changing only the status code is not a fix. If runtime unexpectedly has admin rights, inspect role inheritance and workload identity binding. Record actor, reason, approvals, prior webhook-schema-v2, and expiry during rollback. A break-glass token still valid after the drill is a failed security test.