Webhook lab
Run acceptance tests and make the ship call
Build a FastAPI receiver accepting batch-index completion events and handing validated work to a durable queue as an operable release, not a slide-deck example.
1Try it yourself
Playground
Webhook lab
Webhooks push events — polling pulls. Verify signatures before acting.
Nightly index job finished
Before you start
Why this matters
Before changing code, write the single production outcome this chapter must prove and the signal that would stop you. For this lab, the service boundary is raw bytes plus X-Event-Id, X-Timestamp, and X-Signature headers become a normalized BatchCompleted event only after authentication. Record one request identifier you can follow from ingress through the final decision. If you cannot name the owner of the stop decision, the rollout is not yet controlled.
The source lesson says webhooks push completion events and must be signature-verified and idempotent before work begins. This chapter turns that compact lesson into implementation evidence. The running scenario is a FastAPI receiver accepting batch-index completion events and handing validated work to a durable queue. You will keep the same scenario across all eight chapters so setup decisions, tests, telemetry, and rollback controls accumulate into one coherent system rather than eight disconnected exercises.
2Learn the idea
Read
Assemble the release candidate
Freeze a candidate by recording source SHA, dependency lock digest, configuration revision, fixture digest, and deployment artifact. Run setup from a clean checkout, then execute unit, contract, integration, quality, and failure suites in that order. Do not substitute an old green build for evidence about the frozen candidate.
Review the contract: raw bytes plus X-Event-Id, X-Timestamp, and X-Signature headers become a normalized BatchCompleted event only after authentication. Confirm one realistic request completes the happy path and can be traced end to end. Confirm malformed and unauthorized requests fail before side effects. Confirm duplicate or repeated requests preserve required consistency. Confirm dependency timeout enters the documented degraded mode.
Read
Run the acceptance plan
Execute these acceptance cases and attach evidence:
- Functional: the canonical fixture returns the contracted structured result and correct attribution.
- Quality: the versioned golden set meets the primary threshold with reviewed case-level failures.
- Performance: expected concurrency stays within the latency and saturation budget.
- Resilience: timeout, malformed response, and unavailable dependency produce bounded behavior.
- Security: cross-tenant or forged input is rejected with zero unauthorized side effects.
- Operations: an alert fires, the runbook is followed, and safe behavior is restored within the target.
- Rollback: pause workers while continuing durable inbox writes, disable the new event handler by type, and replay quarantined events after repair.
The metric decision uses signature rejects, duplicate rate, receive-to-ack latency, queue age, processing latency, retries, and dead-letter count. Require complete telemetry and the minimum observation window. Record sample counts and confidence where comparisons are statistical. A passing average cannot override a hard security, correctness, or latency guardrail.
Read
Make the ship decision
Use three outcomes. Ship means all blocking tests pass, rollback is timed, dashboards and alerts are live, ownership is accepted, and residual risks have dates. Hold means evidence is missing or a non-critical defect has an explicit repair owner. Rollback means a hard guardrail, security boundary, data-integrity invariant, or recovery target failed.
Conduct a 30-minute game day after deployment. Inject one dependency failure or risky-control breach, let the on-call responder diagnose from telemetry, execute the runbook, and verify recovery from user-visible symptoms. Capture detection time, decision time, recovery time, and any manual step.
Read
Production ship packet
The final packet contains architecture and trust-boundary diagrams, API or event contracts, configuration, test report, golden-set digest, performance result, dashboard and alert links, security review, rollback transcript, runbook, owner, and dated residual risks. The system is shipped only when another operator can use that packet to understand what is live, detect harm, stop exposure, and recover without the author present.
Checking tutor…