Streaming responses
Cover security and operational gates
Reduce authority and blast radius: build an HTTP endpoint that relays model text as Server-Sent Events while preserving cancellation and final usage as a bounded, testable system whose behavior does not depend on trusting plausible model output.
Before you start
Why this matters
Before coding, write the single observable result this page must add to the previous page. Then name one failure that the implementation must reject. For Streaming responses, use the running project—not a toy explanation—as your test: an HTTP endpoint that relays model text as Server-Sent Events while preserving cancellation and final usage. Predict what the CLI, test, or HTTP client will observe when the page is complete.
1Learn the idea
Read
Reduce authority and blast radius
Assume user text, retrieved content, tool output, and remote server descriptions are untrusted data. They can inform a result but cannot rewrite system policy, select credentials, expand an allowlist, or approve an effect. Enforce authorization in code at execution time.
Production controls include secret rotation, egress restrictions, quotas, concurrency limits, payload caps, audit retention, dependency review, incident ownership, and a kill switch. Log metadata by default; sampling sensitive content requires an explicit retention and access policy.
The project-specific rule is: authenticate before opening the stream, cap duration/output, and avoid leaking internal errors. Apply least privilege before the call, not after inspecting its result. Treat all remote descriptions and generated arguments as proposals. Local code owns allowlists, authorization, quotas, and approval checks. Apply input, output, concurrency, duration, and cost limits.
Threat-model prompt injection, cross-tenant access, secret disclosure, dependency compromise, denial of wallet, and replayed effects. Record safe audit facts: who initiated the action, which capability was requested, policy decision, terminal status, and correlation ID. Avoid raw payload retention by default. Document a kill switch and test that disabling the feature prevents new work while allowing in-flight cancellation.
Read
Implementation
function authorize(ctx: Context, action: Action) {
if (!ctx.permissions.has(action.permission)) {
throw new SafeError("forbidden");
}
if (action.risk === "write" && !ctx.approvalToken) {
throw new SafeError("approval_required");
}
}
The code is intentionally provider-neutral at the adapter boundary. AI_BASE_URL and model names are environment placeholders, not invented services. If the topic uses MCP, install the current official SDK and verify its exported paths against its release documentation. If a provider offers an official SDK, it can replace fetch inside execute without changing the domain contract. Always inspect the real provider’s documented request and response fields before connecting credentials.
Read
Debug the boundary
Debug from deterministic code outward. First print the parsed configuration with secret values replaced by [set]. Next run the parser against a local fixture. Then run the orchestration with a fake dependency. Only after those pass should you inspect the real boundary. Capture status, content type, request ID, elapsed time, and a redacted response shape. Do not “fix” malformed data with a permissive cast; save it as a failing fixture and decide whether the contract or adapter is wrong.
For this chapter, the fastest diagnostic is to assert that every accepted stream ends exactly once as done, error, or client cancellation. When that assertion fails, stop before downstream work. Expected behavior is concrete: The client renders “Hello 👋” in order, receives done once, and closes cleanly. A different but plausible sentence is not enough if the structural and policy checks fail.
Read
Verification notes
- Capabilities are allowlisted and authorized in code.
- Payload, cost, concurrency, and duration limits exist.
- Kill switch and incident owner are documented.
- Run
npm run typecheckandnpm testwithout provider credentials. - Record expected terminal behavior beside the fixture so future changes remain reviewable.
A passing test is necessary but does not establish production quality. Review the fixture set for realistic distributions, stale assumptions, tenant boundaries, and expensive edge cases. When outputs are probabilistic, assert schemas, citations, chosen capabilities, stop reasons, and task rubric scores. Reserve exact string assertions for deterministic adapters and protocol framing.
Continue learning · glossary & guides
- Can you point to the executable check proving that every accepted stream ends exactly once as done, error, or client cancellation?
- Does malformed or unauthorized data stop before external or privileged work?
- Is the observable success criterion exactly this clear: The client renders “Hello 👋” in order, receives done once, and closes cleanly.
- Glossary: tool · Glossary: structured output · Cheatsheet: production ops signals
- Previous · Next