Chapter DStreaming responsesPage 1 of 8

Streaming responses

Define the lab goal and success criteria

Define the executable outcome: build an HTTP endpoint that relays model text as Server-Sent Events while preserving cancellation and final usage as a bounded, testable system whose behavior does not depend on trusting plausible model output.

~13 minLab goal

1Try it yourself

Playground

Streaming tokens

Stream shows tokens as they arrive — better UX for long replies.

Assistant

Before you start

Why this matters

Before coding, write the single observable result this page must add to the previous page. Then name one failure that the implementation must reject. For Streaming responses, use the running project—not a toy explanation—as your test: an HTTP endpoint that relays model text as Server-Sent Events while preserving cancellation and final usage. Predict what the CLI, test, or HTTP client will observe when the page is complete.

2Learn the idea

Read

Define the executable outcome

Begin with one vertical slice and measurable acceptance criteria. Separate model quality from software correctness: transport, parsing, authorization, and termination must be deterministic even when model content varies. The lab is complete only when a fixture can demonstrate the expected behavior and a negative fixture demonstrates a safe rejection.

Write a README-level contract before installing packages. Record the input, output, owner, latency ceiling, cost ceiling, and stop condition. This prevents an attractive demo from silently becoming an unbounded production promise.

Success has two layers. Software success means every accepted stream ends exactly once as done, error, or client cancellation. Product success means a user receives a useful result without hidden authority or an endless wait. For this lab, the concrete oracle is: The client renders “Hello 👋” in order, receives done once, and closes cleanly. Measure fixture pass rate, safe-rejection rate, p95 duration, and external calls per successful task. Do not use “the response looked good” as an acceptance test.

Create src/, test/, .env.example, and package.json. The first deliverable is not a framework; it is a fixture containing representative input, expected structural properties, and prohibited behavior. A useful negative fixture targets split UTF-8 chunks, broken SSE frames, stalled connections, duplicate deltas, and disconnects. That negative case forces the boundary to exist before provider output can surprise it.

Read

Implementation

export const acceptance = {
  project: "an HTTP endpoint that relays model text as Server-Sent Events while preserving cancellation and final usage",
  invariant: "every accepted stream ends exactly once as done, error, or client cancellation",
  expected: "The client renders \u201cHello \ud83d\udc4b\u201d in order, receives done once, and closes cleanly.",
} as const;

The code is intentionally provider-neutral at the adapter boundary. AI_BASE_URL and model names are environment placeholders, not invented services. If the topic uses MCP, install the current official SDK and verify its exported paths against its release documentation. If a provider offers an official SDK, it can replace fetch inside execute without changing the domain contract. Always inspect the real provider’s documented request and response fields before connecting credentials.

Read

Debug the boundary

Debug from deterministic code outward. First print the parsed configuration with secret values replaced by [set]. Next run the parser against a local fixture. Then run the orchestration with a fake dependency. Only after those pass should you inspect the real boundary. Capture status, content type, request ID, elapsed time, and a redacted response shape. Do not “fix” malformed data with a permissive cast; save it as a failing fixture and decide whether the contract or adapter is wrong.

For this chapter, the fastest diagnostic is to assert that every accepted stream ends exactly once as done, error, or client cancellation. When that assertion fails, stop before downstream work. Expected behavior is concrete: The client renders “Hello 👋” in order, receives done once, and closes cleanly. A different but plausible sentence is not enough if the structural and policy checks fail.

Read

Verification notes

  • One positive fixture and one adversarial fixture are written.
  • Success, cost, latency, and stop conditions have numeric targets.
  • Provider variability is separated from deterministic software behavior.
  • Run npm run typecheck and npm test without provider credentials.
  • Record expected terminal behavior beside the fixture so future changes remain reviewable.

A passing test is necessary but does not establish production quality. Review the fixture set for realistic distributions, stale assumptions, tenant boundaries, and expensive edge cases. When outputs are probabilistic, assert schemas, citations, chosen capabilities, stop reasons, and task rubric scores. Reserve exact string assertions for deterministic adapters and protocol framing.

Checking tutor…

Continue learning · glossary & guides
  • Can you point to the executable check proving that every accepted stream ends exactly once as done, error, or client cancellation?
  • Does malformed or unauthorized data stop before external or privileged work?
  • Is the observable success criterion exactly this clear: The client renders “Hello 👋” in order, receives done once, and closes cleanly.
  • Glossary: tool · Glossary: structured output · Cheatsheet: production ops signals
  • Next