Debug buffering the whole answer before first byte in the SSE token relay

Do not start with a speculative fix for the SSE token relay. Force the failure on purpose, save the before output, then change one cause at a time. Retries are allowed only for transient conditions—not for bad input that will fail forever on streaming-responses.

for (let attempt = 0; attempt <= maxRetries; attempt++) {
  try { return await operation(signal); }
  catch (error) {
    const failure = classify(error);
    if (!failure.retryable || attempt === maxRetries) throw failure;
    await sleep(jitteredBackoff(attempt), signal);
  }
}
throw new Error("unreachable");

Expected evidence: The client renders “Hello 👋” in order, receives done once, and closes cleanly.. If you cannot reproduce on demand, you do not yet control the failure mode for streaming-responses.

After repair, rerun the exact reproduction command. Keep the failing fixture as a regression seed for the observability page. For the SSE relay for model tokens, remember the claim you are restoring: client receives ordered data events and a terminal done event; cancel stops further writes.

Store a one-command reproduction for: buffering the whole answer before first byte, or ignoring client disconnect. The command should use fake chunk iterator yielding three tokens then done or a minimal mutant of it. Paste the failing output into notes/failure-before.txt (or your shell scrollback as copied text). After the fix, paste notes/failure-after.txt and keep both.

Retries belong only on transient faults. If the failure is bad input, a bad allowlist, or a logic bug in the SSE token relay, retrying will amplify cost without repairing trust around stream tokens to a client while honoring cancel and partial failure.

Classify the failure as prevent, detect, contain, or recover—using this lab’s language, not a generic poster. For streaming-responses, the first fix should usually be detect+prevent at the boundary, because buffering the whole answer before first byte, or ignoring client disconnect is cheaper to stop early than to explain in production prose.

At the debugging stage for streaming-responses, the job is narrower than finishing a product demo. You are creating one progressive evidence piece about fake chunk iterator yielding three tokens then done that later pages inherit without redefining success. Keep that fixture small enough to inspect by hand, keep outputs copy-pasteable as text, and refuse to narrate this baseline as if it were a production SLA: non-streaming JSON endpoint latency for the same prompt.

For this page specifically, success looks like before/after evidence for the characteristic failure while still centering the user decision to stream tokens to a client while honoring cancel and partial failure. If you cannot point to a file, command, or assertion that proves that for the SSE token relay, stay on this page instead of advancing.

Glossary: tool · Glossary: structured output · Cheatsheet: production ops signals

Previous · Next