Reference · Cheatsheet
Streaming vs batch inference
Last updated
- Interactive chat or copilot
| | Streaming | Batch |
|---|-----------|-------|
| **User sees** | Tokens appear live | Full reply at once |
| **Best for** | Chat UX | JSON parsing, jobs |
| **Server** | Long-lived HTTP/SSE | Single response body |
| **Failure** | Partial text possible | All-or-nothing easier |
Pick streaming when
- Interactive chat or copilot
- Perceived speed matters more than total latency
Pick batch when
**Related:** `streaming-responses`, `serving-llms`