Reference · Glossary
Batching (inference)
Last updated
Running **multiple model requests together** in one forward pass to improve GPU throughput — often at the cost of some latency.
When to use
High-volume serving where maximizing tokens/sec matters more than instant first token for every user.
When not to
Interactive chat where each user expects immediate streaming — tiny batches or single requests may feel snappier.
Example
A server queues 8 embedding requests for 20ms, runs one GPU batch, returns all eight vectors together.