Reference · Glossary
Latency
How long users wait for an AI response — from sending a prompt to seeing the first token (TTFT) or the full reply.
When to use
Sizing production systems, choosing model size, deciding on streaming vs batch.
When not to
Don't optimize latency before correctness — a fast wrong answer still fails.
Example
Streaming shows tokens in ~300 ms (time to first token) while the full 500-token answer finishes in 4 s.