Reference · Glossary

Latency

How long users wait for an AI response — from sending a prompt to seeing the first token (TTFT) or the full reply.

#When to use

Sizing production systems, choosing model size, deciding on streaming vs batch.

Don't optimize latency before correctness — a fast wrong answer still fails.

Streaming shows tokens in ~300 ms (time to first token) while the full 500-token answer finishes in 4 s.