Reference · Glossary
Inference
Last updated
Every production call: chat completions, RAG retrieval embeddings, classification, TTS. Optimize inference for latency, cost, and reliability.
#When to use
Every production call: chat completions, RAG retrieval embeddings, classification, TTS. Optimize inference for latency, cost, and reliability.
#When not to
Do not confuse inference tuning (batching, quantization, caching) with fixing a wrong model — bad training or bad prompts still yield bad answers.
#Quality checklist
- Measure p50/p95 latency and error rates
- Pin model + tokenizer versions
- Separate warm-up from steady-state load tests
- Know your context-length and rate limits
#Example
Serving: gpt-class chat model behind an API
Controls: max tokens, temperature, timeout, retries
Observe: tokens/sec, queue depth, 5xx rate#Learn next
- Lesson: `training-vs-inference`
- Related: latency, quantization