Reference · Glossary
Model serving
Last updated
Anytime users or other services call a model continuously. Serving choices dominate cost and latency after the prototype works.
#When to use
Anytime users or other services call a model continuously. Serving choices dominate cost and latency after the prototype works.
#When not to
One-off notebook experiments. Do not “productionize” until you have evals, timeouts, and a rollback plan.
#Quality checklist
- Health and readiness probes
- Timeouts, retries, and backpressure
- Version pins and canary deploys
- Token/cost accounting per tenant
#Failure modes to watch
- GPU OOM under long contexts
- Silent quality drop after a model swap
- Hot keys saturating one replica
#Learn next
- Lesson: `serving-llms`
- Related: inference, observability