Reference · Glossary
Model router / gateway
Last updated
A layer in front of your app's LLM calls that picks **which model** handles each request — by task type, a difficulty/confidence score, or a fallback chain — so easy requests go to cheap/fast models and only hard ones reach an expensive flagship model.
#When to use
Any production app with meaningful request volume and a mix of easy/hard queries, where routing can cut cost and latency without hurting quality on the requests that matter.
#When not to
A low-volume prototype or internal tool where the cost difference between models is negligible — added routing logic isn't worth the complexity yet.
#Example
def pick_model(query: str) -> str:
if is_simple_faq(query):
return "gpt-4o-mini"
return "gpt-4o"A minimal router: classify first, then call the cheaper model for routine questions and reserve the flagship model for the rest.