Model routing
Sending each AI request to the model best suited to it, such as a cheap, fast one for simple turns and a stronger one for hard cases.
Updated October 2026 · with examples from our own builds
What does model routing mean in practice?
Not every request needs the most capable model. A greeting or a simple classification can go to a small, cheap model; a question with a photo or a long, detailed problem can go to a larger one. A router decides per request, using rules or a quick first call.
Routing is one of the most effective cost levers in an AI product, because the stronger model can cost many times more per turn and most turns don't need it.
It also gives a natural fallback chain. If the chosen model fails, try the cheaper one, then a fixed answer, so users never see an error.
Model routing at a glance
- Routes by
- Rules or a quick classification call
- Saves
- Cost on the many simple requests
- Doubles as
- A fallback chain when a model fails
Where has model routing come up in our builds?
Herb uses a small model for greetings and simple questions and a larger one when a turn has a photo, uses diagnostic words, asks two or more questions, runs past 220 characters or sits deep in a long thread. If a call fails it falls back to the cheaper model and then to a fixed answer.
A short first call classifies each message as build, edit or chat; edits and chat apply operations directly, while builds go through the full plan, assemble and cross-check pipeline.
Common questions about model routing
Will users notice when a cheaper model answers?
Not if the routing rules send the hard cases to the stronger model. Test with real conversations and review the turns the cheaper model handled.
What should happen if the AI provider is down?
Fall back to another model, then to a fixed, honest answer. A chat that says it can't answer right now is better than an error screen.
Related terms
Planning a product and not sure where model routing fits? Ask us, or browse all 116 terms in the glossary.