Small models for small jobs
Use a small model (like gpt-4o-mini) for routing and simple steps, and a big model only where reasoning matters.
Last updated
Analogy
You don't spin up a GPU instance to serve a favicon.
In an agent
The biggest cost lever in most agents. Track cost per request from usage tokens to see it working.
Practice
Flashcards
- Interview: biggest lever for agent cost?Route simple steps to small models and use big models only for hard reasoning.
Interview questions
- How do you reduce cost and latency?
- Small models for routing and simple steps; big models only for hard reasoning.
- Caching and short tool results.
- Parallel tool calls and streaming.
- Track cost per request from usage tokens.
Questions people ask
How do I reduce the cost of an AI agent?
Route by difficulty. Use a small, cheap model, such as gpt-4o-mini, for routing, classification and simple steps, and a large model only where real reasoning is needed. Also keep tool results short and cache repeated work. Track cost per request from the usage tokens to confirm each change helped.
Will a smaller model make my agent worse?
Only if it does a job that needs more reasoning. Small models handle routing, extraction and simple tool calls well. Test each step on your eval set with the small model; keep the large model for the steps where scores really drop. That way you save money without losing quality.