Model Routing: How to Cut LLM Token Costs
TL;DR
- Model routing sends each request to the cheapest model capable of handling it, which is the largest single lever on LLM spend for most applications.
- Load balancing distributes requests across keys and providers for throughput and availability. It does not reduce token count, and treating it as a cost control leads teams to optimize the wrong layer.
- Semantic caching removes the model call entirely on a hit. Bifrost runs exact-match hashing first and embedding similarity only on a miss, because a semantic miss costs an embedding call on top of the full LLM call.
- Code Mode lets a model write Python to orchestrate several tools in one pass, using 50% fewer tokens and 40% less latency than sequential tool calls.
- Budgets and rate limits at the customer, team, virtual key, and provider level turn projected savings into an enforced ceiling.
Model routing is the practice of directing each request to a specific model based on the requirements of that request rather than sending everything to one default. It is the largest lever on LLM token cost available at the infrastructure layer, because most applications route traffic that a smaller model could answer to a frontier model by default. Bifrost, the open-source AI gateway built in Go by Maxim AI, applies routing, caching, and budget rules on the request path across 25+ providers and 10,000+ models through one OpenAI-compatible API. This guide covers what model routing is, what load balancing does and does not do for cost, and which controls actually reduce token spend.
What Is Model Routing?
Model routing is a gateway-layer control that selects which model serves a given request, based on rules covering the request type, the calling application, cost, or measured provider health. A router replaces the hardcoded model name in application code with a policy that can be changed without a deployment.
The economics are straightforward. Frontier models cost multiples of smaller models per token, and a large share of production traffic consists of classification, extraction, summarization, and formatting work that smaller models handle correctly. Routing that traffic down is not a quality compromise; it is a correction of a default that was never chosen deliberately.
Provider routing and routing rules in Bifrost direct requests to specific models, providers, and keys, with weighted strategies and automatic fallback chains attached. Cost calculation draws on the Model Catalog, which holds pricing data synced from provider sources, so routing decisions and spend accounting use the same numbers.
The decision of which request goes where is the substance of the work, and it is an active research area: the paper Universal Model Routing for Efficient LLM Inference formalizes the problem of selecting among a pool of models per query. Our guide to smart LLM routing and picking the optimal model per request covers the classification approaches in depth.
What Is Load Balancing, and Why It Does Not Reduce Token Cost
Load balancing, as AWS defines it, is the distribution of incoming traffic across multiple backend resources so that no single resource is saturated. In an AI context those resources are API keys and provider endpoints rather than web servers, but the mechanism is the same: spread the work to raise throughput and keep any one dependency below its limit.
Load balancing does not reduce token cost. A request routed to key B instead of key A sends the same prompt to the same model and is billed identically. Conflating the two is common and leads teams to tune distribution when the spend problem is model selection.
What load balancing does reduce is waste. Three effects are real and worth counting:
- Fewer rate-limit retries. Every 429 that triggers a retry is latency spent and, on some providers, a partially billed request. Spreading load across a key pool keeps individual quota windows below their ceiling.
- Fewer failed requests paid for twice. A request that fails after partial generation and is retried can be billed on both attempts.
- Better provider price arbitrage. When the same model is available from several providers at different rates, weighted distribution can favor the cheaper one.
Key management and load balancing handles weighted distribution across an API key pool with model-specific filtering. Adaptive load balancing in Bifrost Enterprise adjusts those weights from live error-rate and latency data, adding under 10 microseconds to route selection because weights are recomputed asynchronously every 5 seconds.
The distinction to hold onto: load balancing decides which copy of a resource serves the request, model routing decides which model does. Only the second changes the bill.
Where LLM Token Cost Actually Comes From
LLM spend decomposes into four drivers, and each has a different control. Auditing which one dominates before optimizing is what separates a real reduction from a rounding error.
| Cost driver | What it looks like | Control that addresses it |
|---|---|---|
| Model over-selection | Simple requests served by a frontier model | Model routing to a cheaper capable model |
| Repeated requests | The same or similar prompt answered repeatedly | Semantic caching |
| Oversized tool context | Many tool definitions and sequential tool calls | Code Mode and MCP tool filtering |
| Unbounded consumption | One team or agent loop consuming the budget | Budgets and rate limits |
Most teams find the first two account for the majority of avoidable spend. The fourth does not reduce unit cost at all; it caps exposure, which matters because the failure mode in agentic systems is a loop issuing thousands of requests rather than a gradual drift upward. Our breakdown of cost-aware routing and pushing traffic to the cheapest capable model works through the first row with numbers.
Cost-Aware Model Routing in Practice
Cost-aware routing assigns a model tier to each class of request and enforces that assignment at the gateway. In practice the classification is coarser than teams expect: three tiers covering cheap deterministic work, mid-range reasoning, and frontier-only tasks usually captures most of the available saving.
Routing rules can key on the calling application, the virtual key, the requested capability, or provider health. Because the Bifrost AI gateway evaluates them per request, changing a tier assignment is a configuration change rather than a release across every service that calls a model.
Routing and reliability share the same path. A fallback chain is a routing decision made under failure conditions, so the cheaper primary and the more expensive backup are configured together rather than as separate systems. The mechanics of that interaction are covered in our write-up on how adaptive model routing and fallback logic work.
Two cautions apply. Routing down on a task the smaller model handles badly produces retries, longer outputs, and human rework, none of which appear in the token bill directly. And routing decisions need measurement, so built-in observability on per-model volume and cost is what tells you whether a tier assignment was correct.
Semantic Caching: Removing the Request Entirely
A cached response costs nothing in model tokens, which makes caching the only control that removes spend rather than reducing it. Semantic caching in Bifrost runs two paths: direct hash matching for exact-match replay, and embedding-based similarity search for requests that differ in wording but not in meaning.
Direct matching runs first. The request is normalized and hashed, and an identical request is served from the vector store in sub-millisecond to single-digit-millisecond time with a local Redis or Valkey backend. No embedding provider is required for this path at all, and direct-only mode is a reasonable starting configuration.
Semantic matching runs only on a direct miss, and the cost profile is worth understanding before enabling it. A semantic lookup must embed the incoming request before it can search, which means one embedding API call paid upfront regardless of the outcome. A semantic hit therefore costs roughly an embedding round-trip rather than an instant replay, and a semantic miss pays that embedding call on top of the full LLM call, making it slower than running with no cache at all. Hit rate is what decides whether semantic mode pays for itself.
One operational detail accounts for most reports that caching is not working: a cache key is mandatory. Requests carry it through the x-bf-cache-key header or a configured default, and a request without one bypasses the cache entirely.
| Prompt caching | Semantic caching | |
|---|---|---|
| Where it runs | At the model provider | At the AI gateway |
| What it matches | A repeated prefix within a request | A whole request, exact or similar |
| Effect on the call | Call still happens, prefix billed at a reduced rate | Call is not made at all on a hit |
| Requires embeddings | No | Only for the similarity path |
Caching covers chat completions, text completions, the Responses API, embeddings, transcriptions, speech, and image generation, including their streaming variants. For the wider cost picture, see our gateway-level treatment of reducing LLM costs with semantic caching and the combined view in optimizing token consumption with caching and dynamic routing.
Cutting Tool-Call Tokens in Agentic Workloads
Agentic applications spend tokens differently. Every tool definition sits in the context window on every call, and a multi-step task issues a separate round trip per tool, re-sending the accumulated conversation each time. The token cost scales with the number of tools available, not the number actually used.
Two controls address this directly. Code Mode lets the model write Python that orchestrates several tools in a single pass, using 50% fewer tokens and 40% less latency than sequential tool calling. MCP tool filtering restricts which tools a given virtual key can see, which shrinks the definitions carried in context before any request is made.
The savings compound with scale, and our analysis of the MCP gateway's effect on access control and token costs covers the measured numbers on larger tool inventories.
Enforcing the Savings with Budgets and Rate Limits
Routing and caching reduce expected spend. Budgets make the reduction binding, which matters because the controls above are probabilistic and a runaway agent loop is not.
Budgets and rate limits apply at the customer, team, virtual key, and provider-config levels, and every applicable budget is checked independently on each request. Any single failure blocks the request, and a provider that has exceeded its budget is excluded from routing rather than being allowed to fail downstream. Virtual keys carry those limits per consumer, which is also what makes per-team cost attribution possible in the first place.
Attribution is the precondition for everything else here. Without a key per team or application, per-model spend data cannot be traced to an owner, and routing policy gets set on aggregate numbers that hide the traffic actually worth moving. The governance resource page covers how those controls fit together, and Bifrost Enterprise covers the deployment options for teams that need them in isolated environments.
Frequently Asked Questions
What are model routers?
Model routers are components that choose which model handles a given request, based on rules covering request type, cost, capability, or provider health. They sit between the application and the model providers, usually inside an AI gateway, so the model choice becomes a configuration decision rather than a value hardcoded in application code.
Does load balancing reduce LLM costs?
Not directly. Load balancing distributes requests across API keys and providers to raise throughput and avoid rate limits, but the same prompt sent to the same model costs the same on any key. It reduces waste from retries and failed requests, and can favor a cheaper provider for the same model, which is a smaller and indirect effect than model routing.
What is the difference between prompt caching and semantic caching?
Prompt caching runs at the model provider and discounts a repeated prefix within a request, so the call still happens. Semantic caching runs at the gateway and serves a stored response for an identical or similar request, so the model call is not made at all. They address different costs and can be used together.
What does AI routing mean?
AI routing covers any policy that decides where an AI request goes: which model, which provider, which API key, and what happens on failure. Model routing, as configured through per-request routing policy, is the cost-relevant subset of it. Load balancing and fallback chains are the throughput and reliability subsets, and in practice all three are configured at the same gateway layer.
How much can model routing reduce LLM spend?
The saving depends entirely on the current traffic mix, which is why an audit comes before a target. Applications sending classification, extraction, and formatting work to a frontier model by default have the most available, since the price gap between tiers is large. Measure per-model volume and cost first, then move one class of request at a time.
Should routing logic live in the application or the gateway?
At the gateway in nearly all multi-service cases, because a routing policy that requires a coordinated release to change stops being tuned. Request-specific judgment, such as deciding that a particular user action needs the strongest available model, belongs in the application and can be expressed as a header the gateway honors.
Start Routing with Bifrost
Routing, semantic caching, tool-call reduction, and enforced budgets attack four different parts of the same bill, and they are most effective applied in that order: route first, because it has the largest effect, then cache, then reduce tool context, then cap what remains. Implementing them at the gateway keeps the policy in one place as the number of models and services grows, which is the same reason reliable fallback design belongs there. Bifrost is a drop-in replacement for existing SDKs, so adoption is usually a base URL change.
To see what model routing would do to your own traffic mix, book a demo with the Bifrost team, or check the supported provider matrix for coverage of the models you run today.
