LLM Routing: How Routing Decisions Get Made
LLM routing is the per-request choice of which provider, model, and API key serve an LLM call. This guide walks through the signals a router evaluates, the precedence order Bifrost applies, what happens when a route fails, and how to debug a routing decision from its logs.
TL;DR
- LLM routing produces three outputs per request: a primary provider and model, an ordered fallback chain, and the API key used for each attempt.
- Routing signals fall into four families: request content, caller identity, capacity and cost, and live provider health.
- Bifrost evaluates explicit intent before health: routing rules first, then virtual key weights, then Enterprise adaptive routing and the circuit breaker, then the model catalog as the last resort.
- Retries happen inside one provider and fallbacks happen across providers; each provider in the chain gets its own full retry budget.
- Bifrost writes every routing decision to a per-request routing log, so a misrouted request can be traced to the rule, weight, or health signal that chose it.
LLM routing is the decision, made for each request, about which provider, model, and API key will serve an LLM call and what happens if that choice fails. Bifrost, the open-source AI gateway built by Maxim AI, makes that decision in a fixed, documented order, and that order matters more than any single routing feature. This post covers the signals a router evaluates, how precedence resolves conflicts, how failures are handled, and how to read a routing decision afterward.
What Is LLM Routing?
LLM routing is the process of selecting a provider, model, and credential for each LLM request based on rules, request content, budgets, and provider health, then defining what to try next if that selection fails. The component that performs it is an LLM router, usually running inside an LLM gateway.
For router types and the router-versus-gateway distinction, see our full explainer on what an LLM router is and how model routing works. This post covers what happens between a request arriving and a provider receiving it. Because signals conflict, a router is only predictable when the precedence between them is defined.
Bifrost runs this logic as its LLM gateway layer, in front of 10,000+ models across 25+ providers behind one OpenAI-compatible API.
The Signals an LLM Router Evaluates
An LLM router evaluates four families of signals: what the request asks for, who is asking, how much budget and rate-limit capacity remains, and how healthy each provider and key is right now. Configuration covers the first three; only the fourth requires live measurement of traffic.
| Signal family | Example inputs | Used for | Where it lives in Bifrost |
|---|---|---|---|
| Request content | Model, request type, headers, complexity tier | Matching work to model capability | Routing rules and the complexity router |
| Caller identity | Virtual key, team, customer | Per-tenant policy, compliance | Rule scopes, virtual key provider configs |
| Capacity and cost | Budget and rate-limit usage | Shifting traffic before a limit is hit | budget_used, tokens_used, request variables |
| Live health | Error rate, latency, refused keys | Avoiding degraded providers and keys | Adaptive routing (Enterprise) |
In Bifrost, budget_used is a percentage of the matching budget or rate limit; when several apply the highest wins, and with none configured it is 0, so budget_used > 80 never fires. Per-request model choice is also covered in smart LLM routing and picking the optimal model per request.
Semantic Routing and Complexity Tiers
Semantic routing classifies a request by meaning and sends it to a model sized for that work. In Bifrost, classification produces a complexity_tier of SIMPLE, MEDIUM, or COMPLEX, and that tier is a condition inside a routing rule, so it follows the same precedence as any other rule.
The complexity router offers two classifiers:
- Decision model: reads the latest user request and picks the tier whose definition fits. Typesafe Jev is the default, typically answering in 600 to 800 ms.
- Semantic: embeds the request and takes the tier of its nearest reference phrase, from 150 shipped defaults (up to 750).
Classification runs only when a rule references complexity_tier. If the classifier times out or finds no confident match, evaluation falls through to the next rule or to virtual key weights. With session-aware routing on, a session's tier only moves up, so agent conversations do not switch models mid-task. Accuracy and latency figures are in the benchmarks for the upgraded complexity router.
Learned routing is the research form of this idea: the RouteLLM paper from UC Berkeley and Anyscale researchers trained routers on human preference data and reported over 2x cost reduction without sacrificing response quality. NVIDIA's LLM Router blueprint routes by task type or complexity category. See also our roundup of semantic routing platforms for LLM applications.
Routing Rules and the Order of Precedence
Routing precedence decides which signal wins when signals disagree. In Bifrost, explicit intent is evaluated before measured health: routing rules, then virtual key weights, then Enterprise adaptive routing and the circuit breaker, then the model catalog. Key selection runs last, per attempt, whichever layer chose the provider.
The Bifrost AI gateway runs every routing layer once per request, before any provider call, in the order documented on the provider routing page:
- Governance resolves the virtual key and the access it grants, without picking a provider.
- Routing rules evaluate in scope order (virtual key, team, customer, global) and ascending priority within a scope. The first match wins and sets the provider, model, and fallbacks.
- Virtual key weights apply only if no rule set a provider: eligible
provider_configsare filtered by allowed models, budgets, and rate limits, then one is picked by weighted random selection. - Adaptive routing (Enterprise) keeps an already-set provider as primary, or picks the healthiest provider when none is set.
- The circuit breaker (Enterprise) redirects any target whose circuit is open to the fallback its policy names.
- The model catalog resolver fills the provider only if it is still empty; if no provider resolves, the request returns a 400.
- Key selection runs per attempt inside the chosen provider.
Rules are CEL expressions, so headers["x-tier"] == "premium" && complexity_tier == "COMPLEX" combines identity and content in one condition. A rule with chain_rule: true re-runs the rule set with its result, for example to normalize a model alias first.
The virtual key is checked again on each attempt, so a rule that targets a provider the key does not allow is refused with a 403. Routing chooses among what governance routing on virtual keys permits; it never widens it.
Latency-Based Routing and Load Balancing
Latency-based routing moves traffic toward providers and keys that answer fastest and fail least. In Bifrost Enterprise, adaptive routing makes two choices per request from live traffic: the provider for the model, and the API key within that provider.
Adaptive routing scores routes on error rate first and token-aware latency second, recomputing weights every 5 seconds:
- Named providers are kept by default. Appending fallbacks to them, or redirecting away from a failing one, are opt-in settings.
- No synthetic health checks. Health comes from real traffic; a refused key is held out and rechecked by one real request.
- Health is per node, and ranking is relative; cost decisions stay with governance.
Without Enterprise, key selection uses static weighted load balancing across API keys. Either way, it runs even when the provider was fixed upstream. The dashboard and incident views are covered in a closer look at Bifrost adaptive routing, and clustered, in-VPC deployments on the Bifrost Enterprise page.
Retries and Fallbacks When a Route Fails
A route fails in one of two places: inside a provider, where Bifrost retries or rotates keys, or across providers, where Bifrost moves to the next fallback. Routing does not run again; the chain decided before the first attempt is the one that executes.
The Bifrost platform classifies every failed attempt before choosing the next step, as described in retries and fallbacks:
| Failure class | Example | Next step |
|---|---|---|
transient |
500, 503, network error | Retry the same key with exponential backoff |
rate_limit |
429 | Rotate to another key, with backoff |
credential, quota, model_access |
401, 402, model_not_found |
Mark the key dead for this request, rotate without backoff |
caller_fault |
400 or 422 validation error | No retry |
timeout |
Provider or first-token deadline exceeded | No retry on that provider; fallbacks run unless the request's own deadline expired |
When the primary exhausts its retries, the first fallback runs with its own retry budget, and the first success wins. If every provider fails, the caller receives the primary's original error. For streaming, a routing rule can set ttft_timeout_ms so a provider that accepts a request but sends no first token is cut off in favor of the next fallback; the last attempt is never cut off. See also enabling automatic fallback when a provider fails and our review of LLM failover routing gateways.
Using LLM Observability to Debug a Routing Decision
Debugging a routing decision means answering three questions from logs: which engine chose the primary, which attempts failed and why, and which provider and key served the response. Bifrost records all three on every request log.
The built-in request logs carry a per-request routing trail, written by each layer under its own engine name:
routing-rule: which rule matched, in which scope, and the selected targetgovernanceandloadbalancing: virtual key selection and adaptive provider and key choicesmodel-catalog: the provider resolved for a bare model namecore: every retry and fallback transition
The routing_engines_used field lists which engines touched the request, attempt_trail records every attempt with its failure_class and status code, and complexity routing adds the tier, the mechanism, and any matched phrase. To debug, read routing_engines_used to find the layer that set the provider, check rule scope (a virtual-key-scoped rule outranks a global one regardless of priority number), and read attempt_trail when the serving provider differs from the primary. Tier and mechanism also reach OpenTelemetry and Prometheus (see the AI observability page).
Choosing Model Routing Signals for Your Workload
Model routing stays predictable when each goal maps to one mechanism: rules for stated intent, virtual key weights for planned splits, complexity tiers for capability matching, and adaptive routing for health. Then the routing log stays readable.
| Goal | Signal | Mechanism |
|---|---|---|
| Premium tenants to a frontier model | Header or team | Routing rule, scoped |
| 70/30 split across two providers | Static weight | Virtual key provider_configs |
| Downgrade when budget runs low | budget_used |
Rule with a cheaper target |
| Match model size to task difficulty | complexity_tier |
Complexity router plus rules |
| Avoid degraded providers | Error rate, latency | Adaptive routing (Enterprise) |
| Survive provider outages | Failure class | Retries plus a fallback chain |
Start with one carve-out, such as a single COMPLEX rule, and confirm its matches in the logs before adding tiers. The cost side is covered in how model routing cuts LLM token costs, and the full set of router types in the LLM router guide covering rule-based, semantic, and learned routing. Teams comparing implementations can use the top LLM router solutions.
Frequently Asked Questions
What is model routing?
Model routing is the selection of which model and provider handle a given LLM request, based on rules, request content, budgets, and provider health. A gateway such as Bifrost applies it per request and also defines the fallback chain, so one mechanism covers cost control and reliability.
How does model routing work?
Model routing evaluates signals in a defined order until a layer sets the provider. In Bifrost, routing rules run first, virtual key weights run if no rule matched, adaptive routing handles health in Enterprise, and the model catalog fills any remaining gap. Key selection then runs on every attempt.
What is semantic routing?
Semantic routing classifies a request by meaning and routes it to a model suited to that work. Bifrost implements it through the complexity router, which assigns a SIMPLE, MEDIUM, or COMPLEX tier using a decision model or embedding similarity. Routing rules match on complexity_tier, and an unclassified request falls through to normal routing.
Does LLM routing add latency?
Rule evaluation adds little: CEL expressions are compiled once and cached, and Bifrost's measured overhead is about 11 microseconds per request at 5,000 RPS. Classification costs more: a decision-model call typically takes 600 to 800 ms, a semantic lookup one embedding call, and both run only when a rule references the tier.
What is the difference between routing rules and load balancing?
Routing rules express intent: a condition on headers, identity, budget, or complexity selects a provider and model. Load balancing spreads traffic among eligible options by static weights or, in Bifrost Enterprise, by live error rate and latency. Rules run first, and key-level load balancing still runs after a rule fixes the provider.
What happens when every provider in the fallback chain fails?
Bifrost returns the primary provider's original error, with the primary's response headers, rather than the last fallback's error. Each provider receives its own full retry budget first, and the routing trail records every transition, so the request log shows which providers were tried and how each failed.
Getting Started with Bifrost
Predictable LLM routing depends on a defined evaluation order, explicit fallbacks, and logs that explain each decision. The open-source Bifrost gateway combines routing rules, complexity-aware semantic routing, adaptive load balancing, and a per-request routing trail inside your own infrastructure. To see how Bifrost would route your traffic, book a demo with the Bifrost team.
