---
description: "LLM routing as a decision pipeline: which signals a router reads, the order rules, weights, and health apply, and how fallbacks and routing logs work."
title: "LLM Routing: How Routing Decisions Get Made"
image: https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/size/w1200/2026/10/llm-routing-how-routing-decisions-get-made-bifrost-1920x1080-1.png
---

Try Bifrost Enterprise free for 14 days. [Request access](https://www.getmaxim.ai/#enterprise-trial)

**TL;DR**

- LLM routing produces three outputs per request: a primary provider and model, an ordered fallback chain, and the API key used for each attempt.
- Routing signals fall into four families: request content, caller identity, capacity and cost, and live provider health.
- Bifrost evaluates explicit intent before health: routing rules first, then virtual key weights, then Enterprise adaptive routing and the circuit breaker, then the model catalog as the last resort.
- Retries happen inside one provider and fallbacks happen across providers; each provider in the chain gets its own full retry budget.
- Bifrost writes every routing decision to a per-request routing log, so a misrouted request can be traced to the rule, weight, or health signal that chose it.

LLM routing is the decision, made for each request, about which provider, model, and API key will serve an LLM call and what happens if that choice fails. [Bifrost](https://www.getmaxim.ai/), the [open-source AI gateway built by Maxim AI](https://github.com/maximhq/bifrost), makes that decision in a fixed, documented order, and that order matters more than any single routing feature. This post covers the signals a router evaluates, how precedence resolves conflicts, how failures are handled, and how to read a routing decision afterward.

## What Is LLM Routing?

LLM routing is the process of selecting a provider, model, and credential for each LLM request based on rules, request content, budgets, and provider health, then defining what to try next if that selection fails. The component that performs it is an LLM router, usually running inside an LLM gateway.

For router types and the router-versus-gateway distinction, see our [full explainer on what an LLM router is and how model routing works](https://www.getmaxim.ai/articles/what-is-an-llm-router-how-model-routing-works/). This post covers what happens between a request arriving and a provider receiving it. Because signals conflict, a router is only predictable when the precedence between them is defined.

Bifrost runs this logic as its [LLM gateway](https://www.getmaxim.ai/llm-gateway) layer, in front of [10,000+ models across 25+ providers](https://docs.getbifrost.ai/providers/supported-providers/overview) behind one OpenAI-compatible API.

## The Signals an LLM Router Evaluates

An LLM router evaluates four families of signals: what the request asks for, who is asking, how much budget and rate-limit capacity remains, and how healthy each provider and key is right now. Configuration covers the first three; only the fourth requires live measurement of traffic.

![](https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/2026/10/image-38.png)

***Figure 1: A router reads what is asked, who is asking, what budget remains, and what is healthy, then emits more than a model name.***

| Signal family | Example inputs | Used for | Where it lives in Bifrost |
| --- | --- | --- | --- |
| Request content | Model, request type, headers, complexity tier | Matching work to model capability | [Routing rules](https://docs.getbifrost.ai/providers/routing-rules) and the complexity router |
| Caller identity | Virtual key, team, customer | Per-tenant policy, compliance | Rule scopes, virtual key provider configs |
| Capacity and cost | Budget and rate-limit usage | Shifting traffic before a limit is hit | `budget_used`, `tokens_used`, `request` variables |
| Live health | Error rate, latency, refused keys | Avoiding degraded providers and keys | Adaptive routing (Enterprise) |

In Bifrost, `budget_used` is a percentage of the matching [budget or rate limit](https://docs.getbifrost.ai/features/governance/budget-and-limits); when several apply the highest wins, and with none configured it is 0, so `budget_used > 80` never fires. Per-request model choice is also covered in [smart LLM routing and picking the optimal model per request](https://www.getmaxim.ai/articles/smart-llm-routing-picking-the-optimal-model-per-request/).

## Semantic Routing and Complexity Tiers

Semantic routing classifies a request by meaning and sends it to a model sized for that work. In Bifrost, classification produces a `complexity_tier` of SIMPLE, MEDIUM, or COMPLEX, and that tier is a condition inside a routing rule, so it follows the same precedence as any other rule.

The [complexity router](https://docs.getbifrost.ai/features/complexity-router) offers two classifiers:

- **Decision model:** reads the latest user request and picks the tier whose definition fits. Typesafe Jev is the default, typically answering in 600 to 800 ms.
- **Semantic:** embeds the request and takes the tier of its nearest reference phrase, from 150 shipped defaults (up to 750).

![](https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/2026/10/image-39.png)

***Figure 2: Classification is a condition inside the rule engine, so a missing tier never blocks the request; it only skips the complexity rule.***

Classification runs only when a rule references `complexity_tier`. If the classifier times out or finds no confident match, evaluation falls through to the next rule or to virtual key weights. With session-aware routing on, a session's tier only moves up, so agent conversations do not switch models mid-task. Accuracy and latency figures are in the [benchmarks for the upgraded complexity router](https://www.getmaxim.ai/bifrost/blog/benchmarking-bifrosts-upgraded-complexity-router).

Learned routing is the research form of this idea: the [RouteLLM paper from UC Berkeley and Anyscale researchers](https://arxiv.org/abs/2406.18665) trained routers on human preference data and reported over 2x cost reduction without sacrificing response quality. [NVIDIA's LLM Router blueprint](https://developer.nvidia.com/blog/deploying-the-nvidia-ai-blueprint-for-cost-efficient-llm-routing/) routes by task type or complexity category. See also our roundup of [semantic routing platforms for LLM applications](https://www.getmaxim.ai/articles/top-5-semantic-routing-platforms-for-llm-applications/).

## Routing Rules and the Order of Precedence

Routing precedence decides which signal wins when signals disagree. In Bifrost, explicit intent is evaluated before measured health: routing rules, then virtual key weights, then Enterprise adaptive routing and the circuit breaker, then the model catalog. Key selection runs last, per attempt, whichever layer chose the provider.

The [Bifrost AI gateway](https://www.getmaxim.ai/) runs every routing layer once per request, before any provider call, in the order documented on the [provider routing](https://docs.getbifrost.ai/providers/provider-routing) page:

1. **Governance resolves the virtual key** and the access it grants, without picking a provider.
2. **Routing rules evaluate** in scope order (virtual key, team, customer, global) and ascending priority within a scope. The first match wins and sets the provider, model, and fallbacks.
3. **Virtual key weights apply** only if no rule set a provider: eligible `provider_configs` are filtered by allowed models, budgets, and rate limits, then one is picked by weighted random selection.
4. **Adaptive routing (Enterprise)** keeps an already-set provider as primary, or picks the healthiest provider when none is set.
5. **The circuit breaker (Enterprise)** redirects any target whose circuit is open to the fallback its policy names.
6. **The model catalog resolver** fills the provider only if it is still empty; if no provider resolves, the request returns a 400.
7. **Key selection** runs per attempt inside the chosen provider.

![](https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/2026/10/image-40.png)

***Figure 3: Explicit intent is evaluated first and health-based choices second, and key selection still runs after the provider is fixed.***

Rules are [CEL expressions](https://cel.dev/), so `headers["x-tier"] == "premium" && complexity_tier == "COMPLEX"` combines identity and content in one condition. A rule with `chain_rule: true` re-runs the rule set with its result, for example to normalize a model alias first.

The virtual key is checked again on each attempt, so a rule that targets a provider the key does not allow is refused with a `403`. Routing chooses among what [governance routing on virtual keys](https://docs.getbifrost.ai/features/governance/routing) permits; it never widens it.

## Latency-Based Routing and Load Balancing

Latency-based routing moves traffic toward providers and keys that answer fastest and fail least. In Bifrost Enterprise, adaptive routing makes two choices per request from live traffic: the provider for the model, and the API key within that provider.

[Adaptive routing](https://docs.getbifrost.ai/enterprise/adaptive-routing/overview) scores routes on error rate first and token-aware latency second, recomputing weights every 5 seconds:

- **Named providers are kept by default.** Appending fallbacks to them, or redirecting away from a failing one, are opt-in settings.
- **No synthetic health checks.** Health comes from real traffic; a refused key is held out and rechecked by one real request.
- **Health is per node,** and ranking is relative; cost decisions stay with governance.

Without Enterprise, key selection uses static [weighted load balancing across API keys](https://docs.getbifrost.ai/features/keys-management). Either way, it runs even when the provider was fixed upstream. The dashboard and incident views are covered in [a closer look at Bifrost adaptive routing](https://www.getmaxim.ai/bifrost/blog/a-closer-look-at-bifrost-adaptive-routing-how-it-works-and-what-you-can-now-see), and clustered, in-VPC deployments on the [Bifrost Enterprise page](https://www.getmaxim.ai/enterprise).

## Retries and Fallbacks When a Route Fails

A route fails in one of two places: inside a provider, where Bifrost retries or rotates keys, or across providers, where Bifrost moves to the next fallback. Routing does not run again; the chain decided before the first attempt is the one that executes.

The [Bifrost platform](https://www.getmaxim.ai/) classifies every failed attempt before choosing the next step, as described in [retries and fallbacks](https://docs.getbifrost.ai/features/retries-and-fallbacks):

| Failure class | Example | Next step |
| --- | --- | --- |
| `transient` | 500, 503, network error | Retry the same key with exponential backoff |
| `rate_limit` | 429 | Rotate to another key, with backoff |
| `credential`, `quota`, `model_access` | 401, 402, `model_not_found` | Mark the key dead for this request, rotate without backoff |
| `caller_fault` | 400 or 422 validation error | No retry |
| `timeout` | Provider or first-token deadline exceeded | No retry on that provider; fallbacks run unless the request's own deadline expired |

![](https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/2026/10/image-41.png)

***Figure 4: Each provider in the chain gets a full retry budget, so a chain of three providers with three retries each can make up to twelve attempts.***

When the primary exhausts its retries, the first fallback runs with its own retry budget, and the first success wins. If every provider fails, the caller receives the primary's original error. For streaming, a routing rule can set `ttft_timeout_ms` so a provider that accepts a request but sends no first token is cut off in favor of the next fallback; the last attempt is never cut off. See also [enabling automatic fallback when a provider fails](https://www.getmaxim.ai/bifrost/blog/your-primary-llm-provider-failed-enable-automatic-fallback-with-bifrost) and our review of [LLM failover routing gateways](https://www.getmaxim.ai/articles/top-5-llm-failover-routing-gateways-in-2026/).

## Using LLM Observability to Debug a Routing Decision

Debugging a routing decision means answering three questions from logs: which engine chose the primary, which attempts failed and why, and which provider and key served the response. Bifrost records all three on every request log.

The [built-in request logs](https://docs.getbifrost.ai/features/observability/default) carry a per-request routing trail, written by each layer under its own engine name:

- **`routing-rule`:** which rule matched, in which scope, and the selected target
- **`governance` and `loadbalancing`:** virtual key selection and adaptive provider and key choices
- **`model-catalog`:** the provider resolved for a bare model name
- **`core`:** every retry and fallback transition

The `routing_engines_used` field lists which engines touched the request, `attempt_trail` records every attempt with its `failure_class` and status code, and complexity routing adds the tier, the mechanism, and any matched phrase. To debug, read `routing_engines_used` to find the layer that set the provider, check rule scope (a virtual-key-scoped rule outranks a global one regardless of priority number), and read `attempt_trail` when the serving provider differs from the primary. Tier and mechanism also reach OpenTelemetry and Prometheus (see the [AI observability page](https://www.getmaxim.ai/ai-observability)).

## Choosing Model Routing Signals for Your Workload

Model routing stays predictable when each goal maps to one mechanism: rules for stated intent, virtual key weights for planned splits, complexity tiers for capability matching, and adaptive routing for health. Then the routing log stays readable.

| Goal | Signal | Mechanism |
| --- | --- | --- |
| Premium tenants to a frontier model | Header or team | Routing rule, scoped |
| 70/30 split across two providers | Static weight | Virtual key `provider_configs` |
| Downgrade when budget runs low | `budget_used` | Rule with a cheaper target |
| Match model size to task difficulty | `complexity_tier` | Complexity router plus rules |
| Avoid degraded providers | Error rate, latency | Adaptive routing (Enterprise) |
| Survive provider outages | Failure class | Retries plus a fallback chain |

Start with one carve-out, such as a single COMPLEX rule, and confirm its matches in the logs before adding tiers. The cost side is covered in [how model routing cuts LLM token costs](https://www.getmaxim.ai/articles/model-routing-how-to-cut-llm-token-costs/), and the full set of router types in the [LLM router guide covering rule-based, semantic, and learned routing](https://www.getmaxim.ai/articles/what-is-an-llm-router-how-model-routing-works/). Teams comparing implementations can use the [top LLM router solutions](https://www.getmaxim.ai/articles/top-5-llm-router-solutions-in-2026/).

## Frequently Asked Questions

### What is model routing?

Model routing is the selection of which model and provider handle a given LLM request, based on rules, request content, budgets, and provider health. A gateway such as Bifrost applies it per request and also defines the fallback chain, so one mechanism covers cost control and reliability.

### How does model routing work?

Model routing evaluates signals in a defined order until a layer sets the provider. In Bifrost, routing rules run first, virtual key weights run if no rule matched, adaptive routing handles health in Enterprise, and the model catalog fills any remaining gap. Key selection then runs on every attempt.

### What is semantic routing?

Semantic routing classifies a request by meaning and routes it to a model suited to that work. Bifrost implements it through the complexity router, which assigns a SIMPLE, MEDIUM, or COMPLEX tier using a decision model or embedding similarity. Routing rules match on `complexity_tier`, and an unclassified request falls through to normal routing.

### Does LLM routing add latency?

Rule evaluation adds little: CEL expressions are compiled once and cached, and Bifrost's measured overhead is about [11 microseconds per request at 5,000 RPS](https://www.getmaxim.ai/resources/benchmarks). Classification costs more: a decision-model call typically takes 600 to 800 ms, a semantic lookup one embedding call, and both run only when a rule references the tier.

### What is the difference between routing rules and load balancing?

Routing rules express intent: a condition on headers, identity, budget, or complexity selects a provider and model. Load balancing spreads traffic among eligible options by static weights or, in Bifrost Enterprise, by live error rate and latency. Rules run first, and key-level load balancing still runs after a rule fixes the provider.

### What happens when every provider in the fallback chain fails?

Bifrost returns the primary provider's original error, with the primary's response headers, rather than the last fallback's error. Each provider receives its own full retry budget first, and the routing trail records every transition, so the request log shows which providers were tried and how each failed.

## Getting Started with Bifrost

Predictable LLM routing depends on a defined evaluation order, explicit fallbacks, and logs that explain each decision. The [open-source Bifrost gateway](https://www.getmaxim.ai/) combines routing rules, complexity-aware semantic routing, adaptive load balancing, and a per-request routing trail inside your own infrastructure. To see how Bifrost would route your traffic, [book a demo](https://www.getmaxim.ai/book-a-demo) with the Bifrost team.

## Read next

[![How to Deploy an LLM Stack in an Air-Gapped Environment](https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/size/w720/2026/10/how-to-deploy-an-llm-stack-in-an-air-gapped-environment-bifrost-1920x1080.png) An air-gapped environment has no automated connection to the internet, so every model, image, and update arrives by controlled transfer. This guide walks through each layer of an offline LLM stack, from model serving with vLLM and SGLang to gateway policy, guardrails, identity, and audit logs.](https://www.getmaxim.ai/articles/llm-stack-air-gapped-environment/)

[![GPT Realtime API in Production: Routing, Governing and Billing Voice Sessions](https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/size/w720/2026/10/gpt-realtime-api-in-production-routing-governing-and-billing-bifrost-1920x1080.png) GPT Realtime is OpenAI's speech-to-speech model family on the Realtime API, with GPT-Live as its duration-billed sibling. This guide covers connections, client secrets and billing, then shows how Bifrost routes, governs and prices every voice session.](https://www.getmaxim.ai/articles/gpt-realtime-api-production/)

[![API Key Management for LLM Apps: How to Manage Client Access](https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/size/w720/2026/10/how-to-enforce-ai-budgets-per-team-project-and-key-bifrost-1920x1080--1-.png) API key management for LLM applications means issuing every app, service, and environment its own revocable credential instead of sharing raw provider keys. This guide shows how to do it with Bifrost virtual keys, covering scoping, rate limits, rotation, revocation, and log attribution.](https://www.getmaxim.ai/articles/api-key-management-for-llm-applications/)

## More in AI Gateway

[![How to Deploy an LLM Stack in an Air-Gapped Environment](https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/size/w720/2026/10/how-to-deploy-an-llm-stack-in-an-air-gapped-environment-bifrost-1920x1080.png) An air-gapped environment has no automated connection to the internet, so every model, image, and update arrives by controlled transfer. This guide walks through each layer of an offline LLM stack, from model serving with vLLM and SGLang to gateway policy, guardrails, identity, and audit logs.](https://www.getmaxim.ai/articles/llm-stack-air-gapped-environment/)

```json
{
    "@context": "https://schema.org",
    "@type": "Article",
    "publisher": {
        "@type": "Organization",
        "name": "Maxim Articles",
        "url": "https://www.getmaxim.ai/articles/",
        "logo": {
            "@type": "ImageObject",
            "url": "https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/size/w256h256/2025/08/thumbnail.png",
            "width": 60,
            "height": 60
        }
    },
    "author": {
        "@type": "Person",
        "name": "Kamya Shah",
        "image": {
            "@type": "ImageObject",
            "url": "https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/2025/09/WhatsApp-Image-2025-08-29-at-17.40.40-1.jpeg",
            "width": 1200,
            "height": 1600
        },
        "url": "https://www.getmaxim.ai/articles/author/kamya/",
        "sameAs": []
    },
    "headline": "LLM Routing: How Routing Decisions Get Made",
    "url": "https://www.getmaxim.ai/articles/llm-routing-decisions/",
    "datePublished": "2026-10-30T18:58:38.000Z",
    "dateModified": "2026-10-30T18:58:38.000Z",
    "image": {
        "@type": "ImageObject",
        "url": "https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/size/w1200/2026/10/llm-routing-how-routing-decisions-get-made-bifrost-1920x1080-1.png",
        "width": 1200,
        "height": 675
    },
    "keywords": "AI Gateway",
    "description": "LLM routing is the per-request choice of which provider, model, and API key serve an LLM call. This guide walks through the signals a router evaluates, the precedence order Bifrost applies, what happens when a route fails, and how to debug a routing decision from its logs.\n",
    "mainEntityOfPage": "https://www.getmaxim.ai/articles/llm-routing-decisions/"
}
```