LLM Gateway Routing, Fallback, and Governance in Bifrost

TL;DR

• Bifrost resolves LLM gateway routing in a fixed order: routing rules first, then the virtual key's provider configuration, then adaptive load balancing, which picks a provider only where no explicit decision exists but always selects the API key.
• Retries handle failures within a provider (key rotation for 401, 402, 403, and 429; exponential backoff for 5xx), and fallbacks move to the next provider only after that provider's retry budget is exhausted.
• A virtual key's provider list is a hard boundary: neither routing rules nor fallback chains can send a request to a provider the key does not permit.
• Budgets are checked cumulatively across customer, team, virtual key, and provider configuration, so a fallback to a pricier provider still spends against the same caps.
• Bifrost adds 11 microseconds of overhead per request at 5,000 RPS, and adaptive load balancing (Enterprise) adds under 10 microseconds for route selection.

Bifrost resolves LLM gateway routing, provider fallback, and governance in one request path. Here is the order of operations and how to configure each layer.

LLM gateway routing is the process of deciding which provider, model, and API key serve a given request, and it is the layer where cost policy, access policy, and reliability policy either agree with each other or quietly conflict. When routing, failover, and governance are configured as separate feature sets, the interaction between them is undefined: a fallback can route around a budget cap, or a routing rule can send traffic to a model a team was never granted. Bifrost, the open-source AI gateway built in Go by Maxim AI, resolves all three in one pipeline with a defined precedence order. This post walks that pipeline stage by stage: how a provider is selected, what happens when it fails, how governance bounds both, and how to configure the result.

How does an LLM gateway process a request?

An LLM gateway receives an OpenAI-compatible request, authenticates it, resolves it to a specific provider, model, and API key, executes the call with retry and fallback handling, and returns the response while recording usage against the caller's budget. Each stage can be configured independently, but they execute in a fixed order.

In Bifrost, the request pipeline runs in these stages:

1. Transport and validation. The request is parsed and converted to an internal schema.
2. Authentication and governance resolution. The virtual key on the request determines the caller's permissions, budgets, and limits.
3. Routing decision. Routing rules, then governance provider configuration, then adaptive load balancing determine the provider and model.
4. Key selection. A specific API key is chosen from the eligible pool for that provider.
5. Execution with retries and fallbacks. The call runs, with failures classified and handled before any provider switch.
6. Response and accounting. Usage is recorded against the virtual key, team, and customer hierarchy.

Because every stage sits behind one OpenAI-compatible endpoint, adopting the pipeline is a base URL change in an existing SDK rather than an application rewrite. The broader role of this layer is covered in what an LLM gateway actually does on every request.

Figure 1: Routing, failover, and governance are stages of one pipeline, so their order is fixed rather than negotiated per request.

How does Bifrost decide which provider serves a request?

Bifrost resolves provider selection through three mechanisms that apply in a defined precedence order rather than competing with each other: routing rules, then governance routing on the virtual key, then adaptive load balancing, plus the Model Catalog resolver as a final default. Each later mechanism acts only where an earlier one has not already decided, which keeps LLM routing predictable when all three are configured.

This layering is what separates gateway-level routing from a standalone LLM router that picks a model per request.

Figure 2: Later mechanisms only fill in what earlier ones left undecided, and the allowlist check runs last no matter which one decided.

Routing rules run first. Routing rules evaluate CEL expressions against request context at runtime, and they execute before governance provider selection and can override it. Rules are organized by scope with first-match-wins evaluation:

• Virtual key scope (highest priority)
• Team scope
• Customer scope
• Global scope (applies to all traffic)

Within a scope, rules are sorted by ascending priority, so a rule with priority 0 evaluates before priority 10. When a rule sets chain_rule, the resolved provider and model become the new context and the full scope chain is re-evaluated from the top. If no rule matches anywhere in the chain, the incoming provider and model are used unchanged. Expressions can read the requested model, the current provider, the request type, headers, and query parameters, which is what makes rules like "route premium-tier traffic to a different provider" a configuration change rather than application logic.

Governance routing runs next. Provider routing uses the provider configurations attached to a virtual key, each with a weight and an allowed-model list, to distribute traffic explicitly. This is the mechanism to use when routing is driven by compliance or a specific cost strategy, because the distribution is stated rather than inferred.

Adaptive load balancing runs next. It chooses a provider only where governance has not already pinned one, but it still selects the best API key for whichever provider was chosen. Adaptive load balancing in the enterprise edition adjusts weights from live error-rate, latency, and throughput metrics, applies circuit breaking to failing routes, and shares rate-limit backoff signals with other nodes in the same region. Weight recomputation runs asynchronously every five seconds, so request routing reads pre-computed weights and adds under 10 microseconds to the hot path. Common weighting and cost strategies are compared in LLM routing strategies every AI gateway needs.

Governance routing and adaptive load balancing both read from the Model Catalog, the registry that tracks which models are available from which providers. The catalog combines pricing data refreshed on a periodic schedule with each provider's list-models endpoint, so newly released models are usable before pricing data catches up.

What happens when a provider fails?

Provider failure handling in Bifrost operates as two layers, and the distinction matters because switching providers is usually the wrong first move. Retries and fallbacks work as follows:

• Retries handle failures within a provider. Bifrost classifies each failure as a per-key problem (401, 402, 403, 429) or a transient upstream problem (5xx, network, DNS).
• Per-key failures rotate keys. Permanent credential failures mark the key dead for that request and rotate immediately, since waiting cannot revive a bad credential. Rate-limit failures rotate but still apply backoff, because provider quotas are frequently shared across keys on the same account.
• Transient failures reuse the key with exponential backoff and jitter, calculated as the initial backoff doubled per attempt, capped at the configured maximum, and multiplied by a jitter factor.
• Fallbacks handle provider failure. Only when a provider's retry budget is exhausted does the request move to the next provider in the chain, and each fallback provider receives its own full retry budget.

Retries are disabled by default (max_retries: 0) and are enabled per provider, with a default exponential backoff of 500 ms initial and 5,000 ms maximum. If every fallback also fails, Bifrost returns the original error from the primary provider, and if every key on a provider is permanently dead, it returns 502 upstream_credentials_exhausted.

Figure 3: A rate-limited key is rotated before any provider switch, so a 429 does not change the model or price serving the request.

Underneath both layers, LLM gateway routing continues at the key level: weighted key load balancing distributes requests across the key pool using weighted random selection, with per-key model allowlists and denylists that keep expensive models off general-purpose keys.

This design matters because provider rate limits are the most common production failure. OpenAI and Anthropic enforce request and token limits at the organization level, so a 429 is often a capacity problem on one credential rather than an outage. Rotating to a key on a separate organization or project resolves many 429s without the cost and model change a provider switch introduces. Gateway options for this pattern are compared in LLM failover routing gateways.

How does Bifrost govern access, budgets, and rate limits?

Virtual keys are the primary governance entity. A caller authenticates with a virtual key, and that key carries the permissions, budget, and limits that apply to the request. Bifrost accepts the key through the header conventions of the major provider SDKs, including Authorization, x-api-key, x-goog-api-key, and api-key, so existing clients authenticate without modification. Rollout patterns for large engineering organizations are covered in how platform teams govern LLM access with virtual keys.

Each virtual key carries:

• Access control: provider and model filtering, plus optional restriction to specific provider API keys.
• Budgets: an independent spend limit with a reset duration from one minute to one year, optionally aligned to calendar boundaries in UTC rather than a rolling window.
• Rate limits: token-based and request-based throttling with their own reset periods.
• Attachment: exclusive assignment to one team or one customer, or neither.
• Lifecycle: an optional expiry and an active/inactive status that revokes access immediately.

Budgets and limits are hierarchical. A customer budget contains team budgets, which contain virtual key budgets, which contain per-provider budgets, and every level is checked cumulatively. A team cannot exceed its own cap by spreading spend across its keys, and a customer cannot exceed its cap by adding teams. This is the structure that makes gateway-level governance enforceable rather than advisory.

Worked examples of this hierarchy appear in LLM budget management with virtual keys, and throttling patterns in LLM API rate limiting with virtual keys and budgets.

How do routing, fallback, and governance interact?

LLM gateway routing, fallback, and governance compose in one direction: the governance layer defines the set of providers, models, and keys a request may use; routing selects within that set; fallback stays inside it. A fallback chain cannot reach a provider the virtual key does not permit, and a routing rule cannot grant access that governance withholds. Bifrost enforces this in core after every routing plugin has run: a final provider outside the key's allowlist is rejected (HTTP 403 from governance, or HTTP 400 from core enforcement), and fallbacks that target disallowed providers are filtered out.

Figure 4: The provider list on the virtual key is both the routing distribution and the permission boundary.

The practical consequences are worth stating explicitly:

• Routing rules override provider selection, not permissions. A rule can send a request to a different provider, but the request still spends against the same budget hierarchy.
• Governance takes precedence over adaptive load balancing. When a virtual key states provider weights, those weights are used, because an explicit configuration is treated as an intentional decision.
• Fallback does not bypass budgets. A failover to a more expensive provider still records usage against the same caps, which is why budget alerts and fallback chains should be designed together.
• Rate limits apply at the virtual key and provider configuration levels. Teams and customers carry budgets but no rate limits, so throttling a noisy workload does not require throttling the team it belongs to.

A virtual key that encodes all three layers looks like this:

{
  "name": "Engineering Team API",
  "provider_configs": [
    { "provider": "openai", "weight": 0.5, "allowed_models": ["gpt-4o-mini"] },
    { "provider": "anthropic", "weight": 0.5, "allowed_models": ["claude-sonnet-4-6"] }
  ],
  "team_id": "team-eng-001",
  "budgets": [{ "max_limit": 100.00, "reset_duration": "1M" }]
}

The provider configs define both the routing distribution and the permission boundary. The team attachment places the key inside a larger budget hierarchy. Any fallback triggered by a provider failure operates within the two providers listed, because that list is the permission set.

Routing, fallback, and governance FAQs

Does a routing rule override a virtual key's model restrictions?

No. Routing rules override provider selection, and they execute before governance provider selection, but the virtual key's access control still bounds what the request can reach. Routing decides where inside the permitted set a request goes.

When should governance routing be used instead of adaptive load balancing?

Use governance routing when the distribution is driven by a compliance requirement, a contractual commitment, or a specific cost strategy that should not shift automatically. Use adaptive load balancing when the goal is performance optimization with minimal configuration.

Does the routing and governance pipeline add measurable latency?

Bifrost adds 11 microseconds of overhead per request at 5,000 requests per second in sustained benchmarks, and adaptive load balancing (Enterprise) adds under 10 microseconds for route selection. Both are immaterial against model inference time. Adaptive load balancing recomputes its weights asynchronously every five seconds, so no weight calculation happens on the request path.

How are budgets enforced across multiple gateway instances?

Budget and rate-limit state is held in memory, so a single instance enforces caps directly. Multi-node deployments require clustering, which synchronizes governance state across nodes in real time so a cap means the same thing on every instance.

What is the difference between an LLM router and an AI gateway?

An LLM router chooses which model or provider should answer a request, usually by cost, quality, or latency. An AI gateway such as Bifrost includes routing as one stage, then adds authentication, budgets, rate limits, retries, fallbacks, guardrails, and logging around it. In Bifrost, routing decisions are always bounded by the permissions and budgets on the caller's virtual key, as the LLM router explainer describes from the routing side.

What happens when every provider in a fallback chain fails?

Bifrost returns the original error from the primary provider once every fallback has used its own retry budget. If a plugin on a fallback provider marks its error as non-fallbackable, for example a compliance plugin, the chain stops immediately and that error is returned. When every configured key on a provider is permanently rejected, the response is 502 upstream_credentials_exhausted.

Getting started with LLM gateway routing in Bifrost

LLM gateway routing, provider fallback, and governance are most useful when they are one system rather than three. Bifrost resolves them in a single pipeline: routing rules first, governance provider configuration next, adaptive load balancing where no explicit decision exists, retries before fallbacks, and every path bounded by the permissions and budgets on the virtual key. Bifrost Enterprise extends this with clustering, adaptive routing, and in-VPC or air-gapped deployment for teams whose data cannot transit a hosted control plane. Teams still choosing a gateway can start with the complete LLM gateway guide for enterprise AI.

To see how routing, fallback, and governance would map to your own provider mix and team structure, book a demo with the Bifrost team.