LLM Gateway Routing, Fallback, and Governance in Bifrost

LLM gateway routing is the process of deciding which provider, model, and API key serve a given request, and it is the layer where cost policy, access policy, and reliability policy either agree with each other or quietly conflict. Most gateways expose routing, failover, and governance as three separate feature sets, which leaves the interaction between them undefined: a fallback can route around a budget cap, or a routing rule can send traffic to a model a team was never granted. Bifrost, the open-source AI gateway built in Go by Maxim AI, resolves all three in one pipeline with a defined precedence order. This post walks that pipeline stage by stage: how a provider is selected, what happens when it fails, how governance bounds both, and how to configure the result.

How does an LLM gateway process a request?

An LLM gateway receives an OpenAI-compatible request, authenticates it, resolves it to a specific provider, model, and API key, executes the call with retry and fallback handling, and returns the response while recording usage against the caller's budget. Each stage can be configured independently, but they execute in a fixed order.

In Bifrost, the request pipeline runs in these stages:

1. Transport and validation. The request is parsed and converted to an internal schema.
2. Authentication and governance resolution. The virtual key on the request determines the caller's permissions, budgets, and limits.
3. Routing decision. Routing rules, then governance provider configuration, then adaptive load balancing determine the provider and model.
4. Key selection. A specific API key is chosen from the eligible pool for that provider.
5. Execution with retries and fallbacks. The call runs, with failures classified and handled before any provider switch.
6. Response and accounting. Usage is recorded against the virtual key, team, and customer hierarchy.

Because every stage sits behind one OpenAI-compatible endpoint, adopting the pipeline is a base URL change in an existing SDK rather than an application rewrite.

How does Bifrost decide which provider serves a request?

Bifrost resolves provider selection through three mechanisms that apply in a defined precedence order rather than competing with each other.

Routing rules run first. Routing rules evaluate CEL expressions against request context at runtime, and they execute before governance provider selection and can override it. Rules are organized by scope with first-match-wins evaluation:

• Virtual key scope (highest priority)
• Team scope
• Customer scope
• Global scope (applies to all traffic)

Within a scope, rules are sorted by ascending priority, so a rule with priority 0 evaluates before priority 10. When a rule sets chain_rule, the resolved provider and model become the new context and the full scope chain is re-evaluated from the top. If no rule matches anywhere in the chain, the incoming provider and model are used unchanged. Expressions can read the requested model, the current provider, the request type, headers, and query parameters, which is what makes rules like "route premium-tier traffic to a different provider" a configuration change rather than application logic.

Governance routing runs next. Provider routing uses the provider configurations attached to a virtual key, each with a weight and an allowed-model list, to distribute traffic explicitly. This is the mechanism to use when routing is driven by compliance or a specific cost strategy, because the distribution is stated rather than inferred.

Adaptive load balancing runs last, and only where governance has not already made the decision. Adaptive load balancing in the enterprise edition adjusts weights from live error-rate, latency, and throughput metrics, applies circuit breaking to failing routes, and shares rate-limit signals across nodes so an overloaded key backs off fleet-wide. Weight recomputation runs asynchronously every five seconds, so request routing reads pre-computed weights and adds under 10 microseconds to the hot path.

All three mechanisms read from the Model Catalog, the registry that tracks which models are available from which providers. The catalog combines pricing data refreshed on a periodic schedule with each provider's list-models endpoint, so newly released models are usable before pricing data catches up.

What happens when a provider fails?

Provider failure handling in Bifrost operates as two layers, and the distinction matters because switching providers is usually the wrong first move. Retries and fallbacks work as follows:

• Retries handle failures within a provider. Bifrost classifies each failure as a per-key problem (401, 402, 403, 429) or a transient upstream problem (5xx, network, DNS).
• Per-key failures rotate keys. Permanent credential failures mark the key dead for that request and rotate immediately, since waiting cannot revive a bad credential. Rate-limit failures rotate but still apply backoff, because provider quotas are frequently shared across keys on the same account.
• Transient failures reuse the key with exponential backoff and jitter, calculated as the initial backoff doubled per attempt, capped at the configured maximum, and multiplied by a jitter factor.
• Fallbacks handle provider failure. Only when a provider's retry budget is exhausted does the request move to the next provider in the chain, and each fallback provider receives its own full retry budget.

Underneath both layers, LLM gateway routing continues at the key level: weighted key load balancing distributes requests across the key pool using weighted random selection, with per-key model allowlists and denylists that keep expensive models off general-purpose keys.

This design matters because provider rate limits are the most common production failure. OpenAI and Anthropic enforce request and token limits at the organization level, so a 429 is often a capacity problem on one credential rather than an outage. Rotating keys resolves it without the latency and cost profile change that a provider switch introduces.

How does Bifrost govern access, budgets, and rate limits?

Virtual keys are the primary governance entity. A caller authenticates with a virtual key, and that key carries the permissions, budget, and limits that apply to the request. Bifrost accepts the key through the header conventions of the major provider SDKs, including Authorization, x-api-key, x-goog-api-key, and api-key, so existing clients authenticate without modification.

Each virtual key carries:

• Access control: provider and model filtering, plus optional restriction to specific provider API keys.
• Budgets: an independent spend limit with a reset duration from one minute to one year, optionally aligned to calendar boundaries in UTC rather than a rolling window.
• Rate limits: token-based and request-based throttling with their own reset periods.
• Attachment: exclusive assignment to one team or one customer, or neither.
• Lifecycle: an optional expiry and an active/inactive status that revokes access immediately.

Budgets and limits are hierarchical. A customer budget contains team budgets, which contain virtual key budgets, which contain per-provider budgets, and every level is checked cumulatively. A team cannot exceed its own cap by spreading spend across its keys, and a customer cannot exceed its cap by adding teams. This is the structure that makes gateway-level governance enforceable rather than advisory.

How do routing, fallback, and governance interact?

LLM gateway routing, fallback, and governance compose in one direction: the governance layer defines the set of providers, models, and keys a request may use; routing selects within that set; fallback stays inside it. A fallback chain cannot reach a provider the virtual key does not permit, and a routing rule cannot grant access that governance withholds.

The practical consequences are worth stating explicitly:

• Routing rules override provider selection, not permissions. A rule can send a request to a different provider, but the request still spends against the same budget hierarchy.
• Governance takes precedence over adaptive load balancing. When a virtual key states provider weights, those weights are used, because an explicit configuration is treated as an intentional decision.
• Fallback does not bypass budgets. A failover to a more expensive provider still records usage against the same caps, which is why budget alerts and fallback chains should be designed together.
• Rate limits apply at the virtual key level. Throttling a noisy workload does not require throttling the team it belongs to.

A virtual key that encodes all three layers looks like this:

{
  "name": "Engineering Team API",
  "provider_configs": [
    { "provider": "openai", "weight": 0.5, "allowed_models": ["gpt-4o-mini"] },
    { "provider": "anthropic", "weight": 0.5, "allowed_models": ["claude-3-sonnet-20240229"] }
  ],
  "team_id": "team-eng-001",
  "budget": { "max_limit": 100.00, "reset_duration": "1M" }
}

The provider configs define both the routing distribution and the permission boundary. The team attachment places the key inside a larger budget hierarchy. Any fallback triggered by a provider failure operates within the two providers listed, because that list is the permission set.

Routing, fallback, and governance FAQs

Does a routing rule override a virtual key's model restrictions?

No. Routing rules override provider selection, and they execute before governance provider selection, but the virtual key's access control still bounds what the request can reach. Routing decides where inside the permitted set a request goes.

When should governance routing be used instead of adaptive load balancing?

Use governance routing when the distribution is driven by a compliance requirement, a contractual commitment, or a specific cost strategy that should not shift automatically. Use adaptive load balancing when the goal is performance optimization with minimal configuration.

Does the routing and governance pipeline add measurable latency?

Bifrost adds 11 microseconds of overhead per request at 5,000 requests per second in sustained benchmarks, with route selection accounting for under 10 microseconds of that. Both are immaterial against model inference time.

How are budgets enforced across multiple gateway instances?

Budget and rate-limit state is held in memory, so a single instance enforces caps directly. Multi-node deployments require clustering, which synchronizes governance state across nodes in real time so a cap means the same thing on every instance.

Getting started with LLM gateway routing in Bifrost

LLM gateway routing, provider fallback, and governance are most useful when they are one system rather than three. Bifrost resolves them in a single pipeline: routing rules first, governance provider configuration next, adaptive load balancing where no explicit decision exists, retries before fallbacks, and every path bounded by the permissions and budgets on the virtual key. Bifrost Enterprise extends this with clustering, adaptive routing, and in-VPC or air-gapped deployment for teams whose data cannot transit a hosted control plane.

To see how routing, fallback, and governance would map to your own provider mix and team structure, book a demo with the Bifrost team.