AI Gateway

How an AI Gateway Tackles LLM Rate Limits and Outages

How an AI Gateway Tackles LLM Rate Limits and Outages

Use an AI gateway to handle LLM rate limits and outages with automatic failover, multi-provider routing, and per-tenant quotas. See how Bifrost does it.

LLM rate limits and outages are the two failure modes most likely to take a production AI application offline. A 429 response from the primary provider, a multi-hour incident at a major foundation model lab, or a quota exhausted mid-shift will halt every downstream agent, chatbot, and copilot connected to that endpoint. An AI gateway sits between the application layer and the providers, absorbing these failures through automatic failover, load balancing across keys and providers, and centrally enforced quotas. Bifrost, the open-source AI gateway by Maxim AI, was built to handle exactly this class of problem at production scale, with 11 microseconds of overhead at 5,000 RPS.

The pressure is concrete. By mid-2025, 40% of production LLM teams had multi-provider routing in place, up from 23% ten months earlier, driven by a series of multi-hour incidents at the largest model providers. OpenAI's own postmortems document more than 90% error rates across ChatGPT and several APIs during a single regional power-failure incident, with full recovery taking hours. The cost of single-provider dependence has stopped being theoretical.

Why LLM Rate Limits and Outages Break Production AI

LLM providers operate at materially lower availability than general cloud infrastructure. Tianpan's resilience analysis puts hosted LLM uptime at roughly 99 to 99.5%, against 99.97% for the major cloud providers. A 99% target translates to about 3.5 days of unavailability per year. For an application with revenue, SLA, or customer-facing dependencies, that gap is not absorbable at the application layer.

The failures themselves take a few common shapes:

  • Rate limit storms: bursts of 429 responses when an application crosses TPM (tokens per minute), RPM (requests per minute), or concurrent-request thresholds. These often correlate with marketing launches, batch jobs, or agent loops.
  • Regional or full-provider outages: hours-long incidents triggered by data-center power failures, routing-layer regressions, or capacity shortfalls.
  • Silent quality degradation: HTTP 200 responses with corrupted output, often from configuration drift on the provider side. These bypass naive retry logic entirely.
  • Quota exhaustion: a single team or runaway agent consumes the org's monthly TPM allocation, cutting off every other workload sharing the same key.

Application-level retry logic addresses none of these cleanly. Retrying the same provider during a regional outage just amplifies the problem. Retrying without jitter creates thundering herds. Retrying without circuit breakers can drive cost spikes during long incidents. The clean architectural response is to push reliability concerns out of the application and into a dedicated infrastructure layer.

What an AI Gateway Does for LLM Rate Limits and Outages

An AI gateway is a control plane that intercepts every LLM request from the application before it reaches the provider. It abstracts the provider behind a single API, enforces governance policies, and reroutes traffic when individual providers fail. For rate limits and outages specifically, an AI gateway provides four core mechanisms:

  • Automatic failover across providers and models when the primary returns retryable errors (429, 500, 502, 503, 504).
  • Load balancing across multiple API keys and providers using weighted distribution.
  • Per-consumer rate limits that prevent any single tenant from exhausting shared capacity.
  • Real-time observability into which provider handled each request, retry counts, and failover trigger rates.

These mechanisms are configured once at the gateway and apply uniformly to every workload routed through it. Application code stays unchanged.

How Bifrost Handles LLM Rate Limits and Outages

Bifrost is a Go-based, open-source AI gateway that unifies access to 20+ providers through a single OpenAI-compatible API. It was designed so that the failure-handling logic teams typically scatter across application code (retries, fallbacks, circuit breakers, key rotation) lives in one place and runs at near-zero overhead.

Automatic failover across providers

Bifrost's automatic fallbacks provide failover when a primary provider hits a rate limit, returns an outage error, or becomes unavailable. The fallback array is part of the request body and accepts an ordered list of provider/model combinations:

{
  "model": "openai/gpt-4o-mini",
  "messages": [{"role": "user", "content": "..."}],
  "fallbacks": [
    "anthropic/claude-3-5-sonnet-20241022",
    "bedrock/anthropic.claude-3-sonnet-20240229-v1:0"
  ]
}

If OpenAI returns a 429 or the request fails after configured retries, Bifrost moves to Anthropic. If Anthropic also fails, it tries Bedrock. The first successful response is returned to the caller in OpenAI-compatible format, with an extra_fields.provider field that surfaces which provider actually handled the request. Each fallback is treated as a fresh request: semantic cache lookups run again, plugins re-execute, and governance rules apply per provider.

When multiple providers are configured on a virtual key, Bifrost generates the fallback chain automatically based on weight order, so application code does not need to specify fallbacks on every call.

Retries before fallbacks

Bifrost separates two concerns that application code usually conflates. Retries happen at the provider level for transient failures on the same provider. Fallbacks engage only after retries are exhausted. This distinction matters for cost and latency control: a brief 503 on OpenAI does not need to trigger an Anthropic call if a single retry recovers. Bifrost handles the timing, jitter, and circuit-breaker logic in the gateway itself.

Load balancing across keys and providers

Single-key deployments are vulnerable to per-key rate limits. Bifrost distributes traffic across multiple API keys per provider using weighted distribution, configured at the provider level. Teams running heavy workloads can register several OpenAI keys with different weights and have Bifrost spread requests across them, raising the effective TPM ceiling without provider-tier upgrades. The same mechanism distributes load across providers when multiple are configured on a virtual key.

For enterprise workloads, Bifrost's adaptive load balancing extends weighted routing with predictive scaling and real-time health monitoring, so traffic shifts away from a degrading provider before it starts returning errors.

Virtual keys with per-tenant rate limits

Coarse-grained rate limits at the provider level do not solve the noisy-neighbor problem. One team's batch job can still exhaust the shared organizational quota and starve every other workload. Bifrost's virtual keys are the primary governance entity in the gateway. Each virtual key carries:

  • Token-level rate limits (token_max_limit and token_reset_duration)
  • Request-level rate limits (request_max_limit and request_reset_duration)
  • Hierarchical budgets at the virtual key, team, and customer levels
  • Provider and model allow-lists

A typical configuration assigns each team or application its own virtual key with independent TPM and RPM ceilings, so a runaway agent on one workload cannot affect any other. The full capability set is documented on the Bifrost governance resource page.

Semantic caching to reduce rate-limit pressure

Many rate-limit incidents are self-inflicted. Repeated semantically similar queries from chatbots, RAG pipelines, and agents drive token consumption far above what the workload genuinely needs. Bifrost's semantic caching caches responses based on embedding similarity rather than exact-string match, so common queries hit the cache instead of the provider. The cache reduces both cost and the rate of requests that count against provider quotas, cutting the probability of a 429 storm during traffic spikes.

Implementation: Routing an Existing Application Through Bifrost

Routing an existing application through Bifrost is a single base-URL change. Bifrost is a drop-in replacement for the OpenAI, Anthropic, AWS Bedrock, Google GenAI, LangChain, and LiteLLM SDKs. The application code stays the same:

# Before
client = openai.OpenAI(api_key="sk-...")

# After
client = openai.OpenAI(
    base_url="<http://localhost:8080/openai>",
    api_key="sk-bf-..."  # Bifrost virtual key
)

Bifrost installs in seconds via NPX (npx -y @maximhq/bifrost) or Docker. Once running, providers, virtual keys, fallback chains, and rate limits are configured through the web UI, the API, or a config file. The gateway exposes Prometheus metrics and OpenTelemetry traces, so failover events, retry counts, and per-virtual-key consumption become visible in the existing observability stack.

For platform teams evaluating gateways against this exact resilience problem, the LLM Gateway Buyer's Guide compares feature depth across the major options.

Operational Outcomes from Gateway-Level Resilience

Pushing failover and rate-limit handling into the gateway changes a few measurable production characteristics:

  • Effective availability approaches the union of all configured providers' availability rather than the minimum. Two providers at 99% uptime, configured as a fallback pair, yield a theoretical 99.99% combined availability if their failures are uncorrelated.
  • Rate-limit-induced errors drop to near zero for properly sized workloads, since Bifrost rotates across keys and providers before any single one exhausts.
  • Mean time to mitigation during a provider outage drops from hours (manual intervention, code change, deploy) to seconds (the next request fails over automatically).
  • Cost attribution becomes clean. Per-virtual-key budgets and rate limits give finance and platform teams the tools to enforce policy at the request layer rather than reconciling after the bill arrives.

Performance overhead does not have to be a tradeoff. Bifrost's published benchmarks show 11 microseconds of added latency at 5,000 RPS, which keeps the gateway invisible to end users even at peak load.

Move LLM Reliability Out of Application Code

LLM rate limits and outages will continue to be the dominant failure modes for production AI. Handling them at the application layer with custom retry logic, hand-coded fallbacks, and per-team API keys does not scale past a single workload. An AI gateway centralizes the same concerns into infrastructure that every workload inherits for free, with consistent policy and uniform observability.

To see how Bifrost handles LLM rate limits and outages on your actual provider mix and traffic profile, book a demo with the Bifrost team.

Read next