Handle LLM Rate Limits and Outages With an AI Gateway

Rate-limit errors (429 Too Many Requests) and provider outages are two of the most common causes of production incidents in LLM-powered applications. A single provider hitting a quota window or returning 5xx errors can stall an entire agent workflow if requests have nowhere else to go. Bifrost, the open-source AI gateway built in Go by Maxim AI, is the best overall choice for enterprise teams that need to handle LLM rate limits and outages without writing custom retry logic in every service. This post explains what causes these failures and how an AI gateway routes around them using automatic retries, key rotation, and fallbacks.

What Causes LLM Rate Limit Errors and Provider Outages?

LLM rate limit errors and outages come from two distinct failure classes: credential and account limits (the caller's problem) and upstream provider failures (the provider's problem). Understanding which class a failure belongs to determines how a gateway should recover from it.

Rate limits are enforced per API key and per account. Providers cap requests per minute (RPM) and tokens per minute (TPM), and exceeding either returns a 429. According to OpenAI's rate limit documentation, limits are applied at the organization and project level, so multiple keys on the same account can share a quota ceiling. Anthropic enforces similar per-organization limits on its API.

Outages are different. A provider can return 500, 502, 503, or 504 errors, drop connections, or fail DNS resolution during an incident. These are transient upstream failures that no amount of key rotation can fix, because the provider itself is unavailable. Common triggers include:

• Quota exhaustion: a key or account hits its RPM or TPM ceiling and returns 429.
• Auth and billing failures: a revoked key (401/403) or a billing problem (402) blocks requests.
• Transient server errors: the provider returns 5xx responses or refuses connections during an incident.
• Regional degradation: a specific provider region slows down or becomes unreachable.

Bifrost, the open-source AI gateway, classifies every failure into one of these categories and applies a different recovery strategy to each, which is what makes it possible to handle LLM rate limits and outages from a single control point.

How to Handle 429 Rate Limits With Automatic Retries and Key Rotation

To handle 429 rate limits, Bifrost retries the failed request and rotates to a different API key from the provider's key pool. Retries handle transient errors within a provider; fallbacks (covered below) switch providers when every retry is exhausted. Both layers are configured in Bifrost's retries and fallbacks settings.

When a request fails, Bifrost first classifies it as either a per-key failure or a transient server failure:

• Per-key failures (401, 402, 403, 429): the credential or account is the problem, so Bifrost rotates to a different key in the pool.
• Transient server failures (5xx, DNS, connection refused): the upstream is the problem, so Bifrost reuses the same key and waits with exponential backoff before retrying.

For a 429 rate limit specifically, the rate-limited key is marked as used for the current cycle and Bifrost rotates to another key, but it still applies backoff before the next attempt. This is deliberate: providers often enforce account-level quotas shared across keys, so a fresh key may not have new capacity until the quota window slides. Auth and billing failures (401/402/403) are handled without backoff, because waiting cannot revive a permanently dead credential.

The backoff schedule uses exponential growth with jitter:

backoff = min(retry_backoff_initial × 2^attempt, retry_backoff_max) × jitter(0.8–1.2)

With the defaults of retry_backoff_initial: 500ms and retry_backoff_max: 5000ms, retries wait roughly 500ms, 1s, 2s, 4s, then cap at 5s. Retries are configured per provider in network_config, where max_retries defaults to 0 (no retries), so you set it explicitly:

{
  "providers": {
    "openai": {
      "keys": [
        { "name": "openai-key-1", "value": "env.OPENAI_KEY_1", "models": ["*"], "weight": 1.0 },
        { "name": "openai-key-2", "value": "env.OPENAI_KEY_2", "models": ["*"], "weight": 1.0 },
        { "name": "openai-key-3", "value": "env.OPENAI_KEY_3", "models": ["*"], "weight": 1.0 }
      ],
      "network_config": { "max_retries": 5, "retry_backoff_initial": 500, "retry_backoff_max": 5000 }
    }
  }
}

Rate-limited keys are tracked in a per-request set. Once every key in the pool has been tried, Bifrost resets the set and starts a fresh weighted round, since a previously rate-limited key may have free quota by then. With three keys and max_retries: 5, Bifrost can cycle through all three keys twice before giving up. If every configured credential ends up permanently dead, Bifrost returns 502 upstream_credentials_exhausted so the caller can tell that the provider credentials, not the request, are the problem.

How an AI Gateway Prevents Downtime During Provider Outages

An AI gateway prevents downtime during provider outages by failing over to a different provider once the primary has exhausted its retries. Bifrost supports automatic fallbacks across providers and models, so a request that cannot be served by OpenAI can be completed by Anthropic, AWS Bedrock, or any of the other providers in the supported provider list.

Fallbacks work sequentially and each provider gets its own full retry budget:

1. Primary attempt: the configured provider runs with its complete retry budget.
2. Fallback decision: if the primary fails on a retryable error, Bifrost moves to the first fallback provider.
3. Sequential fallbacks: each fallback provider gets its own full retry budget.
4. First success wins: the response from the first provider that succeeds is returned.
5. All fail: the original error from the primary provider is returned.

Each fallback is treated as a completely fresh request, so governance, caching, and logging plugins all run again for the fallback provider. On the gateway, you specify the fallback chain per request as an array of provider/model strings:

curl -X POST <http://localhost:8080/v1/chat/completions> \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-4o-mini",
    "messages": [{ "role": "user", "content": "Summarize this transcript" }],
    "fallbacks": ["anthropic/claude-3-5-sonnet-20241022", "bedrock/anthropic.claude-3-sonnet-20240229-v1:0"]
  }'

Because Bifrost is a drop-in replacement for existing SDKs, adding this resilience requires changing only the base URL in your application. The response reports which provider actually served the request, so observability stays intact through a failover.

How to Load Balance Across Multiple API Keys and Providers

Bifrost load balances across multiple API keys using weighted random selection, distributing traffic so no single key absorbs enough volume to trip its rate limit. This is the first line of defense against 429 errors, because spreading requests across keys and providers keeps each one below its quota ceiling. Key configuration is covered in Bifrost's load balancing settings.

Weighted load balancing gives each key a probability proportional to its assigned weight, which lets teams:

• Prioritize premium keys: assign higher weights to keys with better rate limits.
• Balance production and backup keys: keep spare capacity in reserve at a lower weight.
• Migrate traffic gradually: shift weight during key rotation without downtime.
• Filter by model: restrict specific keys to specific models for cost and access control.

If a selected key fails, Bifrost automatically falls back to the next available key, combining load balancing and failover in a single selection pass. For teams that want routing decisions driven by live performance rather than static weights, adaptive load balancing in Bifrost Enterprise adjusts key and provider weights automatically based on real-time error rates, latency, and throughput, and shares rate-limit signals across nodes so an overloaded key is backed off fleet-wide. The full routing model, including how governance rules and adaptive balancing interact, is documented in the provider routing guide. Teams evaluating gateways for this can review the governance and routing resources for a capability breakdown.

How to Prevent Quota Exhaustion With Rate Limits and Budgets

Preventing quota exhaustion starts before requests reach a provider, by capping usage at the gateway. Bifrost applies request-based and token-based rate limits and budgets through virtual keys, so a single team or customer cannot consume shared provider quota and trigger 429 errors for everyone else.

Governance in Bifrost is hierarchical. Budgets and rate limits can be set at the virtual key and provider-config levels, and budgets alone can be set at the team and customer levels above them:

• Rate limiting: request-based and token-based throttling at the virtual key and provider-config levels.
• Budget management: independent spend limits at each level, checked cumulatively per request.
• Provider-level governance: per-provider budgets and rate limits within a single virtual key.
• Usage tracking: real-time monitoring and audit trails for every consumer.

Providers that exceed their configured budget or rate limits are excluded from routing, so the gateway steers traffic toward capacity that is still available instead of retrying against an exhausted quota. This turns rate limiting from a source of errors into a routing input. For teams standardizing access control across many projects, the Bifrost governance resources cover virtual keys, budgets, and rate limits in depth.

Common Questions About LLM Rate Limits and Outages

What is a 429 rate limit error?

A 429 Too Many Requests error means an API key or account has exceeded its allowed requests per minute or tokens per minute. Providers enforce these limits per organization, so distributing load across keys and adding retry-with-rotation is the primary way to reduce 429 errors in production.

Do retries alone solve provider outages?

No. Retries handle transient errors within a single provider, but a full provider outage requires failover to a different provider. An AI gateway combines both: retries with key rotation for 429 and 5xx errors, and automatic fallbacks to another provider when retries are exhausted.

How does key rotation reduce rate-limit errors?

Key rotation moves a rate-limited request to a different API key in the same provider's pool. Because Bifrost tracks which keys have been tried and resets the pool after a full cycle, a key whose quota window has since slid can be reused, giving requests more chances to succeed before failover.

Can Bifrost run in high-availability deployments?

Yes. Bifrost supports clustering for high availability with automatic service discovery and zero-downtime deployments, which pairs with retries and fallbacks so the gateway itself is not a single point of failure. Enterprise deployment options are on the Bifrost Enterprise page.

Reliability Without Application Code Changes

The combined effect of retries, key rotation, load balancing, fallbacks, and gateway-level rate limiting is that applications stay available through rate limits, transient outages, and full provider failures without any changes to application code. All of this logic lives in the gateway, configured once and applied to every request.

Bifrost adds only 11 microseconds of overhead per request at 5,000 requests per second in sustained benchmarks, so this resilience layer does not become a latency cost. Requests, retries, and fallbacks are visible through Prometheus and OpenTelemetry observability, and the full configuration reference is in the Bifrost documentation. Additional guides live in the Bifrost resources hub.

Handle LLM Rate Limits and Outages With Bifrost

Rate limits and provider outages are predictable failure modes, and an AI gateway turns them into routing decisions instead of production incidents. With automatic retries, weighted load balancing across keys, key rotation on 429 errors, and provider fallbacks, the open-source Bifrost gateway lets teams handle LLM rate limits and outages from a single, drop-in configuration layer. To see how Bifrost can keep your AI applications running through rate limits and outages, book a demo with the Bifrost team.