LLM API Rate Limiting with Virtual Keys and Budgets

TL;DR

• LLM API rate limiting caps how many requests and tokens a consumer can send in a time window, while budgets cap how much money that consumer can spend; you need both to prevent 429 errors and runaway cost.
• In Bifrost, the open-source AI gateway by Maxim AI, a virtual key is the unit that carries access permissions, budgets, and rate limits, so limits attach to a team, a customer, or a single application rather than to a raw provider key.
• Bifrost enforces budgets hierarchically across customer, team, virtual key, and provider config, and every applicable budget must pass before a request proceeds.
• Rate limits in Bifrost are set as request limits and token limits, applied at the virtual key level and the provider config level, with reset durations from one minute to one year.
• When a provider exceeds its budget or rate limit, Bifrost excludes it from routing rather than failing the request, so other providers on the same key keep serving traffic.

AI applications that call LLM providers hit two ceilings under load: provider rate limits that return HTTP 429 errors, and monthly bills that climb faster than any forecast predicted. LLM API rate limiting is the mechanism that controls both, by throttling request volume and token throughput before either ceiling is reached. Bifrost, the open-source AI gateway built in Go by Maxim AI, applies rate limits and budgets through virtual keys so platform teams can set per-key, per-team, and per-customer controls from one place. This guide covers how API rate limiting and budgets work together, and how to configure them in Bifrost's governance system to stop runaway spend and cascading 429s.

What Is LLM API Rate Limiting?

LLM API rate limiting is a control that restricts how many requests or tokens a consumer can send to a language model within a fixed time window. It protects upstream providers from overload, keeps one tenant from starving others, and gives platform teams a lever to cap traffic per application, team, or customer before provider-side limits reject the traffic first.

Every LLM provider enforces its own rate limits. OpenAI, Anthropic, and Google publish per-model limits on requests per minute and tokens per minute, and a request that exceeds them returns an HTTP 429 response defined by RFC 6585. Those limits are set by the provider and apply to your entire account, which means a single misbehaving job can consume the quota that every other workload depends on. Gateway-level rate limiting moves the enforcement point in front of the provider, so you decide how the shared quota is divided. This is the core reason AI gateways tackle rate limiting for LLM apps more effectively than per-client retry logic.

Bifrost implements this as two limit types that run in parallel on each virtual key: a request limit and a token limit. A request limit caps the number of API calls in a window, for example 100 requests per minute. A token limit caps the total prompt and completion tokens, for example 50,000 tokens per hour. A request must satisfy both checks to proceed, which matters for LLM traffic because a small number of long-context calls can exhaust a token budget while barely touching a request count.

Rate Limiting vs. Budgets: Two Controls, One Goal

Rate limiting and budgets solve different failure modes with the same goal of predictable, governed AI spend. Rate limiting controls the rate of consumption, requests and tokens per unit time, to prevent 429 errors and protect throughput. Budgets control the total dollar amount consumed over a period, to prevent cost overruns. A key can be inside its rate limit and still blow its budget, so both controls run independently.

The distinction is worth making concrete, because teams often reach for one when they need the other:

Bifrost computes cost for every request from real-time provider pricing, input and output token counts, request type, and cache status, then deducts that cost from each applicable budget. Because budgets and rate limits are configured on the same virtual key, a platform team sets both in one place rather than stitching together a proxy for throttling and a separate billing exporter for cost. Bifrost's governance capabilities treat the two as one policy surface. For a deeper treatment of pairing spend caps with alerts, the guide on controlling LLM costs with budget alerts and rate limits walks through the alerting side.

Virtual Keys: The Unit of Access, Budget, and Rate Limiting

A virtual key in Bifrost is a governance entity that bundles access permissions, a budget, and rate limits behind a single credential that applications use in place of a raw provider API key. Consumers authenticate with the virtual key through standard headers, and the gateway resolves which models, budgets, and limits apply. This is what makes per-team and per-customer control practical at scale.

Virtual keys authenticate through the headers your SDKs already send, so adopting them is close to a drop-in change. Bifrost accepts the virtual key on Authorization: Bearer sk-bf-* (OpenAI style), x-api-key (Anthropic style), x-goog-api-key (Gemini style), api-key (Azure style), or the native x-bf-vk header. Because Bifrost is a drop-in replacement for the provider SDKs, pointing an existing OpenAI or Anthropic client at the gateway and swapping the key is usually the entire integration.

Each virtual key carries its own controls, which is what lets one gateway serve many tenants safely:

• Access control: model and provider filtering, so a key can only reach approved models.
• Independent budget: a dollar limit checked alongside any team or customer budget attached to the key.
• Rate limiting: token-based and request-based throttling configured directly on the key.
• Key restrictions: limit a virtual key to specific provider API keys.
• Active status: enable or disable a key instantly to cut off access without rotating credentials.

A virtual key attaches to exactly one team, or one customer, or neither, which keeps the ownership model unambiguous. Teams that also need to govern which MCP tools a key can call can layer per-key tool filtering on top of the same virtual key. For a fuller picture of managing keys and spend limits together, see the walkthrough on virtual keys and hierarchical spend controls.

Hierarchical Budgets for Teams and Customers

Hierarchical budgets let a single spend cap be enforced at multiple levels at once, so a customer, a team within that customer, and an individual virtual key each hold their own limit. Bifrost checks every applicable budget in the hierarchy independently, and a request proceeds only when none of them is exceeded. This is how a multi-tenant platform enforces a customer-wide cap while still limiting each team inside it.

The core hierarchy runs from customer down to provider config:

Customer (independent budget)
    -> Team (independent budget)
        -> Virtual Key (independent budget + rate limits)
            -> Provider Config (independent budget + rate limits)

When a request arrives, Bifrost verifies the provider config budget, the virtual key budget, then the team and customer budgets in sequence. Each level tracks its own usage, and the request cost is deducted from all applicable levels. If any single budget lacks sufficient balance, the request is blocked before it reaches the provider. The budget and limits documentation gives the full checking sequence for standalone keys, direct customer attachments, and the full customer-to-team-to-key chain.

Budgets reset on flexible durations. A rolling budget resets after its reset_duration elapses from the last reset. A calendar-aligned budget instead resets at the start of each calendar period in UTC, so a 1M budget rolls over on the first of the month and a 1Q budget rolls over on the first day of the fiscal quarter. That distinction matters for finance teams that reconcile against calendar months rather than rolling windows. Patterns for dividing budgets across many tenants are covered in the reference on budget and rate limit architecture for multi-tenant LLM platforms.

Setting Token and Request Rate Limits per Key and Provider

Bifrost applies rate limits at two levels: the virtual key and the provider config within that key. Teams and customers hold budgets but not rate limits, so throttling always lives on the key or on a specific provider. This split lets a team set one overall request ceiling on a key while giving each provider inside it a different token limit tuned to that provider's own quota.

Rate limits are checked in hierarchical order, provider config first, then virtual key, and a request must pass both the request limit and the token limit at every applicable level. Provider-level rate limits enable three patterns that a single account-wide provider limit cannot:

• Per-provider throttling: different limits for OpenAI and Anthropic on the same key, matched to each provider's published quota.
• Provider isolation: a rate limit breach on one provider does not affect the others on the key.
• Granular tuning: limits set per provider based on cost and capacity rather than one blanket number.

The key behavior for reliability is what happens at the ceiling. When a provider config exceeds its rate limit, Bifrost excludes that provider from routing rather than returning an error, and other providers on the same virtual key continue serving the request. Combined with automatic fallbacks and weighted load balancing, this turns a hard 429 into a routing decision.

The pattern of pairing per-user limits with fallbacks is detailed in the guide on per-user rate limiting, budget controls, and automatic fallbacks.

Handling OpenAI Rate Limits and Provider 429s

Provider rate limits are account-wide and unavoidable, so the practical question is how to absorb them without dropping requests. OpenAI enforces requests-per-minute and tokens-per-minute limits per model, documented in the OpenAI rate limits guide, and Anthropic enforces similar per-model limits described in the Anthropic rate limits documentation. When your traffic exceeds either, the provider returns a 429 and the request fails unless something catches it.

Bifrost catches it in two ways. First, gateway-level rate limits keep each virtual key inside a share of the account quota, so no single consumer drives the account into provider-side throttling. Second, when a provider does hit its limit, Bifrost routes around it. Because provider routing excludes any provider that has exceeded its budget or rate limit, a request that would have returned a 429 from OpenAI is instead served by Anthropic or another configured provider on the same key.

This is the difference between LLM API rate limiting as a blocking control and rate limiting as a routing input. A raw client that hits an OpenAI rate limit retries with backoff and adds latency. A request through Bifrost that hits the same limit is redirected to a healthy provider, so the caller sees a successful response. The supported providers matrix shows which of the 1,000+ models can serve as fallbacks for one another. For teams managing outages alongside limits, the guide on handling LLM rate limits and outages with an AI gateway covers the reliability side in depth.

Implementing LLM API Rate Limiting with Bifrost

Configuring LLM API rate limiting in Bifrost means creating a virtual key with a budget and rate limits attached, then pointing your application at the gateway. The whole flow is a single API call to create the key, followed by a base-URL change in your client. Below is a virtual key scoped to an engineering team with a monthly budget and per-minute limits.

curl -X POST <http://localhost:8080/api/governance/virtual-keys> \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Engineering Team API",
    "team_id": "team-eng-001",
    "provider_configs": [
      { "provider": "openai", "weight": 0.5, "allowed_models": ["gpt-4o-mini"] },
      { "provider": "anthropic", "weight": 0.5, "allowed_models": ["claude-3-5-sonnet"] }
    ],
    "budget": { "max_limit": 100.00, "reset_duration": "1M" },
    "rate_limit": {
      "token_max_limit": 50000, "token_reset_duration": "1h",
      "request_max_limit": 100, "request_reset_duration": "1m"
    }
  }'

This key allows two providers with equal weight, caps spend at $100 per month, and throttles at 50,000 tokens per hour and 100 requests per minute. An application then sends the returned sk-bf-* key on the Authorization header, and every request is metered and governed automatically. Setup for the gateway itself is covered in the gateway quickstart. Teams standardizing many keys can define reusable access profiles that auto-allocate virtual keys with a fixed budget and rate-limit policy, and enforce who can create them with role-based access control.

Once keys are live, usage needs to be visible. Bifrost exposes real-time request metrics through native Prometheus metrics and records governed activity in immutable audit logs for SOC 2, GDPR, and HIPAA reporting.

For regulated deployments, the same governance runs inside a private network through in-VPC deployment, and the Bifrost Enterprise tier adds clustering and real-time state sync across nodes. The LLM gateway buyer's guide compares these controls against evaluation criteria.

Frequently Asked Questions

What is the difference between rate limiting and a budget for LLM APIs?

Rate limiting caps how fast a consumer sends traffic, measured as requests or tokens per time window, and prevents 429 errors. A budget caps how much money a consumer spends over a period, measured in dollars, and prevents cost overruns. Bifrost enforces both independently on the same virtual key, so a request must be inside its rate limit and its budget to proceed.

How do virtual keys enforce per-team and per-customer rate limits?

A virtual key carries its own budget, rate limits, and access rules, and attaches to exactly one team or one customer. Applications authenticate with the key instead of a raw provider key, so every request is attributed to that team or customer and metered against its limits. This gives platform teams per-team and per-customer control without issuing separate provider credentials.

Can Bifrost prevent OpenAI 429 rate limit errors?

Bifrost reduces 429 errors two ways. Gateway rate limits keep each virtual key inside a share of the account quota so no consumer overloads the provider, and provider routing excludes any provider that has exceeded its rate limit. A request that would have returned a 429 from OpenAI is instead routed to another configured provider on the same key, so the caller receives a successful response.

How do hierarchical budgets work across teams and customers?

Bifrost checks budgets at every level of the hierarchy, from provider config up through virtual key, team, and customer. Each level holds an independent budget and tracks its own usage, and the request cost is deducted from all applicable levels. A request proceeds only if every budget in the chain has sufficient balance, so a customer-wide cap and a per-team cap are enforced simultaneously.

What reset durations does Bifrost support for rate limits and budgets?

Rate limits reset on windows from one minute to one day, typically 1m, 1h, or 1d. Budgets reset on longer periods: 1d, 1w, 1M, 1Q, and 1Y. Budgets can also be calendar-aligned, resetting at the start of each calendar period in UTC rather than on a rolling window, which suits finance teams reconciling against calendar months and fiscal quarters.

Does adding a gateway for rate limiting slow down LLM requests?

Bifrost adds roughly 11 microseconds of overhead per request at 5,000 requests per second in sustained benchmarks. Rate limit and budget checks run in memory as part of the request pipeline, so governance does not add a network round trip. A single open-source instance handles around 3,000 to 5,000 requests per second before clustering is needed.

Getting Started with LLM API Rate Limiting

Controlling LLM API rate limiting and spend comes down to one decision: where the request and token limits live. Enforcing them on raw provider keys means one account-wide quota that every workload shares, while enforcing them on virtual keys in Bifrost means per-key, per-team, and per-customer limits with hierarchical budgets behind them. Bifrost, the open-source AI gateway by Maxim AI, unifies rate limiting, budgets, and routing so a 429 becomes a routing decision instead of a failed request. To see how virtual keys and budgets fit your governance model, book a demo with the Bifrost team, review the governance and access controls in one place, or read the broader guide on centralizing rate limiting for LLM apps at the gateway.