AI Governance

Budget and Rate Limit Architecture for Multi-Tenant LLM Platforms

Bifrost implements a four-tier hierarchical budget and rate-limit system that gives platform teams precise cost isolation and traffic governance across every tenant, team, and provider.

Multi-tenant LLM platforms share infrastructure across many consumers, which means a single design decision about budget scoping or rate limit granularity propagates to every tenant. When one team has no per-key spend cap, a runaway batch job can exhaust a shared provider quota mid-month. When rate limits are applied only at the provider level, a single noisy tenant can starve other tenants of throughput. Getting the governance architecture right before traffic scales is significantly cheaper than rebuilding it afterward. Bifrost, the open-source AI gateway built in Go by Maxim AI, implements hierarchical budget management and dual-dimension rate limiting as core gateway primitives, with no external service dependency and state kept in memory for sub-millisecond enforcement overhead.

This post covers the architecture decisions behind multi-tenant LLM cost control, how Bifrost's governance model maps to real organizational structures, and how to configure budget and rate limit policies for the most common platform patterns.

Why Per-Tenant Budget Isolation Requires a Hierarchy

A flat budget model, where a single cap covers all consumers of the platform, fails in multi-tenant deployments because it provides no protection at the tenant boundary. One heavy consumer can exhaust the shared budget and block every other tenant. The solution is independent budgets at each organizational level, each enforced separately.

Bifrost's budget and rate limit system operates on a four-tier hierarchy:

Customer (top-level org or tenant)
    ↓
Team (department or sub-group within a customer)
    ↓
Virtual Key (individual application or service credential)
    ↓
Provider Config (per-provider allocation within a virtual key)

Every applicable budget in this chain is checked independently on each request. All must have sufficient remaining balance for the request to proceed. When a cost is incurred, it is deducted from every applicable level simultaneously. A single exhausted budget at any tier blocks the entire request, regardless of how much headroom remains at other tiers.

This design has a specific consequence for platform teams: a Customer-level cap provides organizational protection while Team- and Virtual Key-level caps protect individual consumers from each other. Neither cap makes the other redundant.

The Virtual Key as the Core Governance Unit

Virtual keys are the primary governance entity in Bifrost. Every consumer of the platform authenticates with a virtual key rather than a raw provider API key. The virtual key carries:

An independent budget with a configurable reset window
Request and token rate limits
An explicit allowlist of providers and models
An optional attachment to a Team or Customer for hierarchical budget aggregation

Consumers authenticate using standard API key headers (Authorization, x-api-key, x-goog-api-key, or x-bf-vk). Bifrost resolves the virtual key to the correct provider, model, and underlying credential. Provider API keys never leave the gateway.

Because the virtual key is a gateway-managed credential, governance changes take effect on the next request with no key rotation ceremony and no environment variable updates to push across services. Revoking a key, reducing a budget cap, or restricting model access is a single API call.

Budget Configuration and Reset Windows

Each budget in the hierarchy defines two values: a max_limit in USD and a reset_duration that determines how often the budget window refreshes. Supported durations range from one minute (1m) to one year (1Y). Two reset modes are available:

Rolling window (default): The budget resets reset_duration after the most recent reset timestamp. A monthly rolling budget set up mid-month resets 30 days later, not at the calendar month boundary.

Calendar-aligned: With calendar_aligned: true, the budget resets at the start of each calendar period in UTC, regardless of when the budget was created or last modified. A monthly calendar-aligned budget resets on the first of each month at 00:00 UTC for every customer simultaneously. Calendar alignment is only supported for day, week, month, and year reset durations; sub-day periods are not supported.

For SaaS platforms billing customers on a monthly cycle, calendar-aligned budgets simplify cost attribution because every tenant's reset is predictable and synchronized. For internal teams with different onboarding dates, rolling windows avoid partial-period anomalies.

Rate Limiting: Two Dimensions at Two Levels

Budget controls operate on accumulated spend; rate limits operate on request frequency and token throughput. Bifrost supports two independent rate limit dimensions that run in parallel:

Request limits: maximum API calls within a reset window (e.g., 1,000 requests per hour)
Token limits: maximum tokens (prompt + completion) within a reset window (e.g., 2,000,000 tokens per hour)

Both can be configured at the Virtual Key level and at the Provider Config level within a virtual key. Teams and Customers do not carry rate limits; those are enforced only at the VK and provider-config layers.

Provider-level rate limiting matters for platforms with heterogeneous provider quotas. A virtual key configured for both OpenAI and Anthropic might set 1,000 requests per hour on the OpenAI provider config and 500 requests per hour on Anthropic, reflecting different tier limits on each upstream account. A rate limit violation on one provider config excludes that provider from routing for the current window, but other providers within the same virtual key remain available. This means rate-limited traffic can fail over to a backup provider automatically, which is a meaningfully different behavior from a VK-level hard block.

The full rate limits reference covers both dimensions and their interaction with Bifrost's automatic fallback routing.

Provider-Level Budget Allocation Within a Virtual Key

The Provider Config layer adds a second dimension to budget governance: per-provider caps within a single virtual key. A virtual key with a $1,000/month cap can simultaneously carry a $600/month cap on OpenAI and a $300/month cap on Anthropic. Both provider-level budgets and the VK-level budget are checked on every request; all must pass.

This matters for cost-optimized routing strategies. A team running cost-sensitive workloads through a cheaper provider and precision-sensitive workloads through a premium model can set a tight daily budget on the premium provider config while leaving the cheaper config loosely bounded. When the premium daily budget is exhausted, traffic routes automatically to the cheaper provider without any application-layer change.

{
  "governance": {
    "virtual_keys": [
      {
        "id": "vk-product-team",
        "name": "Product Team",
        "provider_configs": [
          { "id": 1, "provider": "openai", "weight": 0.8, "rate_limit_id": "rl-openai" },
          { "id": 2, "provider": "anthropic", "weight": 0.2, "rate_limit_id": "rl-anthropic" }
        ],
        "rate_limit_id": "rl-vk-product"
      }
    ],
    "budgets": [
      { "id": "b-vk", "virtual_key_id": "vk-product-team", "max_limit": 500.00, "reset_duration": "1M", "calendar_aligned": true },
      { "id": "b-openai", "provider_config_id": 1, "max_limit": 350.00, "reset_duration": "1M" },
      { "id": "b-anthropic", "provider_config_id": 2, "max_limit": 100.00, "reset_duration": "1M" }
    ],
    "rate_limits": [
      { "id": "rl-vk-product", "request_max_limit": 5000, "request_reset_duration": "1h", "token_max_limit": 5000000, "token_reset_duration": "1h" },
      { "id": "rl-openai", "request_max_limit": 3000, "request_reset_duration": "1h", "token_max_limit": 3000000, "token_reset_duration": "1h" },
      { "id": "rl-anthropic", "request_max_limit": 1000, "request_reset_duration": "1h", "token_max_limit": 1000000, "token_reset_duration": "1h" }
    ]
  }
}

Three Multi-Tenant Deployment Patterns

The Bifrost governance model maps to three distinct organizational shapes that platform teams encounter in production.

Internal Multi-Team Platform

A single Bifrost deployment serves every engineering team inside an organization. Each business unit becomes a Customer entity with a monthly cap. Each squad under it becomes a Team with a department-level budget that accumulates independently of the Customer cap. Every individual service or agent gets its own Virtual Key.

The platform engineering team holds provider credentials centrally. Application teams interact only with their own virtual keys, with no visibility into raw API keys or other teams' spending.

External SaaS with Per-Tenant Billing

Bifrost sits between the product and upstream LLM providers. Each paying customer becomes a Customer entity with a monthly budget that matches their subscription plan. When a customer reaches their plan limit, subsequent requests return a block response and the application surfaces an appropriate prompt without any custom billing logic in the product code.

Customers with enterprise plans can receive higher limits or access to premium model allowlists through different virtual key configurations. Model access is controlled at the provider config level, so restricting a tenant to gpt-4o-mini while another tenant accesses gpt-4o is a configuration change, not a code change.

Agentic Workload Isolation

Each autonomous agent gets its own virtual key with a bounded budget, a tight rate limit, and a model allowlist restricted to the models appropriate for its task. A runaway agent that enters an unexpected loop is self-terminating at the gateway layer: it hits its request rate limit within the current window and stops generating charges. The budget cap provides a second containment layer for longer-horizon cost drift.

Because rate limit violations on one provider config exclude only that provider from routing, an agent with a tight Anthropic rate limit can continue operating on OpenAI traffic if configured with both providers.

State Management: OSS vs. Enterprise Multi-Node Deployments

Bifrost keeps all governance state in memory, including provider configs, API keys, budgets, current usage, and traffic distribution. This design makes budget enforcement sub-millisecond and keeps the 11-microsecond total request overhead intact even with complex governance hierarchies active.

On a single-node OSS deployment, the memory state is the authoritative source; the database acts as a persistence layer and seed store. This works for approximately 3,000-5,000 RPS on a single instance and covers most startup and medium-scale deployments.

For high-availability multi-node deployments, Bifrost Enterprise uses a modified RAFT protocol to synchronize governance state across nodes in real time. Budget deductions, rate limit counters, and configuration changes propagate to all nodes within the cluster without requiring a shared cache service or external coordination layer. Running multiple OSS nodes with a shared Postgres backend does not replicate this behavior; only the Enterprise clustering implementation provides synchronized in-memory state.

The full governance capability set, including RBAC, SSO via Okta, Zitadel, Keycloak, and Entra, and compliance alignment, is documented on the Bifrost governance resource page.

Observability for Budget and Rate Limit Enforcement

Budget events and rate limit decisions are captured in Bifrost's built-in observability layer. Every request log includes the virtual key identity, the budget tier that evaluated the request, current usage at each level, and whether a limit was applied. This makes it straightforward to audit which tenant triggered a block, when their budget exhausted, and what remained at other tiers.

For teams running Prometheus or OpenTelemetry collectors, Bifrost exports budget utilization, rate limit state, and per-provider request metrics natively. These feed directly into Grafana dashboards or alert rules without custom instrumentation. For enterprise deployments, the Datadog connector surfaces per-virtual-key spend and rate limit utilization in LLM Observability dashboards.

Teams evaluating the full governance feature matrix against specific compliance or procurement requirements can use the LLM Gateway Buyer's Guide as a structured reference.

Conclusion

Budget and rate limit architecture for multi-tenant LLM platforms requires independent enforcement at each organizational boundary, dual-dimension throttling at both the virtual key and provider levels, and state management that keeps governance overhead from becoming the new bottleneck. Flat budget models and application-layer rate limiting both fail at scale for the same reason: they push enforcement to a layer that cannot see the full request context and cannot update policy atomically.

Bifrost implements this as a core gateway capability: a four-tier budget hierarchy, parallel token and request rate limits at two scopes, provider-level cost allocation, and calendar-aligned or rolling reset windows, all enforced in-memory with no external dependency. To see how Bifrost fits your multi-tenant platform architecture, book a demo with the Bifrost team.