LLM Budget Management: Virtual Keys and Hierarchical Spend Controls

LLM budget management is the practice of allocating, enforcing, and tracking spend across the teams, projects, and users that consume large language models, so no single consumer can exceed its allotment. Bifrost, the open-source AI gateway built in Go by Maxim AI, implements this at the request layer: it evaluates budgets before a request reaches a provider and blocks anything that would breach a limit. This post explains why LLM spend is difficult to control, how virtual keys become the unit of budget allocation, and how hierarchical budgets, reset durations, and rate limits give platform teams precise financial governance over production AI workloads.

Why LLM Spend Is Hard to Control

LLM spend is hard to control because cost is generated per token, distributed across many teams, and billed after the fact by each provider. Enterprise generative AI spend reached $8.4 billion by mid-2025, more than doubling in six months even as per-token prices fell, according to Menlo Ventures' State of Generative AI in the Enterprise. A separate a16z survey of enterprise CIOs found that leaders expect roughly 75% growth in LLM spend over the following year.

Three structural problems make this spend difficult to govern:

• No native per-consumer limits. Provider API keys are shared credentials. A raw OpenAI or Anthropic key has one account-level bill and no built-in way to cap what an individual team, project, or user consumes.
• Attribution happens after billing. Costs are computed from token counts in the response, so a runaway agent or a misconfigured retry loop is visible only after the tokens are spent.
• Spend crosses provider boundaries. Teams route across multiple providers and models, and each provider prices differently, so a single organizational budget cannot be enforced from any one provider dashboard.

An AI gateway addresses this by sitting in front of every provider and enforcing spend policy at the point of the request, before tokens are consumed. This is the layer where LLM budget management becomes enforceable rather than observational.

Virtual Keys: The Unit of Budget Allocation

A virtual key is a credential issued by the gateway that carries its own access permissions, budget, and rate limits, decoupled from the underlying provider API keys. In Bifrost, virtual keys are the primary governance entity. Instead of distributing raw provider keys, a platform team issues a virtual key per team, project, application, or user, and attaches a budget to each one.

Each virtual key carries:

• Independent budgets. A dollar limit checked on every request, evaluated alongside any team or customer budget the key is attached to.
• Token- and request-based rate limits. Throttling on both tokens per period and requests per period.
• Access control. Model and provider filtering, so a key can be restricted to specific models or providers.
• Key restrictions. A virtual key can be limited to specific provider API keys, so its traffic only ever uses the credentials you intend.
• Instant activation status. A key can be enabled or disabled immediately to cut off access without rotating provider credentials.

Because the virtual key is the object budgets attach to, it becomes the natural unit of allocation. Callers authenticate with a virtual key (via the x-bf-vk, Authorization, x-api-key, or x-goog-api-key header), and every request inherits that key's spend policy. Bifrost's governance model is built around this pattern, which lets a platform team map financial accountability directly onto the credentials each consumer holds.

Hierarchical and Cumulative Budgets

Hierarchical budgets let an organization set independent dollar limits at the customer, team, and virtual key levels, where a request must pass every applicable budget before it is allowed. Bifrost enforces a hierarchical budget structure that mirrors how organizations actually allocate money: a customer contains teams, a team contains virtual keys, and a virtual key contains per-provider configurations, with each level holding its own budget.

The hierarchy flows top to bottom:

• Customer has an independent budget.
• Team (one-to-many under a customer) has an independent budget.
• Virtual key (one-to-many under a team, or attached directly to a customer, or standalone) has an independent budget plus rate limits.
• Provider config (one-to-many under a virtual key) has an independent budget plus rate limits.

How cumulative budget checking works

When a request arrives, Bifrost checks all applicable budgets independently and the request proceeds only if every one has sufficient remaining balance. The check runs from the most specific level outward: provider config budget, then virtual key budget, then team budget, then customer budget. If any single budget is exhausted, the request is blocked. When the request succeeds, its cost is deducted from every applicable budget at the same time.

This cumulative model means a team can distribute a shared pool across many virtual keys without any one key overspending the team total. A virtual key might have $10 remaining and its parent team $20 remaining; a request that would fit the key's budget is still blocked if it would breach the team's. A virtual key can belong to either one team or one customer, not both, which keeps each key's cost attribution unambiguous.

Reset Durations and Calendar Alignment

Reset durations define the time window over which a budget or rate limit accumulates before it returns to zero. Bifrost supports reset durations of 1m, 5m, 1h, 1d, 1w, 1M, and 1Y, so budgets can be scoped to a minute, an hour, a day, a week, a month, or a year depending on how the team plans spend.

By default a budget uses a rolling window: usage resets once the reset duration has elapsed since the last reset. Bifrost also supports calendar-aligned budgets, where the budget resets at the start of each calendar period in UTC instead of on a rolling clock:

• 1d resets at midnight UTC each day.
• 1w resets Monday at 00:00 UTC each week.
• 1M resets on the first day of each month.
• 1Y resets on January 1 each year.

Calendar alignment applies only to day, week, month, and year periods; sub-day durations such as 1h or 30m cannot use it. This matters for finance and FinOps teams whose reporting runs on calendar boundaries: a calendar-aligned monthly budget resets on the same instant as the billing cycle, so gateway spend maps cleanly onto the month being reconciled rather than to an arbitrary rolling window.

Token- and Request-Based Rate Limits

Rate limits cap how much traffic a virtual key can generate within a time window, using two thresholds that run in parallel: a request limit and a token limit. This distinction matters for LLM budget management because token consumption, not request count, drives cost. A single request with a large context window can spend more than a hundred small requests.

Bifrost supports both limit types together:

• Request limits cap the number of API calls within a period, for example 100 requests per minute.
• Token limits cap the total prompt-plus-completion tokens within a period, for example 50,000 tokens per hour.

For a request to pass, it must satisfy both the request limit and the token limit at every applicable level. Rate limits can be configured at the virtual key level and the provider config level, each with its own reset duration. Provider-level rate limiting lets a team throttle OpenAI and Anthropic differently within the same virtual key, and a rate-limit breach on one provider excludes only that provider from routing while the others stay available. Teams and customers do not carry rate limits; those thresholds live at the virtual key and provider config levels, while budgets span the full hierarchy.

Observability of LLM Spend

Spend observability is the ability to see what each team, key, and model actually costs in real time, which is the feedback loop that makes budgets meaningful. Bifrost includes built-in observability that captures every request and response with its input and output tokens, computed cost, latency, provider, and model, all without changes to application code. The logging runs asynchronously, so it adds no latency to the request path.

For teams that centralize telemetry, Bifrost exports metrics through native Prometheus integration and OpenTelemetry (OTLP) for distributed tracing, compatible with Grafana, New Relic, and Honeycomb. Because cost is calculated per request from provider pricing and token usage, spend can be attributed by virtual key, and therefore by team or project, without a separate cost-allocation pipeline. This closes the loop: budgets set the ceiling, rate limits shape the flow, and observability shows where the spend is going while there is still time to adjust.

Teams looking to reduce spend at the source can also pair budget controls with the MCP gateway's Code Mode, which lowers token costs at scale by having models orchestrate tools through generated code instead of expanding every tool schema inline.

Enterprise Governance at Scale

For large organizations, LLM budget management is one part of a broader access and compliance posture. Bifrost is built for enterprises running mission-critical AI workloads, and its governance layer extends past budgets into role-based access control, SSO and OIDC through providers such as Okta and Microsoft Entra, and immutable audit logs for SOC 2, GDPR, HIPAA, and ISO 27001. Access profiles let a platform team define reusable budget, rate-limit, and access policies that auto-allocate virtual keys as new teams and users onboard, so spend governance holds its shape as headcount grows.

Regulated teams can deploy Bifrost inside their own environment through in-VPC and on-prem options, keeping budget enforcement, spend data, and provider credentials within their security boundary. The same governance resource hub covers how virtual keys, budgets, and rate limits combine into a single control plane for AI spend.

Getting Started with LLM Budget Management in Bifrost

LLM budget management in Bifrost starts with issuing a virtual key per team or project, attaching a budget and a reset duration, and layering token- and request-based rate limits on top. Hierarchical budgets then let you nest team and customer limits above each key, so cumulative checks keep every consumer inside both its own allotment and the organization's total. With built-in observability tracking cost per key in real time, platform teams get financial governance over production AI without building it themselves.

To see how virtual keys, hierarchical budgets, and spend controls work on your own workloads, book a demo with the Bifrost team.