Control LLM Costs with Budget Alerts and Rate Limits

TL;DR

• LLM costs are driven by token volume rather than provisioned capacity, so they can rise by an order of magnitude in a day with no deployment change.
• Bifrost enforces budgets at four independent levels (customer, team, virtual key, provider config), and blocks a request when any applicable budget is exhausted.
• Bifrost Enterprise evaluates CEL alert rules against live governance metrics on a 60 second sweep and dispatches notifications to Slack, Microsoft Teams, PagerDuty, or an HTTP webhook.
• Rate limits in Bifrost apply to both request count and token count, at the virtual key level and the provider config level, so one provider hitting its ceiling does not take the others down with it.
• Bifrost adds 11 microseconds of overhead per request at 5,000 requests per second, so budget and rate limit checks are not paid for in latency.

LLM costs are driven by token volume rather than by seat count or instance hours, which is why they move faster than most budgeting processes can track. Bifrost, the open-source AI gateway built in Go by Maxim AI, enforces budgets, budget alerts, and rate limits at the point every model call passes through, so an agent stuck in a retry loop is stopped by policy rather than discovered on an invoice. This post covers how budget hierarchies, alert rules, and request and token limits work together to control LLM costs across teams, customers, and providers.

What Drives LLM Costs in Production

LLM costs come from four variables: request volume, input and output tokens per request, which model handles it, and how often the same work is repeated. Only the fourth is easy to remove; the other three are properties of application behavior, which changes faster than procurement cycles.

The failure mode that produces surprise invoices is a step change rather than steady growth: an agent framework retrying a failed tool call twenty times, a RAG pipeline whose context window grows after an indexing change, or a batch job pointed at a frontier model. None of these look like incidents. Latency stays normal, error rates stay normal, and the only signal is the cost line. Cost control therefore belongs at the gateway: application-level accounting sees only one application's calls, while an AI gateway sees every call from every service and can enforce a limit that corresponds to an actual budget owner.

Where an LLM Gateway Fits in LLM Cost Control

An LLM gateway is a single ingress point every model request passes through, which makes it the one place in the stack where usage can be attributed, capped, and alerted on consistently. Because the gateway sits before the provider call, it refuses a request that would exceed a budget rather than reporting the overspend afterward. Observability tools report what a system spent; a gateway decides what it is allowed to spend, and a weekly cost report is a lagging signal.

Bifrost is built for that enforcement role. Governance state (budgets, usage counters, rate limit windows) is held in memory and checked inline on the request path. Published benchmarks show 11 microseconds of added overhead per request at 5,000 requests per second, so enforcement does not become the latency problem it was added to prevent. Teams evaluating the category can use the LLM gateway buyer's guide, while what an LLM gateway does for cost tracking covers the reporting side.

Bifrost is a drop-in replacement for existing provider SDKs, so putting a gateway in front of an application that already calls OpenAI or Anthropic directly means changing a base URL rather than rewriting call sites.

Setting Budgets Across Customers, Teams, and Virtual Keys

Bifrost budgets are hierarchical. A budget attaches to a customer, a team, a virtual key, or an individual provider config inside a virtual key. Each tracks its own usage independently, and every applicable budget must have remaining balance for a request to proceed.

Virtual keys are the primary governance entity. A client authenticates with a virtual key (sent as x-bf-vk, or in the Authorization, x-api-key, api-key, or x-goog-api-key header), and that key carries its own permissions, budget, and rate limits. A virtual key attaches to exactly one team or one customer, never both, which keeps attribution unambiguous. The budget and limits model checks every level in sequence, blocks on any failure, and deducts cost from all applicable budgets.

Two details make these budgets usable in a real finance process. Budgets can be calendar aligned, resetting at the start of a UTC day, week, month, quarter, or year instead of on a rolling window; setting the quarter start month moves a quarterly budget onto a fiscal calendar such as the April year used in the UK and India. And a budget can carry a temporary override that adds capacity on top of the configured limit for a set number of cycles without editing the base limit or clearing usage, which handles a month-end spike without permanently raising a cap someone will forget to lower.

At scale, access profiles in Bifrost Enterprise define a reusable policy (providers, model whitelist, budget, rate limits, MCP tool access) and auto-issue a per-user virtual key with independent counters when a user takes on a role.

Budget Alerts: Warnings Before the Cap

Budget alerts notify a team that spend is approaching a limit while there is still room to act. Bifrost Enterprise evaluates alert rules written as CEL expressions against live governance snapshots on a 60 second sweep, and notifies when a rule evaluates to true.

A hard cap alone is blunt. A budget that blocks at 100% protects the invoice but breaks the application, and the owning team finds out from a support ticket. Alerts at 50%, 80%, and 100% turn the same budget into a graduated control: the owner is warned, judges whether the spend is legitimate, and either raises the limit deliberately or investigates what changed.

Rules are scoped to a virtual key, team, or customer, and can target one named budget. The CEL expressions reference governance variables directly, so conditions read without a query language:

# Warn at 80 percent of any budget in scope
budget_usage_percent >= 80.0

# Page on budget exhaustion or token saturation
budget_usage_percent >= 100.0 || rate_limit_token_usage_percent >= 100.0

# High spend that is also high volume, usually a loop
budget_usage_percent > 80.0 && request_usage > 10000

Available variables include budget_usage_percent, budget_spent, rate_limit_request_usage_percent, rate_limit_token_usage_percent, request_usage, and token_usage, plus scope and target identifiers. Full syntax sits in the alert rules reference.

Notifications go to alert channels: Slack (Block Kit), Microsoft Teams (Adaptive Cards, 28 KB payload limit), PagerDuty (Events API v2, with a deduplication key derived from rule, scope, and target so repeat triggers update one incident), or a generic HTTPS webhook with custom headers. Channel configuration is encrypted at rest.

Three properties keep alerting usable rather than noisy. Per-rule cooldowns, with optional per-channel cooldowns, suppress repeat notifications for the same rule, scope, and target. In a clustered deployment only the leader node evaluates and dispatches, so a three node gateway does not send three copies of one alert. Every outcome, sent, failed, or skipped, is recorded in the alert history log.

Handling OpenAI Rate Limits and Token Quotas at the Gateway

Rate limits at the gateway serve a different purpose than the ones providers impose. OpenAI rate limits and Anthropic rate limits protect the provider's infrastructure and are shared across everything an organization runs on that account. Gateway rate limits protect the budget owner and are scoped to the consumer rather than the account.

Bifrost supports two limit types checked in parallel. Request limits cap calls in a window, for example 100 requests per minute. Token limits cap prompt plus completion tokens, for example 50,000 tokens per hour. Token limits matter more for cost, because request count says nothing about how expensive each request is: a hundred short classification calls and a hundred long-context summarization calls differ by two orders of magnitude in spend.

Limits apply at the virtual key and provider config levels, with reset durations from 1m through 1Y (1Q is budgets only). Provider-level limits give isolation: if a virtual key's OpenAI config exceeds its limit, that provider is excluded from routing while the key's Anthropic config keeps serving traffic, which pairs with automatic fallbacks across supported providers.

Model limits extend the same idea to individual models. A global cap on gpt-4o across all traffic, or a per virtual key cap on one expensive model, closes the gap budgets alone leave open: a budget constrains total spend without constraining which model consumed it. The application-layer version of this problem is described in how AI gateways handle rate limiting for LLM apps.

Cutting Repeat Spend with Semantic Caching

Semantic caching removes cost rather than capping it, by serving a stored response instead of paying for a provider call. Bifrost offers two lookup paths: direct hash matching, which replays an exact repeat with no embedding step, and semantic similarity matching, which serves a cached answer when a new request is close enough to a previous one. Direct runs first and semantic search runs only on a direct miss. Both require a cache key on the request (the x-bf-cache-key header or a configured default), which is the most common reason a newly enabled cache appears to do nothing.

Semantic caching carries a trade-off that decides where it pays off. A semantic lookup must embed the incoming request before it can search, so a semantic miss pays an embedding call on top of the full model call. On workloads with high prompt repetition (support deflection, documentation Q&A, classification over a bounded input space) the hit rate more than covers that. Where every prompt is unique, direct-only mode is the better configuration. The economics are worked through in reducing LLM costs with semantic caching at the gateway and in optimizing LLM cost and latency together. Cached responses are also priced correctly: Bifrost's cost calculation accounts for cache status, so a cache hit does not draw down a budget as though it were a full provider call.

Attributing AI Spend to Teams, Products, and Customers

Attribution is the precondition for every other control. AI spend that cannot be assigned to a team, product, or customer cannot be budgeted, alerted on, or charged back, so most cost problems are attribution problems first.

The Bifrost AI gateway attributes spend through the structure that enforces it. Every request carries a virtual key, the key belongs to a team or a customer, and cost is calculated per request from real-time model pricing, token counts, request type, cache status, and batch status. That produces the per-request cost records an enterprise LLM gateway is meant to hold, instead of a monthly provider total.

For multi-tenant products, Bifrost Enterprise supports customer-scoped requests. When a team is attached to more than one customer, sending x-bf-customer-id or x-bf-customer-name charges and rate limits one named customer for that request. The scope is validated before the request reaches a provider and fails closed: an unknown customer is rejected with a 400 rather than silently charged to the wrong account.

The governance resource page covers how these pieces fit together, and tracking GenAI spend across an enterprise covers the reporting workflow. For regulated environments, audit logs provide immutable trails for SOC 2, GDPR, HIPAA, and ISO 27001 evidence.

Configuring Budgets, Alerts, and Rate Limits in Bifrost

Configuration follows the hierarchy: create the budget owner, attach a virtual key with limits, then add alert rules. All of it is available through the Bifrost dashboard, the HTTP API, or config.json, so policy can be version controlled rather than clicked into place. Start the gateway (setup guide), then create a virtual key carrying a budget and rate limits:

curl -X POST <http://localhost:8080/api/governance/virtual-keys> \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Engineering Team API",
    "provider_configs": [
      {"provider": "openai", "weight": 0.5, "allowed_models": ["gpt-4o-mini"]},
      {"provider": "anthropic", "weight": 0.5, "allowed_models": ["claude-3-sonnet-20240229"]}
    ],
    "team_id": "team-eng-001",
    "budget": { "max_limit": 100.00, "reset_duration": "1M" },
    "rate_limit": {
      "token_max_limit": 10000,
      "token_reset_duration": "1h",
      "request_max_limit": 100,
      "request_reset_duration": "1m"
    },
    "is_active": true
  }'

Three choices carry most of the value. allowed_models restricts the key to inexpensive models, removing the most common source of unplanned spend. Splitting weight across two providers means a rate limit on one does not stop the workload. And reset_duration: "1M" aligns the cap to a billing month.

Then attach alert rules so the cap is not the first warning:

• 50% on budget_usage_percent >= 50.0 to a Slack channel, as an informational signal.
• 80% on budget_usage_percent >= 80.0 to the owning team, the point where someone should look.
• 100% on budget_usage_percent >= 100.0 to PagerDuty, since requests are now refused.

Set a cooldown long enough that a saturated budget does not notify on every sweep, then review the delivery record after a week to confirm rules fire where expected. For teams running this alongside routing policy, routing, fallback, and governance in Bifrost covers how the two interact, and token cost optimization with caching and smart routing covers the reduction side. Air-gapped and regulated deployments are covered on the Bifrost Enterprise page.

Frequently Asked Questions

How much does LLM cost?

LLM cost is charged per token, and rates differ by model and by whether the token is input or output. Frontier models typically cost ten to fifty times more per token than small models. Actual spend depends on token volume, model mix, and cache hit rate, which is why per-request cost records at the gateway are more useful than a headline rate.

How much is 1 million tokens in LLM?

One million tokens is the standard pricing unit for most providers, and rates span a wide range: small models are priced in cents per million input tokens, while frontier models run to several dollars per million, with output priced higher than input. Bifrost calculates cost from real-time model pricing rather than a static table.

Which LLM is most cost-effective?

The most cost-effective model is the smallest one that meets the quality bar for a given task, so the answer varies by task. Routing classification and extraction to small models while reserving large models for open-ended generation usually cuts spend more than any other single change, and a gateway makes that routing a configuration decision rather than a code change.

How to optimize LLM cost?

Optimize LLM cost in this order: remove repeated work with caching, route each task to the smallest adequate model, cap what each consumer can spend with budgets, and cap throughput with token rate limits. Attribution comes first, because optimization without per-team cost records is guesswork about where the spend originates.

What does LLM optimization mean?

LLM optimization covers reducing cost, latency, and token consumption without degrading output quality. It spans prompt and context reduction, model selection, caching, batching, and routing policy. At the gateway layer it also includes governance controls (budgets, rate limits, model whitelists) that keep those gains from being erased by unconstrained usage elsewhere.

Can an LLM gateway enforce a hard spending cap?

Yes. Bifrost checks every applicable budget before forwarding a request and refuses it when any one is exhausted, so the cap is enforced rather than reported. Budgets exist independently at the customer, team, virtual key, and provider config levels, and a temporary override can add capacity for a set number of cycles without changing the base limit.

Start Controlling LLM Costs with Bifrost

Controlling LLM costs is a governance problem before it is an optimization problem. Budgets decide what each team and customer may spend, budget alerts warn the owner while there is time to act, and request and token rate limits stop one misbehaving workload from consuming a shared allocation. Bifrost enforces all three inline, at 11 microseconds of overhead per request, and exports the cost records to the tooling a finance team already uses.

To see budgets, alert rules, and rate limits configured against your own traffic, book a demo with the Bifrost team, or start from the governance resources and run the gateway locally.