5 Best Practices to Optimize Your LLM Costs in Production

Optimize LLM costs in production with five gateway-level practices: semantic caching, model routing, MCP Code Mode, virtual keys, and observability.

LLM API spend is now one of the fastest-growing line items in enterprise infrastructure budgets, and most teams only discover the scale of the problem after the invoice arrives. A single customer-support agent handling ten thousand daily conversations can cost over $7,500 per month on a flagship model, and at 500+ MCP tools an unoptimized agent run can hit $377 per benchmark round before the user prompt is even processed. The good news: teams that optimize LLM costs at the infrastructure layer routinely cut total spend by 40% to 80% without sacrificing output quality. This post walks through five best practices to optimize LLM costs in production, each one implementable through Bifrost, the open-source AI gateway built by Maxim AI.

Why LLM Cost Optimization Belongs at the Gateway Layer

LLM costs accumulate from many sources, raw input and output tokens, retries, embedding calls, tool definitions injected into context, verbose system prompts, and routing inefficiencies. A 2025 industry analysis found that 40% to 60% of LLM budgets go to operational inefficiencies rather than necessary model usage. Solving this in application code means duplicating the same logic across every service that calls a model. Solving it at the gateway layer means every consumer of the gateway inherits the same optimizations with no code changes.

A gateway-level approach to LLM cost optimization centralizes five things: caching, routing, agent token management, governance, and observability. Bifrost handles all five behind a single OpenAI-compatible API that adds only 11 microseconds of overhead per request at 5,000 RPS. The five practices below map directly to those five capabilities.

1. Use Semantic Caching to Eliminate Redundant LLM Calls

The cheapest LLM call is the one you never make. Production traffic is full of semantically identical queries phrased differently: "what is your refund policy," "how do I get a refund," and "can I return this order" all expect the same answer. Exact-match caches miss every one of them.

Bifrost ships semantic caching as a first-class gateway feature. Each incoming prompt is embedded into a vector and compared against previously cached prompts using cosine similarity. When the score crosses a configurable threshold, Bifrost returns the cached response without round-tripping to the provider. The cache is dual-layer: an exact-match hash layer catches identical requests at sub-millisecond latency, and the semantic layer catches similar ones with vector search.

Reported impact for cache-friendly workloads:

• Up to 70% reduction in LLM operational costs and latency on workloads with strong semantic repetition.
• Cache hit rates of 60% to 85% in customer-support and FAQ-style applications.
• Sub-millisecond cache retrieval versus multi-second provider calls on cache hits.
• Direct hash mode is available for teams that want exact-match deduplication without an embedding provider, with zero embedding overhead.

Semantic caching is the single highest-ROI optimization for any application where users frequently ask variations of the same question.

2. Route Requests to the Cheapest Capable Model

The second-largest source of waste is using a flagship model for tasks a smaller model handles equally well. Independent research benchmarks show that classifier-based and cascade routers can match best-single-model quality while reducing average inference cost by 20% to 60%. The pattern is straightforward: classify the request by complexity, route simple tasks to a smaller model, and reserve frontier models for genuine reasoning.

Bifrost implements cost-aware routing through provider routing rules that evaluate request attributes and dispatch each call to the most appropriate provider and model. Routing rules in Bifrost support:

• Conditional rules: route by request type, header value, or budget consumption (for example, redirect embeddings to a lower-cost endpoint when a budget threshold is crossed).
• Tier-based routing: send premium users to high-quality models and standard users to cost-efficient alternatives based on request headers.
• Probabilistic A/B splits: send 70% of traffic to one provider and 30% to a cheaper alternative to test cost-quality tradeoffs without committing fully.
• Fallback chains: automatically retry on a backup provider when the primary fails, so you never pay for a failed request and then pay again for an application-level retry.
• Weighted load balancing: distribute requests across multiple API keys and providers to avoid the cascade of retries that inflates costs during rate-limit events.

Routing decisions execute inline at the gateway with negligible overhead, so the optimization is transparent to application code.

3. Cut Agent Token Costs With MCP Code Mode

The fastest-growing category of LLM cost in 2026 is agent infrastructure. Every Model Context Protocol server connected to an agent injects its full tool catalog into the model's context on every single turn. With 16 servers and roughly 500 tools, an agent can spend over 1.15 million input tokens per query just on tool definitions, before the user prompt is even read. At that scale, tool schemas, not user content, become the majority of the bill.

Bifrost solves this with MCP Code Mode, a gateway-level execution path that exposes connected MCP servers as a virtual filesystem of lightweight Python stubs. The model reads only the stubs it needs, writes a short Python script to chain calls together, and Bifrost executes that script in a sandboxed Starlark interpreter. Only the final result returns to the model. This pattern is conceptually similar to the code-execution-with-MCP approach published by Anthropic's engineering team, where one Google Drive to Salesforce workflow dropped from 150,000 tokens to 2,000.

Bifrost's published benchmarks across three rounds of controlled scaling tests show:

• 58% input token reduction at 96 connected tools.
• 84% input token reduction at 251 connected tools.
• 92.8% input token reduction at 508 connected tools, dropping cost from $377 to $29 per benchmark round.
• 100% pass rate maintained across all rounds, with no accuracy tradeoff.
• 30% to 40% latency improvement on multi-step workflows because orchestration runs server-side instead of round-tripping through the model.

Code Mode is a single per-client toggle, not a rewrite. For a deeper architectural breakdown, the Bifrost MCP Gateway resource page and the companion post on access control, cost governance, and 92% lower token costs at scale cover the full design.

4. Enforce Hierarchical Budgets and Rate Limits With Virtual Keys

Optimization controls cost per request. Governance controls total exposure. Without hard ceilings, a single misconfigured agent loop or a runaway Full Auto coding session can burn through a monthly budget in a weekend.

Bifrost's virtual keys are the primary governance entity in the gateway. Each virtual key carries its own budget, rate limits, and model access policy, and budgets cascade through a four-tier hierarchy: Customer, Team, Virtual Key, and Provider Configuration. When any tier is exhausted, the request is blocked before it incurs cost.

Concrete patterns this unlocks:

• Per-team monthly caps: the frontend team gets $500 per month, the platform team gets $1,000, and the gateway enforces both automatically.
• Per-customer cost isolation: SaaS companies can assign each tenant a budget-limited key, preventing any single customer from generating runaway costs.
• Calendar-aligned reset windows: budgets reset at midnight UTC, weekly Monday 00:00 UTC, monthly on the first, or yearly on January 1.
• Dual-axis rate limits: request limits (calls per minute or hour) and token limits (prompt plus completion tokens per window) operate in parallel at virtual key and provider levels.
• Model access rules: a contractor key can be locked to open-source models, while a senior engineer's key permits flagship reasoning models.

The full design surface, including hierarchical rollups and per-tool cost attribution, is documented on the Bifrost governance resource page.

5. Build Closed-Loop Observability for Cost Attribution

You cannot optimize what you cannot measure. Most teams know their total LLM bill but cannot answer which feature, which user, or which prompt is driving it. That visibility gap is the reason "we spent a week shortening prompts" rarely produces durable savings.

Bifrost's observability stack captures every request with full metadata: provider, model, virtual key, token counts (input, output, cached), latency, cache hit status, and resolved cost in dollars. Telemetry is exposed natively through Prometheus metrics and OpenTelemetry traces, and the built-in dashboard surfaces cost per endpoint, per user, per model, and per virtual key without a custom exporter. This pairs well with provider-native prompt caching from OpenAI and Anthropic, where Bifrost's logs make it possible to verify that cache discounts are actually landing.

Patterns to operationalize:

• Identify the top three features by token spend and ask whether each one needs a flagship model.
• Track cache hit rate per route; rates below 30% signal a tuning opportunity on the similarity threshold.
• Watch failover frequency per provider; sustained failovers indicate a routing rule that needs rebalancing.
• Set Slack, PagerDuty, or webhook alerts at 80% of monthly budget so policy conversations happen before the cap is hit, not after.

Closed-loop observability is what turns the previous four practices from one-time wins into a continuous cost program.

Compounding the Five Practices

The five practices stack. Semantic caching removes redundant calls. Routing sends the surviving calls to the cheapest capable model. Code Mode collapses agent context for tool-heavy workflows. Virtual keys cap total exposure. Observability closes the loop. In aggregate, production teams running this combination through Bifrost report 40% to 60% reductions in total LLM spend on typical applications, with up to 92% reduction on agent workloads heavy in MCP tools.

Start Optimizing LLM Costs With Bifrost

Reducing LLM costs in production is an infrastructure problem, not a prompt-tuning exercise. Bifrost gives platform teams a single open-source gateway that combines semantic caching, cost-aware routing, MCP Code Mode, virtual key governance, and native observability behind one OpenAI-compatible API. The full deployment runs in a single command, and every feature described here is available out of the box.

To see how Bifrost can optimize LLM costs across your production stack, book a Bifrost demo with the team for a walkthrough tailored to your architecture.