---
description: A step-by-step migration guide from LiteLLM to Bifrost, the AI gateway with native semantic caching, dual-layer hit matching, and zero application code changes.
title: "From LiteLLM to a Gateway with Native Semantic Caching: A Migration Guide"
image: https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/size/w1200/2026/07/from-litellm-to-a-gateway-with-native-semantic-caching-a-mig-bifrost-paper-waves.optimized.png
---

Try Bifrost Enterprise free for 14 days. [Request access](https://www.getmaxim.ai/#enterprise-trial)

**A step-by-step migration guide from LiteLLM to** [**Bifrost**](https://www.getmaxim.ai)**, the AI gateway with native semantic caching, dual-layer hit matching, and zero application code changes.**

Teams running LiteLLM in production often hit the same wall: response caching works in isolation, but wiring semantic caching into the proxy introduces Redis module dependencies, embedding configuration drift, and operational overhead that the proxy was supposed to absorb in the first place. This migration guide walks through moving from LiteLLM to [Bifrost](https://www.getmaxim.ai), the [open-source AI gateway](https://github.com/maximhq/bifrost) built by [Maxim AI](https://www.getmaxim.ai) with native semantic caching, dual-layer hit matching, and drop-in SDK compatibility. The goal is to keep application code unchanged while replacing the caching and routing layer with infrastructure that performs better at scale.

## Why Teams Are Migrating Away from LiteLLM

LiteLLM popularized the unified LLM API pattern, but production teams increasingly push against its architectural limits. The Python-based proxy adds measurable latency under load, and configuring semantic caching requires a specific Redis deployment (Redis Stack with the RediSearch module) that many cloud-managed Redis offerings do not ship by default. A recurring reported error, "redis semantic caching requires redis-py client >= 4.2.0 and the redisearch module to be loaded on redis-server," surfaces in Windows and managed Redis environments where the search module is not installed.

Beyond the semantic cache setup friction, LiteLLM's proxy architecture concentrates failure surface in a single Python process. LiteLLM's own high-availability documentation notes that production deployments handling 1000+ requests per second can experience PostgreSQL connection exhaustion and database deadlocks when running 10+ instances simultaneously updating the same user, team, or key records. Teams that scale past a single instance begin writing custom Redis transaction buffers, running side-car caches, and monitoring deadlock patterns, all work that should be handled by the gateway itself.

The migration rationale usually comes down to three factors:

- **Performance overhead**: Python proxies cannot match the throughput of a native Go gateway at the same infrastructure cost.
- **Caching setup complexity**: Redis Stack with RediSearch is a specific deployment requirement, not a commodity.
- **Single-process blast radius**: Scaling the proxy horizontally introduces coordination problems that require additional infrastructure.

## What Bifrost Changes About the Caching Layer

[Bifrost](https://www.getmaxim.ai) is built in Go and designed to operate as middleware between application code and 23+ LLM providers, with caching, routing, and governance as first-class capabilities rather than bolt-on plugins. The [semantic caching plugin](https://docs.getbifrost.ai/features/semantic-caching) ships with dual-layer matching: exact hash deduplication for identical requests and vector similarity search for semantically similar ones. Both layers can be active at the same time, or direct hash mode can run standalone when embedding calls are undesirable.

The key operational differences compared to a LiteLLM deployment:

- **Choice of vector store**: Bifrost's semantic cache works with Weaviate, Redis/Valkey-compatible endpoints, Qdrant, and Pinecone. Teams are not locked into Redis Stack.
- **Direct hash mode**: A fully embedding-free exact-match mode is available by setting `dimension: 1` with no embedding provider. This gives LiteLLM-style exact caching without any vector compute, and Redis/Valkey is the recommended backend.
- **Per-request cache control**: Cache behavior (type, TTL, threshold, no-store) is controlled per request via HTTP headers like `x-bf-cache-key`, `x-bf-cache-ttl`, and `x-bf-cache-threshold`, which keeps application-layer logic flexible without requiring redeploys.
- **Cache debug metadata**: Every cached response includes `cache_debug` fields with `hit_type`, `similarity`, and `cache_id`, making it straightforward to audit hit rates and clear specific entries.

Bifrost also publishes [independent performance benchmarks](https://www.getmaxim.ai/resources/benchmarks) showing approximately 11 microseconds of gateway overhead at 5,000 requests per second, which keeps caching decisions out of the latency budget for production workloads.

## Pre-Migration Checklist

Before starting the migration, inventory the LiteLLM features currently in use. The migration path differs depending on whether you rely on LiteLLM's SDK, its proxy server, or both. Bifrost supports three compatibility modes that cover most LiteLLM deployments:

- **Drop-in SDK replacement**: Keep using the OpenAI, Anthropic, or LiteLLM SDK. Change only the base URL to point to Bifrost. See the [drop-in replacement docs](https://docs.getbifrost.ai/features/drop-in-replacement) for the one-line change per SDK.
- **LiteLLM SDK passthrough**: Bifrost provides a dedicated [LiteLLM SDK integration](https://docs.getbifrost.ai/integrations/litellm-sdk) so existing `litellm.completion()` calls continue to work unchanged.
- **LiteLLM compatibility mode**: For text completion requests against chat-only models, Bifrost's [LiteLLM compatibility layer](https://docs.getbifrost.ai/features/litellm-compat) performs text-to-chat conversion transparently, preserving the `choices[0].text` response shape that legacy code expects.

Confirm which of these applies to your codebase. In most cases, migration touches configuration only, not application code.

## Step-by-Step Migration from LiteLLM to Bifrost

### Step 1: Deploy Bifrost Alongside LiteLLM

Run Bifrost in parallel with the existing LiteLLM proxy. This enables side-by-side traffic comparison without a cutover risk.

```bash
# Run Bifrost with zero configuration
npx -y @maximhq/bifrost

# Or with Docker
docker run -p 8080:8080 maximhq/bifrost
```

Bifrost starts with a web UI at `http://localhost:8080` for provider configuration. Add the same provider API keys that LiteLLM is using. Full setup details are covered in the [gateway setup guide](https://docs.getbifrost.ai/quickstart/gateway/setting-up).

### Step 2: Configure a Vector Store for Semantic Caching

Bifrost's semantic cache requires a vector store. For teams migrating from LiteLLM with an existing Redis deployment, Redis or Valkey is often the simplest path. The recommended Redis configuration:

```json
{
  "vector_store": {
    "enabled": true,
    "type": "redis",
    "config": {
      "addr": "localhost:6379"
    }
  }
}
```

For teams that want to avoid embedding API calls entirely (replicating LiteLLM's exact-match caching behavior), direct hash mode is the equivalent:

```json
{
  "plugins": [
    {
      "enabled": true,
      "name": "semantic_cache",
      "config": {
        "dimension": 1,
        "ttl": "5m",
        "cleanup_on_shutdown": true,
        "cache_by_model": true,
        "cache_by_provider": true
      }
    }
  ]
}
```

Omitting the `provider` field triggers direct-only mode with zero embedding overhead. This is the lowest-latency, lowest-cost configuration.

### Step 3: Enable Semantic Caching with Embeddings

For teams that want true semantic matching (cache hits on paraphrased queries, not just identical strings), configure the plugin with an embedding provider. The configuration mirrors LiteLLM's `redis-semantic` cache type but runs natively in the gateway:

```json
{
  "plugins": [
    {
      "enabled": true,
      "name": "semantic_cache",
      "config": {
        "provider": "openai",
        "embedding_model": "text-embedding-3-small",
        "dimension": 1536,
        "ttl": "5m",
        "threshold": 0.8,
        "conversation_history_threshold": 3,
        "cache_by_model": true,
        "cache_by_provider": true
      }
    }
  ]
}
```

The `threshold` parameter (0.8 by default) controls similarity strictness. This is directly analogous to LiteLLM's `similarity_threshold`, but Bifrost additionally supports per-request threshold overrides through the `x-bf-cache-threshold` header, which is useful for isolating high-precision endpoints (such as billing or compliance queries) from general-purpose ones.

### Step 4: Add the Cache Key on the Application Side

Bifrost's semantic cache only activates when a cache key is supplied in the request. This is a deliberate design choice that prevents unintended caching of sensitive or session-specific content. Update the application to include the header:

```bash
curl -H "x-bf-cache-key: session-123" \\
     -H "Content-Type: application/json" \\
     -d '{"model": "openai/gpt-4o-mini", "messages": [...]}' \\
     <http://localhost:8080/v1/chat/completions>
```

For SDK-based applications, set the header via the existing request options. This is the only application-side change typically required.

### Step 5: Point Application Traffic at Bifrost

Change the base URL in the SDK configuration from the LiteLLM proxy endpoint to the Bifrost endpoint. For teams using the OpenAI SDK, this is a single line:

```python
from openai import OpenAI

client = OpenAI(
    base_url="<http://localhost:8080/v1>",  # Bifrost gateway
    api_key="any-value"  # Bifrost handles provider auth
)
```

Run a percentage of production traffic through Bifrost first to validate cache hit rates, latency, and error handling before a full cutover. Bifrost's built-in [telemetry](https://docs.getbifrost.ai/features/telemetry) exposes Prometheus metrics and OpenTelemetry traces, so existing Grafana or Datadog dashboards can ingest Bifrost data directly.

### Step 6: Validate Cache Hits and Tune Thresholds

Every cached response from Bifrost returns `extra_fields.cache_debug` metadata with hit type, similarity score, and cache ID. Use this to audit the migration:

```json
{
  "extra_fields": {
    "cache_debug": {
      "cache_hit": true,
      "hit_type": "semantic",
      "similarity": 0.95,
      "threshold": 0.8
    }
  }
}
```

Monitor hit rates per endpoint and adjust the threshold and `conversation_history_threshold` settings accordingly. A threshold between 0.85 and 0.92 is typical for customer-facing applications where precision matters more than recall.

## What Else You Get Beyond the Cache

Migration to Bifrost unlocks capabilities beyond caching that were either missing or required separate tools in a LiteLLM deployment. [Automatic failover](https://docs.getbifrost.ai/features/fallbacks) across providers activates when a primary provider returns errors or times out, with zero application code changes. [Virtual keys](https://docs.getbifrost.ai/features/governance/virtual-keys) provide per-team budgets, rate limits, and access control as a first-class governance layer. And the [MCP gateway](https://docs.getbifrost.ai/mcp/overview) centralizes tool connections, OAuth, and authorization for agentic workloads, which is the next phase of infrastructure that LiteLLM does not natively address. [MCP code mode](https://docs.getbifrost.ai/mcp/code-mode) reduces token usage by 50%+ when using multiple MCP servers.

Teams specifically evaluating alternatives can reference [Bifrost as a drop-in LiteLLM alternative](https://www.getmaxim.ai/bifrost/alternatives/litellm-alternatives) for a full feature comparison, and the [LLM Gateway Buyer's Guide](https://www.getmaxim.ai/resources/buyers-guide) covers evaluation criteria beyond caching. External industry data tracks the broader context as well: LLM API spending roughly doubled from $3.5 billion to $8.4 billion between late 2024 and mid-2025, with 72% of organizations planning further AI budget increases in 2026, which makes gateway-level cost optimization (caching, routing, governance) a strategic investment rather than a micro-optimization.

## Common Migration Pitfalls to Avoid

A few patterns come up repeatedly when teams migrate gateways:

- **Forgetting the cache key header**: Bifrost intentionally does not cache requests without `x-bf-cache-key`. If hit rates are unexpectedly zero, check that the header is being forwarded by the application.
- **Mismatched cache dimensions**: If the embedding model or dimension changes after initial deployment, existing entries in the vector store become unreadable. Either set `cleanup_on_shutdown: true` or use a new `vector_store_namespace` when changing dimensions.
- **Over-threshold tuning**: Setting the similarity threshold too low (below 0.75) produces false-positive cache hits on semantically related but materially different queries. Start at 0.8 and narrow down from observed traffic.
- **Mixing direct and semantic modes**: The `x-bf-cache-type` header only has effect when the plugin is initialized in dual-layer mode (with an embedding provider). In direct-only mode, all requests use hash matching regardless of the header.

## Migrate to Bifrost Today

Migration from LiteLLM to Bifrost is primarily a configuration change. Application code stays put, the cache layer becomes dual-layer and vector-store-flexible, and the gateway adds failover, governance, and MCP infrastructure that LiteLLM does not provide. The [migration from LiteLLM guide](https://www.getmaxim.ai/resources/migrating-from-litellm) covers additional edge cases, and the [Bifrost GitHub repo](https://github.com/maximhq/bifrost) has a one-command install path for a local evaluation.

To see how Bifrost's native semantic caching and LiteLLM compatibility apply to your production workload, [book a demo](https://www.getmaxim.ai/book-a-demo) with the Bifrost team.

## Read next

[![What Is an Agent Harness? Harness vs Framework and Governance](https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/size/w720/2026/10/what-is-an-agent-harness-harness-vs-framework-and-governance-bifrost-1920x1080.png) An agent harness is the software around a model that runs the agent loop, executes tools, and enforces permissions. This guide compares agent harnesses with frameworks, SDKs, and runtimes, and shows which governance controls belong in the harness and which belong in an AI gateway.](https://www.getmaxim.ai/articles/what-is-an-agent-harness-harness-vs-framework-and-governance/)

[![OpenClaw vs Hermes: Which Agent to Run and How to Control It](https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/size/w720/2026/10/openclaw-vs-hermes-which-agent-to-run-and-how-to-control-it-bifrost-1920x1080.png) OpenClaw and Hermes Agent are open-source, self-hosted AI agents that run on your own hardware and reach you through chat apps. This openclaw vs hermes guide compares how each is built and secured, says which to run for which job, and shows how to cap its models, spend, and MCP tools.](https://www.getmaxim.ai/articles/openclaw-vs-hermes-which-agent-to-run-and-how-to-control-it/)

[![AnythingLLM With Every Model: Connect and Govern It Through One Gateway](https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/size/w720/2026/10/anythingllm-with-every-model-connect-and-govern-it-through-o-bifrost-1920x1080.png) AnythingLLM is an open-source AI application for document chat, agents, and team workspaces. This guide shows how to route AnythingLLM through Bifrost so every workspace reaches any model through one endpoint, with budgets, rate limits, failover, and request logs.](https://www.getmaxim.ai/articles/anythingllm-with-every-model-connect-and-govern-it-through-one-gateway/)

## More in AI Gateway

[![Smart LLM Routing: Picking the Optimal Model per Request](https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/size/w720/2026/07/smart-llm-routing-picking-the-optimal-model-per-request-bifrost-paper-waves.optimized.png) Smart LLM routing in production cuts cost and latency by matching each request to the right model. Learn how Bifrost routes at the gateway layer. Every production AI application eventually hits the same wall: the team picked one default model, costs keep climbing, and latency on simple requests is indistinguishable](https://www.getmaxim.ai/articles/smart-llm-routing-picking-the-optimal-model-per-request/)

[![Claude Opus 4.5 vs GPT 5.2: Which AI Model Leads in 2026?](https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/size/w720/2026/07/claude-opus-4-5-vs-gpt-5-2-which-ai-model-leads-in-2026-bifrost-circuit.optimized.png) TL;DR: Claude Opus 4.5 and GPT 5.2 represent the cutting edge of AI capabilities in early 2026, each excelling in distinct areas. Claude Opus 4.5 dominates software engineering with 80.9% on SWE-bench Verified and superior long-horizon coding tasks, while GPT 5.2 excels](https://www.getmaxim.ai/articles/claude-opus-4-5-vs-gpt-5-2-which-ai-model-leads-in-2026/)

```json
{
    "@context": "https://schema.org",
    "@type": "Article",
    "publisher": {
        "@type": "Organization",
        "name": "Maxim Articles",
        "url": "https://www.getmaxim.ai/articles/",
        "logo": {
            "@type": "ImageObject",
            "url": "https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/size/w256h256/2025/08/thumbnail.png",
            "width": 60,
            "height": 60
        }
    },
    "author": {
        "@type": "Person",
        "name": "Kamya Shah",
        "image": {
            "@type": "ImageObject",
            "url": "https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/2025/09/WhatsApp-Image-2025-08-29-at-17.40.40-1.jpeg",
            "width": 1200,
            "height": 1600
        },
        "url": "https://www.getmaxim.ai/articles/author/kamya/",
        "sameAs": []
    },
    "headline": "From LiteLLM to a Gateway with Native Semantic Caching: A Migration Guide",
    "url": "https://www.getmaxim.ai/articles/from-litellm-to-a-gateway-with-native-semantic-caching-a-migration-guide/",
    "datePublished": "2026-04-21T20:51:00.000Z",
    "dateModified": "2026-10-13T11:03:12.000Z",
    "image": {
        "@type": "ImageObject",
        "url": "https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/size/w1200/2026/07/from-litellm-to-a-gateway-with-native-semantic-caching-a-mig-bifrost-paper-waves.optimized.png",
        "width": 1200,
        "height": 630
    },
    "keywords": "AI Gateway",
    "description": "A step-by-step migration guide from LiteLLM to Bifrost, the AI gateway with native semantic caching, dual-layer hit matching, and zero application code changes.\n\nTeams running LiteLLM in production often hit the same wall: response caching works in isolation, but wiring semantic caching into the proxy introduces Redis module dependencies, embedding configuration drift, and operational overhead that the proxy was supposed to absorb in the first place. This migration guide walks through moving fro",
    "mainEntityOfPage": "https://www.getmaxim.ai/articles/from-litellm-to-a-gateway-with-native-semantic-caching-a-migration-guide/"
}
```