LLM Gateway: What It Is and How It Works

An LLM gateway is one API in front of every model provider. See how it routes requests, fails over, redacts PII, enforces budgets and differs from API gateways.

9 min read

LLM Gateway: What It Is and How It Works | Bifrost

An LLM gateway is a middleware layer between applications and large language model providers that exposes one API for every model, then authenticates, routes, retries, inspects and logs each request on the way through. The gateway lets a team change providers, add failover, or enforce budgets and guardrails in one place instead of in the code of every application.

Production applications rarely call one provider for long. Anthropic's API returns a 529 overloaded_error when its servers are under load, and OpenAI caps every account in requests and tokens per minute, so a second provider becomes a reliability requirement.

Bifrost, an open-source AI gateway, sits in that path as a self-hosted gateway, inside your own infrastructure.

How does an LLM gateway work?

An LLM gateway works as a reverse proxy that understands model requests. The application sends a request in one format to one endpoint; the gateway identifies the caller, applies policy, picks a provider and key, translates the request into that provider's API, and returns the answer in the format the application expects.

An application calls an LLM gateway that authenticates, applies policy and guardrails, checks the cache, routes to OpenAI, and fails over to Anthropic or AWS Bedrock

Figure 1: The application holds one gateway credential; the gateway holds the provider keys and decides where each request goes.

Each request passes through the same stages:

  1. Authenticate. The caller presents a gateway key, not a provider key, and the gateway resolves it to a user, team or service.
  2. Apply policy. Budgets, rate limits and the list of allowed models for that identity are checked before any tokens are spent.
  3. Inspect input. Guardrails scan the prompt for personal data, secrets or prompt injection, then block, redact or pass it.
  4. Check the cache. An identical or semantically similar earlier request can be answered without calling a provider.
  5. Route and translate. The gateway chooses a provider, model and key, converts the request, and retries or fails over on errors.
  6. Inspect output and log. The response is scanned, then tokens, cost, latency and the serving provider are recorded.

Every stage adds latency, so the measure that matters is overhead: total latency minus the time spent waiting on the provider. A gateway that keeps policy and keys in memory adds microseconds; one that queries a database per request adds milliseconds. A PII check with Microsoft Presidio, for example, adds one analyzer call before the prompt can leave.

How an LLM gateway unifies provider APIs

An LLM gateway unifies provider APIs by accepting one request schema, usually OpenAI's Chat Completions format, and translating it into each provider's native API. Model choice moves into a parameter such as anthropic/claude-sonnet-4, so switching providers means changing a string, not rewriting the client.

The translation is more than renaming fields. Anthropic's Messages API takes the system prompt as a top-level field and requires max_tokens, while OpenAI's API places the system prompt in the message list. Tool-call schemas, streaming event formats, token usage fields and error bodies also differ by provider. The gateway normalizes all of them, so application code reads one response shape and one error shape whichever model answered.

Many gateways also expose each provider's native route, such as /anthropic alongside /openai, so existing SDKs only change their base URL. Configuring several providers behind one gateway shows that setup end to end.

Fallbacks, retries and automatic failover

An LLM gateway handles fallbacks and retries by classifying each failed attempt. Transient errors are retried on the same provider with exponential backoff; a rate-limited or rejected key is swapped for another key; and when a provider stays down, automatic failover sends the request to the next model in a fallback chain.

FailureTypical statusRetry same key?Rotate to another key?Fail over to next provider?
Network error or provider outage500, 502, 503, 504, Anthropic 529Yes, with backoffNoYes, after retries run out
Rate limit429, often with Retry-AfterAfter backoffYesYes, after retries run out
Rejected key or exhausted quota401, 402, 403NoYes, immediatelyYes, if no key works
Invalid request400, 413, 422NoNoNo; the same request fails everywhere

HTTP 429 and its Retry-After header are defined in RFC 6585. Backoff should be capped and jittered, so many clients retrying together do not hit the recovering provider in the same second.

Fallback chains carry two costs. Every hop adds latency, and a fallback model can produce different output for the same prompt, so chains should list models with comparable quality and the response should say which model served it. A walkthrough of automatic fallback shows the configuration in practice.

Guardrails and PII redaction at the gateway

An LLM gateway enforces guardrails by running checks on the prompt before it reaches a provider and on the response before it reaches the user. Checks detect personal data, leaked secrets, prompt injection and unsafe content, then block the request, redact the matched spans, or log a finding.

The OWASP Top 10 for LLM Applications lists prompt injection (LLM01) and sensitive information disclosure (LLM02) as its first two risks, and both are visible at the gateway because every prompt and response passes through it. PII redaction replaces names, account numbers or email addresses with placeholders before the text leaves your network, so the provider never stores them.

Enforcing guardrails centrally means one policy for every application instead of a library in each codebase. The trade-offs are latency, since a remote classifier adds a network round trip per check, and streaming, where output checks must buffer chunks or run after the stream closes. Guardrails at the gateway compares built-in checks with external providers.

Access control, budgets and caching

An LLM gateway controls access by issuing its own keys to people, teams and services, then mapping each key to allowed models, a token or spend budget, and rate limits. Provider keys stay inside the gateway, so revoking one person's access never means rotating a key shared by a whole team.

In larger organizations the gateway connects to the identity provider. OIDC (OpenID Connect) signs users in with their corporate account, and SCIM (System for Cross-domain Identity Management) pushes user and group changes from the directory, so a person removed in Okta or Microsoft Entra ID loses model access without a manual step.

Because every token passes through one place, the gateway can attribute spend per key, team or customer, and stop a runaway agent when its budget is gone. Caching lowers the same spend: exact-match caching replays identical requests, while semantic caching answers prompts that are close in meaning, with a similarity threshold that trades hit rate against wrong answers. Bifrost's governance controls cover these layers.

LLM gateway vs API gateway

An LLM gateway differs from an API gateway in what it understands. An API gateway routes HTTP requests by host, path and method and counts requests; an LLM gateway reads the model request itself, so it can count tokens, translate between providers, fail over to another model, and scan prompts and responses.

DimensionAPI gatewayLLM gateway
Routes byHost, path, methodModel name, provider health, cost, policy
UpstreamsYour own servicesExternal and self-hosted model providers
Rate limits inRequests per secondRequests and tokens per minute
Cost unitRequestInput and output tokens
Failure handlingRetry the same upstreamRotate keys, fail over to another model
Payload inspectionHeaders, schema validationPrompt and response content
StreamingPass-throughParse server-sent events per provider

The two often run together: the API gateway at the network edge and the model gateway behind it. An MCP gateway plays the same role for tool calls that agents make to MCP servers.

Build, self-host or buy an LLM gateway?

Whether to build, self-host or buy a model gateway depends on where prompts are allowed to go, how much latency the product can absorb, and how many teams will use it. Building suits one service with one or two providers; a self-hosted gateway suits regulated data; a managed gateway suits teams that accept a third party in the request path.

OptionPrompts and keysWhat you maintainFits
Build in-houseYour networkProvider adapters, retries, budgets, logsOne application, few providers
Self-host an open-source LLM gatewayYour networkDeployment, upgrades, scalingRegulated data, many teams
Managed gateway servicePass through a vendorConfiguration onlyTeams that allow third-party processing

Built gateways usually start as a retry wrapper and grow into provider adapters, key rotation and spend tracking that someone must update whenever a provider changes its API. Measure overhead at p99 under your own traffic before choosing. The LLM gateway buyer's guide lists the criteria enterprise teams weigh.

LLM gateway with Bifrost

Bifrost, the open-source AI gateway built by Maxim AI, is an LLM gateway that runs as a single binary inside your infrastructure and routes to 10,000+ models across 25+ providers through one OpenAI-compatible API. It adds about 11 microseconds of overhead per request at 5,000 RPS.

Applications using the OpenAI and Anthropic SDKs and coding agents call Bifrost, which applies virtual keys, fallbacks and guardrails before routing to model providers

Figure 2: One Bifrost deployment holds the provider keys, policies and logs for every application.

Existing code moves over as a drop-in replacement: the OpenAI, Anthropic, Google GenAI, LangChain and LiteLLM SDKs change only their base URL and send a Bifrost virtual key. A request can carry a fallbacks list of provider/model entries; each provider in the chain gets its own retry budget, with backoff starting at 500 ms and capped at 5 seconds, as set out in retries and fallbacks. Within a provider, weighted key load balancing spreads traffic across keys and rotates away from a key that returns 401, or 429 when retries are enabled.

Budgets and rate limits apply at the customer, team, virtual key and provider level. Semantic caching combines exact-hash and similarity matching on Redis, Weaviate, Qdrant or Pinecone.

Bifrost Enterprise adds guardrails for LLM and MCP traffic, including Presidio and AWS Bedrock Guardrails, plus OIDC sign-in and inbound SCIM 2.0 provisioning.

Every log entry splits latency into upstream time and Bifrost's own overhead, with a per-stage overhead breakdown for each request.

To run it locally in under a minute, start with the gateway quickstart.

Frequently asked questions

What is an LLM gateway?

An LLM gateway is a service between your applications and model providers such as OpenAI and Anthropic. Applications call one API; the gateway authenticates the caller, applies budgets and guardrails, routes the request to a provider, fails over when that provider errors, and logs tokens, cost and latency for every call.

Is there an open-source LLM gateway?

Yes. Several open-source gateways exist, and they can run on your own servers so prompts and provider keys never leave your network. Bifrost is an open-source LLM gateway licensed under Apache 2.0, written in Go, and deployable with npx, Docker, Kubernetes or a single binary. Enterprise features such as clustering and guardrails are a paid tier.

What is the difference between an LLM gateway and an LLM proxy?

An LLM proxy forwards requests to a model provider, sometimes translating formats, with little else. A gateway adds the controls a shared service needs: per-caller keys, budgets, rate limits, guardrails, caching, failover across providers and request logs. Many products start as a proxy and add these controls over time.

Does an LLM gateway add latency?

Yes, but the amount depends on the design. A gateway written in a compiled language that keeps configuration in memory adds microseconds per request, which is small next to provider response times measured in hundreds of milliseconds. Remote guardrail checks and semantic cache lookups add more, because each one makes its own network call.

Do I need an LLM gateway if I only use one provider?

A gateway is still useful with one provider when several teams share it. It keeps the provider key out of application code, gives each team its own budget and rate limit, rotates across multiple keys, caches repeated prompts and logs usage. Adding a second provider later then becomes a configuration change.

[ Keep reading ]

More on AI Gateway

View all →