---
description: What is an AI gateway? It routes, governs, and observes every LLM call from one entry point. How it works inside, when you need one, and how to set one up.
title: "AI Gateway Explained: What It Is and How It Works"
image: https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/size/w1200/2026/08/ai-gateway-explained-what-it-is-and-how-it-works-bifrost-isometric.png
---

Try Bifrost Enterprise free for 14 days. [Request access](https://www.getmaxim.ai/#enterprise-trial)

## TL;DR

- An AI gateway is a service that sits between your applications and every LLM provider they call, exposing one API while applying routing, failover, cost control, and observability to every request.
- You need one once more than one team calls models or provider rate limits start throttling production; rate limits are usually the forcing function.
- Internally it authenticates the caller, applies governance, routes and caches, calls the provider with automatic failover, and logs the result; the overhead that path adds is the number that decides whether it belongs on the hot path.
- At enterprise scale it adds SSO, role-based access, signed audit logs, and clustering; governance, not throughput, is usually what pushes teams over the line.
- Bifrost delivers all of it at a [benchmarked 11 microseconds of overhead](https://www.getmaxim.ai/resources/benchmarks), open source and self-hostable across 10,000+ models; the [LLM gateway buyer's guide](https://www.getmaxim.ai/resources/buyers-guide) covers what to evaluate.

An AI gateway is a service that sits between your applications and every LLM provider they call, exposing one API for all of them while applying routing, failover, cost controls, and observability to every request. [Bifrost](https://www.getmaxim.ai), the [open-source AI gateway](https://github.com/maximhq/bifrost) built in Go by Maxim AI, is the best overall choice for enterprise teams running mission-critical AI workloads that require best-in-class performance, scalability, and reliability. This post covers what an [AI gateway](https://www.getmaxim.ai/llm-gateway) is, when you need one, how it works internally, what changes at enterprise scale, and how to set one up.

## What an AI Gateway Does

An AI gateway unifies access to multiple LLM providers behind a single API and enforces reliability, cost, and access policies on the traffic passing through it. Applications call the gateway instead of calling OpenAI, Anthropic, or AWS Bedrock directly, so switching providers or adding a model never requires an application change.

![Three applications call the Bifrost AI gateway through one API, and the gateway routes each request to OpenAI, Anthropic, AWS Bedrock, or a self-hosted model](https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/2026/10/ai-gateway-explained-what-it-is-and-how-it-works-one-api.svg)

*Figure 1: Applications speak one protocol, so adding a provider or switching models never touches application code.*

It does five things on every request:

- **Routing:** directs the request to a provider and model based on configured rules
- **Reliability:** retries transient failures and fails over to a backup provider when one becomes unavailable
- **Cost control:** enforces budgets and rate limits per team, project, or consumer
- **Caching:** replays stored responses for identical or similar prompts instead of paying for a new call
- **Observability:** records latency, token usage, and cost with attribution to the caller

An AI gateway is not an API gateway with a new label. The two solve different problems, and the differences decide which one belongs in front of model traffic:

| Concern | API gateway | AI gateway |
| --- | --- | --- |
| Traffic direction | Inbound, to services you operate | Outbound, to third-party model providers |
| Unit of cost | Requests | Input and output tokens, priced per model |
| Failure handling | Retry or return an error | Retry, then fail over to another provider or model |
| Caching | Exact-match on URL and headers | Exact-match plus semantic similarity on prompts |
| Content policy | Schema and rate validation | Prompt and completion screening for PII, secrets, and injection |
| Agent traffic | Not modeled | MCP tool calls governed alongside model calls |

A fuller breakdown of where each fits is in [AI gateway vs API gateway, with five options compared](https://www.getmaxim.ai/articles/ai-gateway-vs-api-gateway-5-options-compared-for-llm-traffic-in-2026/).

The alternative is provider logic scattered across services. Each team wires its own SDK, holds its own copy of the provider keys, implements its own retry behavior, and reports its own spend, which means no single system can answer what the organization spends on AI or which models it is actually running. Bifrost consolidates that into one layer, letting teams [reach GPT, Gemini, Claude, and Mistral through a single gateway](https://www.getmaxim.ai/bifrost/blog/access-gpt-gemini-claude-mistral-etc-through-1-gateway-configure-providers-in-bifrost/) across 25+ providers and 10,000+ models.

## When You Actually Need One

You need an AI gateway once more than one team calls more than one model in production. Below that threshold an SDK and an environment variable are enough. Above it, three questions get expensive to answer any other way: which team spent what, which model served which request, and what happens when a provider starts returning 429s.

![A decision flow asking whether more than one team calls models and whether audit is required, ending in provider SDK, self-hosted gateway, or enterprise deployment](https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/2026/10/ai-gateway-explained-what-it-is-and-how-it-works-when-needed.svg)

*Figure 2: Any one of the three signals is enough; compliance requirements usually decide the enterprise tier.*

Provider [rate limits](https://developers.openai.com/api/docs/guides/rate-limits) are usually the forcing function. Handling them properly requires a pool of API keys, rotation logic on per-key failures, and a fallback chain to a second provider, and nobody wants to maintain that in five separate services. The second forcing function is a finance question nobody can answer, which tends to arrive after the first production deployment. The failure modes behind both are covered in this guide to [how an AI gateway handles LLM rate limits and outages](https://www.getmaxim.ai/articles/how-an-ai-gateway-tackles-llm-rate-limits-and-outages/).

## How an AI Gateway Works

An AI gateway terminates the client request, authenticates the caller, selects a provider and key, forwards the request upstream, then returns the response while recording usage. All provider-specific work happens inside the gateway, so the client only ever speaks one protocol.

![A request is authenticated by virtual key, checked against the cache, routed to a provider and key, retried or failed over on error, then recorded with tokens and cost](https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/2026/10/ai-gateway-explained-what-it-is-and-how-it-works-five-stages.svg)

*Figure 3: Every stage runs on each request, which is why per-request overhead decides whether a gateway belongs on the hot path.*

In [Bifrost](https://www.getmaxim.ai) that path runs through five stages:

1. **Authenticate the caller.** A [virtual key](https://docs.getbifrost.ai/features/governance/virtual-keys) identifies the consumer and carries its permissions, budget, and rate limits. Provider credentials stay in the gateway and are never distributed to clients.
2. **Check the cache.** [Caching](https://docs.getbifrost.ai/features/semantic-caching) tries an exact-match hash lookup first, then an embedding-based similarity lookup on a miss. A direct hit is a single vector-store round-trip, sub-millisecond against a local store, while a semantic lookup must embed the request first, which costs an embedding call of tens to a few hundred milliseconds. Both still sit well under a multi-second provider call, but only the direct path is effectively free. Tuning the similarity threshold is its own problem: set it too loose and the cache answers questions it was not asked.
3. **Select a provider and key.** Weighted load balancing distributes requests across an API key pool with model-specific filtering, and adaptive load balancing goes further by scoring provider health rather than routing on latency alone.
4. **Handle failure.** Retries and fallbacks work as two layers. Transient 5xx errors retry against the same provider with exponential backoff and jitter, while 429 and auth failures rotate to a different key. Once retries are exhausted the request moves to the next provider in the fallback chain, which gets its own full retry budget. [Configuring that chain](https://www.getmaxim.ai/bifrost/blog/your-primary-llm-provider-failed-enable-automatic-fallback-with-bifrost/) is a per-provider setting, not application code.
5. **Record and return.** Latency, tokens, and cost are attributed to the virtual key and exported to Prometheus or OpenTelemetry.

The overhead this adds is the number that decides whether a gateway can sit in the production path at all. Bifrost adds 11 microseconds per request at 5,000 requests per second in [sustained benchmarks](https://www.getmaxim.ai/resources/benchmarks), which keeps the gateway well below the noise floor of any provider call it fronts.

## Running an AI Gateway at Enterprise Scale

An enterprise AI gateway adds identity integration, role-based access control, audit trails, content guardrails, and high-availability deployment on top of routing and governance. The difference is not feature count but blast radius. At this scale the gateway is a compliance boundary, and a policy gap or an outage reaches every AI application in the organization at once.

What a single-node deployment does not give you:

- **Identity and access:** OIDC single sign-on with Okta, Microsoft Entra, Keycloak, or Google Workspace, plus role-based access control with custom roles, sitting alongside the [governance and observability layer](https://www.getmaxim.ai/bifrost/blog/ai-governance-and-observability-in-bifrost/)
- **Auditability:** [audit logs](https://docs.getbifrost.ai/enterprise/audit-logs) of administrative activity that can be signed with an HMAC key, retained for a configurable period, and archived to S3 or GCS for SOC 2, GDPR, HIPAA, and ISO 27001 evidence
- **Content safety:** [guardrails at the gateway](https://www.getmaxim.ai/bifrost/blog/guardrails-at-the-gateway-what-bifrost-ships-and-what-it-integrates/) with secrets detection and custom regex redaction applied to prompts and completions
- **High availability:** clustering with real-time state synchronization across nodes and zero-downtime deployments
- **Deployment control:** in-VPC, on-premise, and air-gapped options for regulated environments

| Capability | Open-source Bifrost | Bifrost Enterprise |
| --- | --- | --- |
| Providers, routing, fallbacks | Included | Included |
| Virtual keys, budgets, rate limits | Included | Included |
| MCP tool filtering | Included | Included |
| SSO and role-based access control | Not included | OIDC with Okta, Entra, Keycloak, Google Workspace |
| Guardrails and signed audit logs | Not included | Included |
| Multi-node clustering | Not included | Real-time state synchronization |

Throughput is rarely what pushes teams over this line. The open-source build handles roughly 3,000 to 5,000 requests per second on a single instance, which covers most companies. What pushes them over is state: budgets and rate limits have to stay consistent across every node, and that requires real-time synchronization rather than a shared database, which is where [Bifrost Enterprise](https://www.getmaxim.ai/enterprise) starts. Organizations aligning to the [NIST AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework) generally need the audit and access-control layer regardless of how much traffic they run.

## Choosing an Open Source AI Gateway

The best open source AI gateway is the one that adds the least latency while enforcing the most granular governance, because those two properties decide whether it can carry production traffic without becoming either a bottleneck or a policy gap. The criteria worth testing before you commit traffic: measured overhead under sustained load, provider breadth, per-consumer budget and access control, self-hosting and air-gapped support, and whether the same layer governs tool calls as well as model calls.

**Best for:** Bifrost is built for enterprises running mission-critical AI workloads that require best-in-class performance, scalability, and reliability. It serves as a centralized AI gateway to route, govern, and secure all AI traffic across models and environments with ultra low latency. Bifrost unifies [LLM gateway](https://www.getmaxim.ai/llm-gateway), [MCP gateway](https://www.getmaxim.ai/mcp-gateway), and Agents gateway capabilities into a single platform. Designed for regulated industries and strict enterprise requirements, it supports air-gapped deployments, VPC isolation, and on-prem infrastructure. It provides full control over data, access, and execution, along with robust security, policy enforcement, and governance capabilities.

That last criterion is the one most evaluations miss. Agents spend more of their cost and carry more of their risk in tool execution than in inference, so a gateway that governs model calls but leaves MCP traffic unmanaged covers only half the surface. Bifrost measured [92% lower token costs at scale](https://www.getmaxim.ai/bifrost/blog/bifrost-mcp-gateway-access-control-cost-governance-and-92-lower-token-costs-at-scale/) once MCP traffic ran through the same layer. The [LLM Gateway Buyer's Guide](https://www.getmaxim.ai/resources/buyers-guide) has the full capability matrix, and this comparison of [open-source LLM gateways](https://www.getmaxim.ai/articles/top-5-open-source-llm-gateways-compared-2026/) covers how the self-hosted options differ.

## Setting One Up

Setting up an AI gateway takes three steps: run it, add provider credentials once, and repoint your existing SDK at it. No application logic changes, because the gateway speaks an OpenAI-compatible API, and reverting is the same base URL change in reverse. The [reference architecture for scaling LLMs safely](https://www.getmaxim.ai/articles/enterprise-ai-gateway-a-reference-architecture-for-scaling-llms-safely/) covers what to put around it in production.

Bifrost starts with no configuration file:

```bash
npx -y @maximhq/bifrost
```

Provider keys are configured once in the gateway, and each consuming team gets a virtual key carrying its own budget, rate limits, and allowed models. Docker and Kubernetes options are in the [gateway setup guide](https://docs.getbifrost.ai/quickstart/gateway/setting-up).

Then only the base URL and key change in your application:

```python
# Before
client = openai.OpenAI(api_key="<OPENAI-API-KEY>")

# After
client = openai.OpenAI(
    base_url="http://localhost:8080/openai",
    api_key="<BIFROST-VIRTUAL-KEY>"
)
```

![Without a gateway each service holds its own keys and retries; with a gateway all three services share one layer holding keys, failover, and attribution](https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/2026/10/ai-gateway-explained-what-it-is-and-how-it-works-without-with.svg)

*Figure 4: The scattered version cannot answer what the organization spends or which models it runs.*

Every request now routes through the gateway with failover, caching, budget enforcement, and usage attribution applied. The same one-line change works against the Anthropic, Google GenAI, LangChain, and LiteLLM SDKs, and against coding agents: [pointing Claude Code at the gateway](https://www.getmaxim.ai/bifrost/blog/integrating-claude-code-with-bifrost-gateway/) follows the identical pattern.

## Frequently Asked Questions

### When do you actually need an AI gateway?

The trigger is usually one of three signals: more than one team calling models, provider rate limits throttling production, or a compliance requirement for centralized audit and access control. Below that, direct provider SDKs are fine; past it, the scattered integration code and missing governance become the bottleneck an AI gateway removes.

### How does an AI gateway work internally?

It terminates the client request, authenticates the caller (in [Bifrost](https://www.getmaxim.ai) via a [virtual key](https://docs.getbifrost.ai/features/governance/virtual-keys)), applies governance and budget checks, routes to a provider with caching and failover, calls the upstream API, and logs the full request. Each stage runs on every request, which is why the gateway's per-request overhead matters so much.

### How much overhead does an AI gateway add?

It depends entirely on the runtime. Bifrost adds roughly 11 microseconds at 5,000 requests per second in [sustained benchmarks](https://www.getmaxim.ai/resources/benchmarks), which is imperceptible against model latencies measured in seconds. A gateway built on a slower runtime can add milliseconds per call that compound across an agentic session, so ask for published figures before committing.

### What changes when running an AI gateway at enterprise scale?

Single-node deployment covers routing and failover, but enterprise scale adds identity integration (OIDC SSO with Okta or Entra), role-based access control, signed audit logs, and [clustering](https://www.getmaxim.ai/enterprise) for high availability. Governance and compliance requirements, rather than raw throughput, are usually what push a team from a single node to an enterprise deployment.

### What is the difference between an AI gateway and an API gateway?

An API gateway manages inbound traffic to services you operate, priced and scaled by request. An AI gateway manages outbound traffic to model providers, where cost is counted in tokens, failure means failing over to another provider, caching can match on meaning rather than exact URLs, and content policy applies to prompts and completions. Most teams run both.

### What is the best open-source AI gateway?

The best open-source gateway is the one that adds the least overhead while covering routing, governance, and observability without a commercial tier gating the essentials. Bifrost is written in Go for low overhead, is fully self-hostable, and pairs the open-source core with an [enterprise tier](https://www.getmaxim.ai/enterprise) for SSO, RBAC, and clustering when teams need them.

### How do you set up an AI gateway?

Three steps: run the gateway, add provider keys once, and point your application's base URL at it. [Bifrost starts with no configuration file](https://docs.getbifrost.ai/quickstart/gateway/setting-up) via a single Docker container or NPX command, after which only the base URL and key change in application code and every request routes through the gateway with failover and logging applied.

## Getting Started with Bifrost as an AI Gateway

An AI gateway consolidates provider access, failover, spend control, and observability into one layer that every AI application in the organization shares. Bifrost delivers that with 11 microseconds of overhead, 25+ providers and 10,000+ models behind a single API, and governance that extends from model calls through to MCP tool calls.

Teams still comparing options can work through the [gateway evaluation criteria](https://www.getmaxim.ai/resources/buyers-guide) before running traffic through anything, and readers who want the architecture in more depth can continue with [AI gateway architecture, features, and why it matters](https://www.getmaxim.ai/llm-gateway). The routing side is covered in [five LLM routing strategies every AI gateway needs](https://www.getmaxim.ai/articles/5-llm-routing-strategies-every-ai-gateway-needs-in-2026/).

To see how Bifrost works as an AI gateway in your environment, [book a demo](https://www.getmaxim.ai/book-a-demo) with the Bifrost team.

## Read next

[![AI Data Residency: How to Control Where LLM Traffic Goes](https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/size/w720/2026/11/ai-data-residency-how-to-control-where-llm-traffic-goes-bifrost-1920x1080.png) AI data residency is the practice of keeping AI data in approved geographic locations. This guide traces where LLM data travels, compares provider controls on AWS and Azure, and shows how to enforce residency across routing, fallbacks, logs, and caches.](https://www.getmaxim.ai/articles/ai-data-residency-how-to-control-where-llm-traffic-goes/)

[![Rust vs Go for AI gateways: raw benchmark data on latency, throughput, and memo](https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/size/w720/2026/11/rust-vs-go-for-ai-gateways-raw-benchmark-data-on-latency-thr-bifrost-1920x1080.png) Rust vs Go is a comparison of two compiled languages with different memory models. This guide uses raw data from a shared AI gateway benchmark to show where each leads on latency, throughput, and memory, and what buyers should weigh above language.](https://www.getmaxim.ai/articles/rust-vs-go-ai-gateway/)

[![LiteLLM Rust Gateway: What Buyers Should Verify in 2026](https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/size/w720/2026/11/litellm-rust-gateway-what-buyers-should-verify-in-2026-bifrost-1920x1080.png) LiteLLM Rust is the beta Rust core behind LiteLLM's latest gateway benchmarks. This guide checks which routes run on Rust today, what the raw benchmark files show against Bifrost, and what platform teams should verify before choosing a gateway.](https://www.getmaxim.ai/articles/litellm-rust-gateway-what-buyers-should-verify-in-2026/)

## More in AI Gateway

[![Enterprise AI Gateway for Multi-Model Routing: Top 5 in 2026](https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/size/w720/2026/09/enterprise-ai-gateway-for-multi-model-routing-top-5-in-2026-bifrost-isometric.png) An AI gateway is the control point that decides which provider, model, and API key serve each LLM request. This guide compares Bifrost, Kong AI Gateway, LiteLLM, Cloudflare AI Gateway, and Azure API Management on routing rules, load balancing, fallbacks, and governance of routes.](https://www.getmaxim.ai/articles/enterprise-ai-gateway-for-multi-model-routing-top-5-in-2026/)

[![Choosing an AI Gateway for Claude Code: Complete Guide](https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/size/w720/2026/09/choosing-an-ai-gateway-for-claude-code-complete-guide-bifrost-isometric.png) A guide to choosing an AI gateway for Claude Code: six evaluation criteria, per-developer budgets, the configuration, and the defaults Claude Code changes when ANTHROPIC\_BASE\_URL points at a gateway.](https://www.getmaxim.ai/articles/choosing-an-ai-gateway-for-claude-code-complete-guide/)

```json
{
    "@context": "https://schema.org",
    "@type": "Article",
    "publisher": {
        "@type": "Organization",
        "name": "Maxim Articles",
        "url": "https://www.getmaxim.ai/articles/",
        "logo": {
            "@type": "ImageObject",
            "url": "https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/size/w256h256/2025/08/thumbnail.png",
            "width": 60,
            "height": 60
        }
    },
    "author": {
        "@type": "Person",
        "name": "Kamya Shah",
        "image": {
            "@type": "ImageObject",
            "url": "https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/2025/09/WhatsApp-Image-2025-08-29-at-17.40.40-1.jpeg",
            "width": 1200,
            "height": 1600
        },
        "url": "https://www.getmaxim.ai/articles/author/kamya/",
        "sameAs": []
    },
    "headline": "AI Gateway Explained: What It Is and How It Works",
    "url": "https://www.getmaxim.ai/articles/ai-gateway-explained-what-it-is-and-how-it-works/",
    "datePublished": "2026-10-01T04:09:00.000Z",
    "dateModified": "2026-10-14T13:20:25.000Z",
    "image": {
        "@type": "ImageObject",
        "url": "https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/size/w1200/2026/08/ai-gateway-explained-what-it-is-and-how-it-works-bifrost-isometric.png",
        "width": 1200,
        "height": 675
    },
    "keywords": "AI Gateway",
    "description": "What is an AI gateway? It is the layer between your applications and every LLM provider they call, exposing one API while applying routing, failover, budgets, and observability. This guide covers the five stages of a request, when a gateway pays for itself, and what changes at enterprise scale.",
    "mainEntityOfPage": "https://www.getmaxim.ai/articles/ai-gateway-explained-what-it-is-and-how-it-works/"
}
{
  "@context": "https://schema.org",
  "@type": "FAQPage",
  "mainEntity": [
    {
      "@type": "Question",
      "name": "When do you actually need an AI gateway?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "The trigger is usually one of three signals: more than one team calling models, provider rate limits throttling production, or a compliance requirement for centralized audit and access control. Below that, direct provider SDKs are fine; past it, the scattered integration code and missing governance become the bottleneck an AI gateway removes."
      }
    },
    {
      "@type": "Question",
      "name": "How does an AI gateway work internally?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "It terminates the client request, authenticates the caller (in Bifrost via a virtual key), applies governance and budget checks, routes to a provider with caching and failover, calls the upstream API, and logs the full request. Each stage runs on every request, which is why the gateway's per-request overhead matters so much."
      }
    },
    {
      "@type": "Question",
      "name": "How much overhead does an AI gateway add?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "It depends entirely on the runtime. Bifrost adds roughly 11 microseconds at 5,000 requests per second in sustained benchmarks, which is imperceptible against model latencies measured in seconds. A gateway built on a slower runtime can add milliseconds per call that compound across an agentic session, so ask for published figures before committing."
      }
    },
    {
      "@type": "Question",
      "name": "What changes when running an AI gateway at enterprise scale?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "Single-node deployment covers routing and failover, but enterprise scale adds identity integration (OIDC SSO with Okta or Entra), role-based access control, immutable audit logs, and clustering for high availability. Governance and compliance requirements, rather than raw throughput, are usually what push a team from a single node to an enterprise deployment."
      }
    },
    {
      "@type": "Question",
      "name": "What is the best open-source AI gateway?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "The best open-source gateway is the one that adds the least overhead while covering routing, governance, and observability without a commercial tier gating the essentials. Bifrost is written in Go for low overhead, is fully self-hostable, and pairs the open-source core with an enterprise tier for SSO, RBAC, and clustering when teams need them."
      }
    },
    {
      "@type": "Question",
      "name": "How do you set up an AI gateway?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "Three steps: run the gateway, add provider keys once, and point your application's base URL at it. Bifrost starts with no configuration file via a single Docker container or NPX command, after which only the base URL and key change in application code and every request routes through the gateway with failover and logging applied."
      }
    }
  ]
}
```