Why AI gateways exist
The first generation of AI features was built by calling a provider SDK straight from application code. That works for one team and one model. It stops working when there are twenty teams, five providers, a finance department asking who spent what, and a security review asking what data left the building.
Each of those questions can be answered inside every application, with every team writing its own retry logic, its own key storage, its own usage counters and its own redaction. In practice they are not answered consistently, and the gaps show up as surprise bills, leaked keys and outages when a single provider has a bad day.
An AI gateway moves those cross-cutting concerns to one place. The pattern is the same one that produced API gateways for REST traffic and service meshes for east-west traffic: take the behaviour every caller needs, make it policy, and enforce it on the path rather than in the code.
What an AI gateway does
Products in this category vary, but the core responsibilities are stable. A gateway that lacks several of these is usually a router or a proxy rather than a gateway.
- One interface for many providers. Applications call a single, usually OpenAI-compatible, API and the gateway translates to each provider’s own format. Switching models becomes a configuration change.
- Credential management. Provider API keys live in the gateway. Callers get scoped virtual keys that can be rotated, limited and revoked without touching provider accounts.
- Routing, load balancing and failover. Requests are spread across keys, regions and providers, and rerouted when one fails or slows down.
- Budgets and rate limits. Spend and request volume are capped per key, team, customer or model, with the counters kept in one place so limits actually hold.
- Guardrails. Inputs and outputs are checked for personal data, secrets, prompt injection and unsafe content, either with built-in checks or by calling an external guardrail service.
- Caching. Identical or semantically similar requests can be answered from cache, which cuts cost and latency for repetitive workloads.
- Observability. Every request produces a log entry, a trace span and metrics with the fields that matter for AI: tokens, cost, model, latency and outcome.
- Tool and agent governance. Newer gateways also broker tool calls made through the Model Context Protocol, applying the same identity, policy and audit controls to what an agent can do, not just what it can ask. See What is an MCP gateway?.
Where it sits in the architecture
An AI gateway is deployed as a network service, typically inside the organisation’s own cloud account or cluster, and applications are pointed at it by changing a base URL and a key. Nothing about the application’s business logic changes, which is why migrations are usually measured in hours rather than weeks.
Downstream, the gateway holds the provider configuration: which accounts and regions exist, which models are approved, and which fallbacks apply. Upstream, it integrates with the identity provider for who is allowed in, with the observability stack for where telemetry goes, and with finance or platform tooling for budgets.
Because it is on the request path, latency matters. A gateway written for throughput adds overhead measured in microseconds; one that re-serialises JSON through several layers can add tens of milliseconds, which is noticeable on streaming responses. Any performance claim should come with a published method. Ours is at Performance benchmarks.
What an AI gateway is not
It is not a model. The gateway does no inference of its own; it decides where inference happens and under what rules.
It is not an API gateway with a new name. A general API gateway understands HTTP routes, authentication and rate limits, but it does not understand tokens, model names, provider failover semantics or the content of a prompt. Some API gateway vendors ship AI plugins that add parts of this; the comparison table below shows where the lines are.
It is not an observability platform or an evaluation tool. A gateway produces the telemetry those tools consume, and some ship dashboards, but judging whether an answer was good is a separate discipline. See What is AI observability?.
Deployment models and trade-offs
Gateways come in three shapes. Managed gateways run as a vendor-hosted service: nothing to operate, but every prompt and response transits the vendor, which rules them out for many regulated workloads. Self-hosted gateways run in your own VPC or cluster as a binary or container: full data control and the lowest latency, at the cost of operating one more service. Library gateways are code embedded in each application: zero infrastructure, but policy lives in every deployment and cannot be enforced centrally.
Open-source gateways let a team start self-hosted without a procurement cycle and inspect exactly what the data path does. Commercial tiers on top of them usually add single sign-on, fine-grained access control, clustering and support. Closed commercial gateways compete on breadth of integrations and on the vendor doing the operating.
How to evaluate an AI gateway
Start from the traffic you have. A team with one provider and a handful of apps needs routing and cost visibility; an enterprise with regulated data needs in-VPC deployment, identity integration and audit logs before it needs anything else.
- Overhead under load, measured with a published method at a request rate close to yours, on streaming and non-streaming calls.
- Provider coverage for the models you use today and the ones on your roadmap, including self-hosted models.
- Governance depth: virtual keys, hierarchical budgets, rate limits, model allowlists, and whether they bind to your identity provider.
- Guardrail model: built-in checks, pluggable external providers, or both; and whether they run on responses as well as requests.
- Observability: native OpenTelemetry and Prometheus, log export to the stores you already run, and request-level cost attribution.
- Deployment and data path: where it runs, what leaves your network, and how it is upgraded and clustered.
- Agent readiness: MCP support, tool-call policy and audit, since agent traffic is the fastest-growing share of most estates.
