The problem an LLM gateway solves
Every provider has its own API, its own authentication, its own rate limits, its own error codes and its own idea of how streaming and tool calls look. An application that wants to use two of them has to implement both; one that wants to fall back from one to the other has to implement the mapping between their failure modes. Multiply that by every application in a company and the duplicated effort is large, and the inconsistency is larger.
Provider keys are the second problem. Without a gateway, the key that bills the whole company sits in the environment of every service that calls a model. Rotating it is a coordinated deployment, scoping it to a team is impossible, and attributing a bill to a feature means parsing provider dashboards.
An LLM gateway solves both by owning the provider relationship. Applications get one API and a scoped credential; the gateway gets the keys, the translation layer and the policy.
What a gateway does with each request
The capabilities below are what separate a gateway from a thin proxy. Not every product has all of them, and the order in which a team needs them is usually the order listed.
- Unified API. One request format for chat, completions, embeddings, images, audio and tool calls, with the gateway translating to each provider. Most gateways expose the OpenAI shape, and some also accept the Anthropic and Google shapes so existing SDKs keep working.
- Virtual keys. Scoped credentials that stand in for provider keys and carry the identity of a team, customer or feature.
- Routing and load balancing. Weighted or health-based distribution across keys, deployments, regions and providers.
- Fallbacks. Ordered alternatives that take over on errors, timeouts and rate limits, including across providers with different model names.
- Budgets and rate limits. Token, dollar and request ceilings per key, with hierarchy so a team limit contains its members’ limits.
- Caching. Exact-match and semantic caching, which matter most for retrieval and classification workloads with repeated inputs.
- Guardrails. Input and output checks for personal data, secrets, injections and policy violations, built in or delegated to a guardrail provider.
- Telemetry. Logs with prompts and responses where policy allows, OpenTelemetry traces and Prometheus metrics with cost and token fields.
Gateway, SDK wrapper or hosted router?
Three things get called an LLM gateway and only one of them is. An SDK wrapper is a library that normalises provider APIs inside your process. It gives you the unified interface and nothing else: no shared budgets, no central keys, no policy a platform team can enforce. It is the right choice for a single service that wants provider independence and nothing more.
A hosted router is a third-party service that fronts many providers under one key and bills you for the tokens. It removes provider accounts entirely and is convenient for experiments and small products, at the price of every prompt transiting the router and of a dependency you cannot run yourself.
A gateway is a service you run, or a vendor runs for you in your cloud, that holds your own provider accounts and applies your policy. It is the only one of the three that gives a platform team control over the whole estate, and the only one that keeps the data path inside your network when that matters.
Performance: why overhead matters more than it looks
A gateway is on the path of every token. For a non-streaming request that takes a provider two seconds, ten milliseconds of gateway overhead is noise. For a streaming request, overhead shows up as time to first token, and for high-volume classification or embedding workloads at thousands of requests per second it becomes the bottleneck.
The variables are the runtime, how much of the request body the gateway parses, and how it handles streaming. Gateways written in compiled languages and designed to stream bytes through add overhead measured in microseconds; gateways that deserialise the full body in an interpreted runtime add milliseconds and need more replicas for the same load.
Vendor numbers are only useful with a method. Ours, including the hardware, request mix and comparison setup, is published at Performance benchmarks so it can be reproduced or disputed.
Deployment: where the gateway runs and what that implies
Self-hosting puts the gateway in your VPC or cluster as a container or single binary. Prompts never leave your network, latency to your services is minimal, and you control upgrades; you also operate it, including clustering and state for budgets and caches.
Managed gateways remove the operating burden and add a hop through the vendor. For many products that is fine; for health, finance and government workloads it is often a non-starter, which is why most vendors offer an in-VPC option.
The hybrid that works for most enterprises is an open-source gateway self-hosted on the data path, with a commercial tier for identity integration, access control, clustering and support. It keeps the data path inspectable and the operating model familiar.
How to choose an LLM gateway
Make the shortlist with the deployment model and the provider list, because those are hard constraints. Then test the rest.
- Compatibility: do your existing SDKs and frameworks work by changing a base URL, including streaming and tool calls?
- Overhead: measured with a published method at your request rate, on your hardware class.
- Failover behaviour: what counts as a failure, how fast fallback happens, and whether model names map across providers.
- Governance: virtual keys, hierarchical budgets, rate limits and model allowlists, tied to your identity provider.
- Observability: OpenTelemetry and Prometheus out of the box, and export to your log store.
- Operations: clustering, zero-downtime config changes, upgrade path and what state it keeps.
- Roadmap fit: MCP and agent tooling, since the next workload will be agents rather than chat.
