Best LLM Gateways for LangChain and LangGraph

TL;DR

• A LangChain LLM gateway is a service that LangChain chat models call in place of provider APIs, so keys, budgets, fallbacks, and logs sit in one place instead of in every agent process.
• Bifrost, LangSmith LLM Gateway, Kong AI Gateway, OpenRouter, Cloudflare AI Gateway, and Vercel AI Gateway all accept LangChain traffic, but they differ on deployment, response caching, and whether MCP tool calls and A2A agent calls go through the same gateway.
• Bifrost exposes a dedicated /langchain endpoint, a /mcp endpoint for LangGraph tools, and an Agent Gateway for A2A, all self-hosted and governed by the same virtual keys.
• LangSmith LLM Gateway is in public beta and adds spend, rate-limit, and model-access policies to LangSmith; its MCP and agent gateways are still on the roadmap.
• Hosted routers such as OpenRouter and Vercel AI Gateway suit teams that want one bill across hundreds of models and do not need traffic to stay inside their own network.

A LangChain LLM gateway is a network service that sits between LangChain or LangGraph code and model providers, holding provider credentials and enforcing access, spend, routing, and logging policy before any request leaves for OpenAI, Anthropic, or Bedrock. Teams adopt one when a prototype becomes a fleet of agents, each with its own keys and retry logic. Bifrost, the open-source AI gateway built in Go by Maxim AI, is the best choice for enterprises running mission-critical AI workloads that require top-tier performance, scalability, and reliability, and it covers LangGraph's tool and agent calls as well as its model calls. This guide compares six LLM gateways for LangChain and LangGraph on integration path, governance, caching, deployment, and MCP and A2A coverage.

What Is an LLM Gateway for LangChain and LangGraph?

An LLM gateway for LangChain is an OpenAI-compatible or provider-compatible endpoint that LangChain chat models use as their base URL. The gateway authenticates each call, checks budgets and rate limits, routes it to a provider with fallbacks, and records tokens and cost, so application code keeps calling ChatOpenAI or init_chat_model unchanged.

LangChain already abstracts providers inside one process. A gateway moves that abstraction onto the network, where every service, notebook, and LangGraph deployment shares the same keys, limits, and logs. Our explainer on what an LLM gateway is covers the general architecture; for LangGraph the important detail is that a graph sends three kinds of traffic, not one.

Figure 1: A model-only gateway covers the first path; a gateway that also speaks MCP and A2A puts tool and agent calls under the same keys and logs.

As Figure 1 shows, a LangGraph agent calls models from its chat model nodes, calls tools from MCP servers (LangChain now ships MCP support in langchain.mcp, which replaced the older langchain-mcp-adapters package), and can call other agents over the Agent2Agent protocol. Most products sold as LLM gateways govern only the first path, while the Bifrost LLM gateway also covers tools and agents.

How to Evaluate LLM Gateways for LangChain

Evaluate an LLM gateway for LangChain on six things: how LangChain connects to it, which providers it reaches, how it fails over, what spend and rate controls it enforces, whether it governs MCP tools and A2A agents, and where it runs. The table below turns those into checks a platform team can run during a trial.

Figure 2: The application only changes its base URL; key checks, spend limits, caching, and failover all run before a provider is called.

The difference between a gateway that handles models only and one that also governs agents is covered in agent gateway vs LLM gateway. Teams that care mostly about API compatibility can start from our comparison of LLM gateways with an OpenAI-compatible API.

LangChain LLM Gateways Compared at a Glance

The six gateways below all accept LangChain traffic. They split on three lines: self-hosted versus hosted, response caching versus provider prompt caching only, and model traffic only versus models plus MCP tools and A2A agents. Bifrost is the only one in this list that is open source, self-hosted, and governs all three traffic types today.

Sourced from each vendor's documentation as read on November 5, 2026. "Not published" means the capability did not appear in the vendor's docs.

For a broader evaluation framework beyond LangChain, the LLM gateway buyer's guide lists the questions procurement and platform teams usually ask.

1. Bifrost

Bifrost is a self-hosted, open-source AI gateway that governs model calls, MCP tool calls, and agent-to-agent calls through one deployment. LangChain apps connect through a dedicated /langchain endpoint by changing the base URL, and LangGraph agents can load tools from Bifrost's /mcp endpoint and reach other agents through the Agent Gateway.

Best for: Bifrost is built for enterprises running mission-critical AI workloads that require top-tier performance, scalability, and reliability. It serves as a centralized AI gateway to route, govern, and secure all AI traffic across models and environments with ultra low latency. Bifrost unifies LLM gateway, MCP gateway, and Agents gateway capabilities into a single platform. Designed for regulated industries and strict enterprise requirements, it supports air-gapped deployments, VPC isolation, and on-prem infrastructure. It provides full control over data, access, and execution, along with strong security, policy enforcement, and governance capabilities.

The LangChain integration keeps each provider's native LangChain class. ChatOpenAI, ChatAnthropic, ChatGoogleGenerativeAI, and ChatBedrockConverse point at the same endpoint, and a virtual key travels in default_headers or as the API key:

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    model="gpt-4o-mini",
    base_url="http://localhost:8080/langchain",
    default_headers={"x-bf-vk": "<YOUR-BIFROST-VIRTUAL-KEY>"},
)

The same endpoint serves LangChain embeddings and the CohereRerank and BedrockRerank compressors used in retrieval pipelines. One limit applies: a request works only for providers that both LangChain and Bifrost support.

Figure 3: Model, tool, and agent access are separate grants on the same virtual key, and a shared session ID traces one agent run across all three.

Key capabilities for LangChain and LangGraph teams:

• Virtual keys and budgets. Each agent or team gets a virtual key with allowed models, and budgets and rate limits stack from customer to team to key; an exhausted budget returns a 402.
• Retries and fallbacks. Bifrost rotates keys on 429 and moves to the next provider in a fallback chain after retries, so a graph run survives a provider outage.
• Semantic caching. Exact and semantic caching works across every process that shares the gateway, unlike an in-memory LangChain cache.
• MCP gateway for LangGraph tools. Bifrost exposes its connected tools at a single /mcp endpoint, filtered per virtual key, for MCP protocol versions up to 2026-07-28; LangChain's MCPAdapter can load tools from that endpoint by URL.
• Code Mode for tool-heavy agents. Code Mode cuts input tokens by up to 92.8% across large tool sets, as detailed in the post on 92% lower MCP token costs.
• Agent Gateway for A2A. The Agent Gateway registers A2A agents by their agent card URL and supports JSON-RPC, REST, and gRPC, which fits LangGraph assistants published over A2A; the Agent Gateway launch post explains why models, tools, and agents share one control plane.

Bifrost adds about 11 microseconds of overhead per request at 5,000 RPS, reaches 10,000+ models across 25+ providers, and runs as a single binary inside the customer's VPC with no prompts or telemetry leaving it.

Enterprise guardrails screen both LLM and MCP traffic, and OpenTelemetry export honors an inbound W3C traceparent, so gateway spans join the LangGraph run's own trace.

2. LangSmith LLM Gateway

LangSmith LLM Gateway is LangChain's own gateway, built into LangSmith. It launched in private beta on May 13, 2026, and moved to public beta on July 30, 2026. LangChain chat models reach it without code changes by setting LANGSMITH_GATEWAY=true alongside a LangSmith API key, and every call becomes a LangSmith trace.

Best for: Teams already on LangSmith that want spend and rate policies tied to their existing LangSmith organization, workspaces, and traces.

• Integration: the zero-code path needs recent packages (for example langchain-openai 1.4.1 or later); the explicit path is init_chat_model with base_url set to the gateway and the LangSmith key as the API key.
• API formats: OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages, with translation between providers.
• Policies: spend policies by organization, workspace, user, or API key (hourly to monthly, 402 on block), request and token rate limits, model access allowlists, and header-scoped policies.
• Fallbacks: one primary plus one to five fallback models, which can cross providers and formats.
• Data protection: PII and secrets redaction on requests only; responses are not scanned and scanner failures let requests through. LangChain lists data protection controls for Enterprise customers.
• Deployment: LangSmith Cloud, bring-your-own-cloud, or self-hosted LangSmith through its Helm chart.

The gateway is included with LangSmith Plus and Enterprise during the beta, with general availability pricing still unannounced. No response cache or load balancing is documented, and tool, MCP, and agent-to-agent gateways appear only as roadmap items in the launch post. Teams that need shared response caching across agents can compare it against gateway-side semantic caching for LLMs.

3. Kong AI Gateway

Kong AI Gateway extends Kong's API gateway to LLM, MCP, and A2A traffic. Version 2.x, released July 10, 2026, replaced the plugin model with AI entities and policies. LangChain connects with ChatOpenAI pointed at a Kong route, as shown in Kong's v1 how-to.

Best for: Organizations that already run Kong for API management and want AI traffic under the same operations team.

• Integration: Kong's only LangChain page sits in the older v1 docs; in 2.x, AI Models expose OpenAI-compatible endpoints by default.
• Routing: load balancing by weighted round-robin, lowest latency, lowest usage, semantic matching, or priority, with retries and failover across targets.
• Controls: AI Rate Limiting Advanced limits tokens or cost per consumer, and AI Cost Management sets budgets with notification thresholds.
• Guardrails and caching: PII sanitization, prompt guards, cloud guardrail connectors, and a semantic cache.
• MCP and A2A: an AI MCP Server entity aggregates REST and MCP sources, and an AI Agent entity proxies A2A.

The 2.x entity model runs in hybrid mode with a Konnect-managed control plane and self-managed data planes, fully hosted on Dedicated Cloud Gateways, or fully on-prem from version 2.2 with an Enterprise license. AI Proxy Advanced, the semantic cache, advanced rate limiting, and PII sanitization are Enterprise features on self-managed Kong and paid add-ons on Konnect Plus. Teams weighing that licensing model can see how other options compare in Kong AI Gateway alternatives.

4. OpenRouter

OpenRouter is a hosted model router that exposes 600+ models from 80+ providers through one API. LangChain has an official ChatOpenRouter class in the langchain-openrouter package, which LangChain recommends over ChatOpenAI with a custom base URL so that router-specific fields are preserved.

Best for: Teams that want hundreds of hosted models on one bill without operating a gateway.

• Routing: model fallbacks through a models list, and provider routing that orders or sorts providers by price, throughput, or latency.
• Caching: response caching, with cache hits not billed.
• Controls: workspace budgets and per-key budgets that reset daily, weekly, or monthly; the lower limit wins.
• Data policy: zero data retention enforcement and per-request data_collection: "deny"; in-region routing on Business and Enterprise plans.
• Guardrails: a sensitive-info guardrail that redacts or blocks inputs, without scanning model responses.

OpenRouter is hosted only, charges a platform fee on credits (5.5% on the Standard plan), and does not publish an MCP gateway for application tool traffic or A2A support. Teams that need a self-hosted option can compare OpenRouter alternatives.

5. Cloudflare AI Gateway

Cloudflare AI Gateway runs on Cloudflare's network and adds analytics, logging, caching, rate limiting, retries, and fallbacks to provider calls. Its docs have no LangChain page; LangChain apps use the OpenAI-compatible REST endpoint with ChatOpenAI, or Cloudflare's own langchain-cloudflare package for Workers AI models.

Best for: Teams already on Cloudflare that want observability, spend limits, and DLP in front of hosted model APIs.

• Integration: the newer REST API at api.cloudflare.com/.../ai/v1/chat/completions is the recommended path, since the older /compat endpoint is deprecated for single-model calls.
• Caching: exact-match response caching, off by default; semantic caching is listed as planned.
• Controls: request rate limiting per gateway and spend limits that block with a 429, enforced with eventual consistency.
• Reliability: gateway-level retries up to five attempts, and model aliases that fall back in order.
• Security: Llama Guard-based guardrails and DLP scanning of prompts and responses.

Cloudflare AI Gateway is hosted only, and its documentation lists no MCP or A2A gateway. Core features are free, and Unified Billing adds a 5% fee on purchased credits. For self-hosted options, see Cloudflare AI Gateway alternatives.

6. Vercel AI Gateway

Vercel AI Gateway is a managed gateway with one API for 400+ models, routing, fallbacks, and usage monitoring. It became generally available on August 21, 2025. Vercel documents LangChain usage in JavaScript with ChatOpenAI and a custom baseURL; Python apps use the same OpenAI-compatible endpoint.

Best for: TypeScript teams deploying on Vercel that want model access and spend tracking without separate infrastructure.

• Formats: AI SDK, OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages.
• Fallbacks: a models list under providerOptions.gateway, plus provider ordering and filtering.
• Budgets: limits by team, project, API key, or user, though bring-your-own-key spend does not count toward them.
• Data: zero data retention by default, prompt-training opt-out, and provider and model allowlists.
• Pricing: no markup or platform fee on tokens.

Vercel AI Gateway is Vercel-hosted, does not rate limit paid-tier requests, and publishes no PII guardrails, MCP gateway, or A2A support. Teams that outgrow it can review Vercel AI Gateway alternatives.

How to Choose a Gateway for a LangGraph Stack

Choose by two questions before comparing features: must model traffic stay inside your own network, and do tool calls and agent calls need the same governance as model calls? Self-hosted requirements narrow the field quickly, and MCP plus A2A coverage narrows it further.

Figure 4: Data residency and the scope of traffic decide most of the choice before any feature comparison does.

• Regulated or VPC-bound LangGraph deployments need a self-hosted gateway. Bifrost runs as a single binary inside the VPC and covers models, tools, and agents; the Enterprise page lists its clustering and compliance options.
• Agent-heavy graphs with many tools benefit most from an MCP gateway that filters tools per key and trims tool definitions from the prompt.
• Multi-agent systems that call other agents over A2A need registration, access grants, and logs per agent; the A2A protocol overview explains the moving parts.
• Teams standardized on LangSmith can use its gateway for spend policies and add a separate layer for tools and agents.
• Prototypes without residency requirements are served by hosted routers.

The same pattern appears for other agent frameworks; the guide to LLM gateways for Claude Code applies it to coding agents, and Pydantic AI Gateway alternatives applies it to Pydantic AI. For the general model, revisit our guide to how LLM gateways work.

Frequently Asked Questions

What is a LangChain LLM gateway?

A LangChain LLM gateway is a service that LangChain chat models call instead of provider APIs. It stores provider keys, authenticates callers, applies budgets and rate limits, retries or fails over between providers, and logs tokens and cost. LangChain code only changes its base_url, so ChatOpenAI, ChatAnthropic, and init_chat_model keep working while governance moves to one shared layer.

Can I use an LLM gateway with LangGraph?

Yes. LangGraph nodes call models through LangChain chat model classes, so any gateway that LangChain can reach also serves LangGraph. Tool nodes and subagent calls are separate: tools need a gateway with an MCP endpoint, and agent-to-agent calls need A2A support. Bifrost covers all three; most hosted gateways cover only model calls.

Does LangChain have its own LLM gateway?

Yes. LangSmith LLM Gateway is LangChain's gateway inside LangSmith, in public beta since July 30, 2026. It adds spend policies, rate limits, model access rules, fallbacks, and request redaction, and traces every call in LangSmith. It runs in LangSmith Cloud, in a customer's cloud, or self-hosted with LangSmith, and its MCP and agent gateways are not yet released.

How do I connect LangChain to an OpenAI-compatible gateway?

Set base_url on ChatOpenAI or pass base_url and model_provider="openai" to init_chat_model, and use the gateway's key as the API key. LangChain warns that provider-specific fields may not survive this path, which is why it recommends ChatOpenRouter for OpenRouter, and why Bifrost's /langchain route accepts each provider's native LangChain class.

How do LangGraph agents use MCP tools through a gateway?

LangChain's MCPAdapter in langchain.mcp loads tools from a remote MCP server URL over streamable HTTP and returns plain LangChain tools for a graph. Pointing it at a gateway's MCP endpoint, such as Bifrost's /mcp, gives the agent every tool its virtual key allows, with access checks and logs applied centrally; the MCP gateway guide covers the setup.

Should I use LangChain fallbacks or gateway fallbacks?

LangChain's ModelFallbackMiddleware switches models inside one process, which helps a single agent. Gateway fallbacks apply to every service and also rotate keys on rate limits, retry transient errors, and record each attempt. Many teams keep the middleware for model choice and rely on the gateway for provider outages and key-level failures.

Run LangChain and LangGraph Through Bifrost

A LangChain LLM gateway should govern everything a LangGraph agent sends out, not only its model calls. Bifrost gives LangChain a dedicated endpoint, gives LangGraph tools a governed MCP endpoint, and registers A2A agents in the same self-hosted deployment, with virtual-key access and session-linked logs across all three. Start with the open-source LLM gateway, review AI governance in Bifrost, or book a demo to see Bifrost run your LangGraph agents.