# Bifrost > Bifrost is an open-source, high-performance AI gateway from Maxim. It puts a single OpenAI-compatible API in front of 15+ model providers with automatic failover, load balancing, semantic caching, governance, guardrails, MCP support, and full observability. Index: https://www.getmaxim.ai/llms.txt --- title: "Bifrost pricing" description: "Compare Bifrost OSS and Enterprise pricing for an open-source AI gateway with drop-in SDK compatibility, automatic fallbacks, observability, MCP Gateway, and enterprise governance." url: "https://www.getmaxim.ai/pricing" markdown: "https://www.getmaxim.ai/pricing.md" --- # Bifrost pricing > Compare Bifrost OSS and Enterprise pricing for an open-source AI gateway with drop-in SDK compatibility, automatic fallbacks, observability, MCP Gateway, and enterprise governance. ## Important Links - [View MCP Gateway](https://www.getmaxim.ai/resources/mcp-gateway.md) - [Features](https://www.getmaxim.ai/#features) - [Enterprise](https://www.getmaxim.ai/enterprise) - [Pricing](https://www.getmaxim.ai/pricing) - [Docs](https://docs.getbifrost.ai) - [GitHub](https://github.com/maximhq/bifrost) - [Book a Demo](https://www.getmaxim.ai/book-a-demo) ## Pricing Plans _OSS and Enterprise_ Bifrost pricing is split into a free open-source plan and a custom-priced enterprise plan. Bifrost OSS is free forever for developers, small teams, and self-managed deployments. Bifrost Enterprise is for teams running production AI systems at scale with private deployment, governance, identity, compliance, and support requirements. - **Free forever - OSS:** Self-hosted with Docker, Kubernetes, or Go binary. Includes drop-in replacement, OpenTelemetry-compatible metrics and traces, built-in observability, budget management, virtual keys, routing rules, Bifrost CLI, automatic fallbacks, caching, MCP Gateway with Code Mode, prompt repository, custom plugins, Maxim AI integration, docs, and community support. [GitHub](https://github.com/maximhq/bifrost), [Docs](https://docs.getbifrost.ai). - **Custom pricing - Enterprise:** Includes everything in OSS plus guardrails, cluster mode, adaptive load balancing, enterprise SSO via SAML and OIDC, vault support, MCP with federated auth, log exports, audit logs, role-based access control, SLA-backed enterprise support, and enterprise-ready VPC, on-prem, and air-gapped deployment. [Book a Demo](https://www.getmaxim.ai/book-a-demo), [Enterprise](https://www.getmaxim.ai/enterprise). ## Bifrost OSS _Free forever_ The OSS plan is meant for developers, small teams, and self-managed deployments that want a high-performance LLM gateway without a commercial license. It includes the core routing, fallback, observability, virtual key, budget, cache, MCP, CLI, plugin, and drop-in replacement capabilities needed to run Bifrost locally or in self-managed infrastructure. - [GitHub](https://github.com/maximhq/bifrost) - [Quickstart docs](https://docs.getbifrost.ai/quickstart/gateway/setting-up) - [Bifrost CLI resource](https://www.getmaxim.ai/resources/bifrost-cli.md) ## Bifrost Enterprise _Custom pricing_ The Enterprise plan adds production controls for organizations standardizing AI usage across teams, applications, and regulated environments. Enterprise capabilities focus on private deployment, identity provider integration, RBAC, auditability, guardrails, adaptive load balancing, clustering, vault-backed secrets, log exports, federated MCP authentication, and commercial support. - **Private deployment:** Deploy Bifrost in VPC, on-premise, air-gapped, or multi-cloud environments. [Enterprise deployment](https://www.getmaxim.ai/resources/enterprise-deployment.md). - **Governance:** Use SSO, RBAC, virtual keys, budgets, audit logs, and policy controls for production AI access. [Governance resource](https://www.getmaxim.ai/resources/governance.md). - **Guardrails:** Add enterprise safety controls for content moderation, PII handling, and policy enforcement. [Guardrails resource](https://www.getmaxim.ai/resources/guardrails.md). ## OSS vs Enterprise Feature Comparison _Capability matrix_ OSS includes the core open-source gateway capabilities. Enterprise adds private deployment, identity, governance, compliance, reliability, and support controls for production organizations. | Capability | OSS | Enterprise | | --- | --- | --- | | Single OpenAI-compatible API | Included | Included | | Drop-in Replacement | Included | Included | | OpenTelemetry | Included | Included | | Built-in Observability | Included | Included | | Budget and Rate Limits | Included | Included | | Virtual Keys | Included | Included | | Custom Routing Rules and Flows | Included | Included | | Bifrost CLI | Included | Included | | MCP Gateway | Included | Included | | MCP Code Mode | Included | Included | | Semantic Caching | Included | Included | | Prometheus Metrics | Included | Included | | Prompt Repository | Included | Included | | Custom Plugin Development | Included | Included | | Guardrails | Not included | Included | | Clustering | Not included | Included | | Adaptive Load Balancing | Not included | Included | | SAML, OpenID, and Audit Reports | Not included | Included | | Vault Support | Not included | Included | | Log Exports | Not included | Included | | In-VPC Deployments | Not included | Included | | Identity Providers like Okta and Entra | Not included | Included | | Role-Based Access Control | Not included | Included | | MCP with Federated Auth | Not included | Included | | Audit Logs | Not included | Included | | External OTel Connectors | Not included | Included | | Commercial onboarding and production support | Community | Included | ## When Enterprise Is The Right Fit _Production requirements_ - **Private networking and deployment:** Choose Enterprise when Bifrost must run inside private cloud infrastructure with network isolation and stronger security boundaries. - **Advanced governance:** Choose Enterprise when identity provider integration, Okta or Entra support, RBAC, audit logs, vault-backed key management, and compliance workflows are required. - **Enterprise-grade reliability:** Choose Enterprise when production AI systems need guardrails, adaptive load balancing, clustering, zero-downtime deployment support, log exports, and SLA-backed support. ## Features ### OSS Features - **01 - Model Catalog:** Access 8+ providers and 1000+ AI models through a unified interface. Also supports custom deployed models. - **02 - Budgeting:** Set spending limits and track costs across teams, projects, and models. - **03 - Provider Fallback:** Automatic failover between providers ensures 99.99% uptime for your applications. - **04 - MCP Gateway:** Centralize all MCP tool connections, governance, security, and auth. Your AI can safely use MCP tools with centralized policy enforcement. [MCP Gateway resource](https://www.getmaxim.ai/resources/mcp-gateway.md). - **05 - Virtual Key Management:** Create different virtual keys for different use cases with independent budgets and access control. - **06 - Unified Interface:** One consistent API for all providers. Switch models without changing code. - **07 - Drop-in Replacement:** Replace your existing SDK with just one line change. Compatible with OpenAI, Anthropic, LiteLLM, Google GenAI, LangChain, and more. [Drop-in replacement docs](https://docs.getbifrost.ai/features/drop-in-replacement). - **08 - Built-in Observability:** Out-of-the-box OpenTelemetry support. Built-in dashboard for quick visibility without complex setup. - **09 - Community Support:** Active Discord community with responsive support and regular updates. ### Enterprise Features - **01 - Governance:** SAML support for SSO and role-based access control with policy enforcement for team collaboration. [Governance resource](https://www.getmaxim.ai/resources/governance.md). - **02 - Adaptive Load Balancing:** Automatically optimizes traffic distribution across provider keys and models based on real-time performance metrics. - **03 - Cluster Mode:** High availability deployment with automatic failover and load balancing. Peer-to-peer clustering where every instance is equal. - **04 - Alerts:** Real-time notifications for budget limits, failures, and performance issues on Email, Slack, PagerDuty, Teams, Webhook, and more. - **05 - Log Exports:** Export and analyze request logs, traces, and telemetry data from Bifrost with enterprise-grade data export for compliance, monitoring, and analytics. - **06 - Audit Logs:** Comprehensive logging and audit trails for compliance and debugging. - **07 - Vault Support:** Secure API key management with HashiCorp Vault, AWS Secrets Manager, Google Secret Manager, and Azure Key Vault integration. - **08 - VPC Deployment:** Deploy Bifrost within your private cloud infrastructure with VPC isolation, custom networking, and enhanced security controls. [Enterprise deployment resource](https://www.getmaxim.ai/resources/enterprise-deployment.md). - **09 - Guardrails:** Automatically detect and block unsafe model outputs with real-time policy enforcement and content moderation across all agents. [Guardrails resource](https://www.getmaxim.ai/resources/guardrails.md). ## Drop-in replacement for compatible AI SDKs Change one line of code to point compatible SDKs at Bifrost. Works with OpenAI, Anthropic, LiteLLM, Google GenAI, LangChain, and Vercel AI SDK. ### OpenAI _openai.py_ ``` import os from openai import OpenAI client = OpenAI( api_key=os.environ.get("OPENAI_API_KEY"), base_url="https:///openai", ) response = client.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": "Hello!"}], ) ``` ### Anthropic _anthropic.py_ ``` import os from anthropic import Anthropic anthropic = Anthropic( api_key=os.environ.get("ANTHROPIC_API_KEY"), base_url="https:///anthropic", ) message = anthropic.messages.create( model="claude-3-5-sonnet-20241022", max_tokens=1024, messages=[{"role": "user", "content": "Hello, Claude"}], ) ``` ### LiteLLM _litellm.py_ ``` import litellm # Set the base URL to your Bifrost deployment litellm.api_base = "https://" response = litellm.completion( model="gpt-4o", messages=[{"role": "user", "content": "Hello!"}], ) ``` ### Google GenAI _genai.py_ ``` import google.generativeai as genai genai.configure( api_key="YOUR_API_KEY", transport="rest", client_options={"api_endpoint": "/google"}, ) model = genai.GenerativeModel("gemini-pro") response = model.generate_content("Hello!") ``` - Point the SDK base URL at your Bifrost deployment. - Keep API keys in your environment or secret manager. - See the docs for provider-specific configuration and deployment steps. - [Gateway setup docs](https://docs.getbifrost.ai/quickstart/gateway/setting-up) - [Drop-in replacement docs](https://docs.getbifrost.ai/features/drop-in-replacement) ## Trust And Deployment - **Open Source:** Bifrost is open source under the Apache 2.0 License. [GitHub](https://github.com/maximhq/bifrost). - **Publisher:** Bifrost is published by H3 Labs Inc. and Maxim AI. - **Compliance:** The site references SOC 2 Type II, GDPR, HIPAA, and ISO 27001 signals. [Enterprise deployment](https://www.getmaxim.ai/resources/enterprise-deployment.md). - **Deployment:** Enterprise resources cover VPC, on-premise, air-gapped, and multi-cloud use. [Enterprise deployment](https://www.getmaxim.ai/resources/enterprise-deployment.md). ## FAQ ### What is Bifrost? Bifrost is the fastest AI and MCP Gateway built for enterprise-grade reliability, governance, and scale. [Docs](https://docs.getbifrost.ai), [GitHub](https://github.com/maximhq/bifrost). ### How is my data protected? Bifrost offers zero-touch in-VPC deployments, so no data ever leaves your environment or passes through Bifrost/Maxim servers. [Governance](https://www.getmaxim.ai/resources/governance.md), [Enterprise deployment](https://www.getmaxim.ai/resources/enterprise-deployment.md). ### Can Bifrost integrate with my existing AI stack? Yes. Bifrost works with major LLM SDKs and frameworks. Compatible SDKs include OpenAI, Anthropic, Mistral, LangChain, LangGraph, and LiteLLM. [Drop-in replacement docs](https://docs.getbifrost.ai/features/drop-in-replacement). ### How much does Bifrost cost? You can get started with the open-source version in seconds for free. Book a demo to learn more about Bifrost Enterprise pricing. [Book a Demo](https://www.getmaxim.ai/book-a-demo). ### How can I get started with Bifrost? You can get started with the open-source version in seconds: npx @maximhq/bifrost [Docs](https://docs.getbifrost.ai), [GitHub](https://github.com/maximhq/bifrost). ## Related Resources - [GitHub: Bifrost OSS](https://github.com/maximhq/bifrost) - [Docs: Bifrost docs](https://docs.getbifrost.ai) - [Resources: Bifrost resources](https://www.getmaxim.ai/resources.md) - [Buyer guide: LLM gateway buyer guide](https://www.getmaxim.ai/resources/buyers-guide.md) - [Enterprise deployment: private deployment](https://www.getmaxim.ai/resources/enterprise-deployment.md) - [Governance: Bifrost governance](https://www.getmaxim.ai/resources/governance.md) - [Guardrails: Bifrost guardrails](https://www.getmaxim.ai/resources/guardrails.md) - [Book a Demo: Bifrost demo](https://www.getmaxim.ai/book-a-demo) --- *This is a markdown version of [https://www.getmaxim.ai/pricing](https://www.getmaxim.ai/pricing) for AI/LLM consumption.* --- title: "Bifrost resources library" description: "Benchmarks, buyer guidance, and integration playbooks for evaluating and deploying Bifrost as an LLM gateway." url: "https://www.getmaxim.ai/bifrost/resources" markdown: "https://www.getmaxim.ai/bifrost/resources.md" --- # Bifrost resources library > Benchmarks, buyer guidance, and integration playbooks for evaluating and deploying Bifrost as an LLM gateway. ## Important Links - [View MCP Gateway](https://www.getmaxim.ai/resources/mcp-gateway.md) - [Features](https://www.getmaxim.ai/#features) - [Enterprise](https://www.getmaxim.ai/enterprise) - [Pricing](https://www.getmaxim.ai/pricing) - [Docs](https://docs.getbifrost.ai) - [GitHub](https://github.com/maximhq/bifrost) - [Book a Demo](https://www.getmaxim.ai/book-a-demo) ## Resources - [Performance Benchmarks](https://www.getmaxim.ai/resources/benchmarks.md): Live comparisons, latency metrics, and throughput data that show why Bifrost is the fastest LLM gateway. - [LLM Gateway Buyer's Guide](https://www.getmaxim.ai/resources/buyers-guide.md): A comprehensive comparison of leading AI gateway platforms, capabilities, and trade-offs. - [Claude Code Integration](https://www.getmaxim.ai/resources/claude-code.md): Enterprise controls for Claude Code with multi-provider routing, governance, and observability. - [CLI Coding Agents](https://www.getmaxim.ai/resources/cli-agents.md): Enterprise controls for Claude Code, Codex CLI, Gemini CLI, and OpenCode with unified governance and multi-provider access. - [Bifrost CLI](https://www.getmaxim.ai/resources/bifrost-cli.md): Interactive terminal tool to launch Claude Code and coding agents through Bifrost with automatic configuration and MCP integration. - [MCP Gateway](https://www.getmaxim.ai/resources/mcp-gateway.md): High-performance tool execution for AI agents with explicit approvals and full audit trails. - [Governance](https://www.getmaxim.ai/resources/governance.md): Virtual keys, budgets, rate limits, routing, MCP tool filtering, and enterprise RBAC with SSO for Okta and Microsoft Entra. - [Guardrails](https://www.getmaxim.ai/resources/guardrails.md): Real-time LLM validation with PII detection, content moderation, prompt injection defense, and multi-provider compliance. - [Migrating from LiteLLM](https://www.getmaxim.ai/resources/migrating-from-litellm.md): Step-by-step guide to move LiteLLM-compatible SDK traffic through Bifrost with no application code changes. - [LiteLLM Alternative](https://www.getmaxim.ai/resources/litellm-alternative.md): Comparison page covering gateway performance, deployment, observability, and governance differences with LiteLLM. - [OSS for Startups](https://www.getmaxim.ai/resources/oss-for-startups.md): Open source LLM gateway with failover, caching, and 20+ providers. Apache 2.0, free for startups and SMEs. - [Enterprise Gateway for Scalability](https://www.getmaxim.ai/resources/enterprise-scalability.md): Scale AI workloads with model routing, circuit breaker, semantic caching, cost analytics, RBAC, guardrails, in-VPC deployment, and MCP gateway. - [Enterprise Deployment](https://www.getmaxim.ai/resources/enterprise-deployment.md): Deploy Bifrost in your VPC, on-premise, air-gapped, or multi-cloud with Terraform, Helm, and zero data egress. - [AWS Bedrock + Bifrost](https://www.getmaxim.ai/resources/aws-bedrock.md): Enterprise governance, guardrails, and multi-region failover for AWS Bedrock with native SDK compatibility. ## Features ### OSS Features - **01 - Model Catalog:** Access 8+ providers and 1000+ AI models through a unified interface. Also supports custom deployed models. - **02 - Budgeting:** Set spending limits and track costs across teams, projects, and models. - **03 - Provider Fallback:** Automatic failover between providers ensures 99.99% uptime for your applications. - **04 - MCP Gateway:** Centralize all MCP tool connections, governance, security, and auth. Your AI can safely use MCP tools with centralized policy enforcement. [MCP Gateway resource](https://www.getmaxim.ai/resources/mcp-gateway.md). - **05 - Virtual Key Management:** Create different virtual keys for different use cases with independent budgets and access control. - **06 - Unified Interface:** One consistent API for all providers. Switch models without changing code. - **07 - Drop-in Replacement:** Replace your existing SDK with just one line change. Compatible with OpenAI, Anthropic, LiteLLM, Google GenAI, LangChain, and more. [Drop-in replacement docs](https://docs.getbifrost.ai/features/drop-in-replacement). - **08 - Built-in Observability:** Out-of-the-box OpenTelemetry support. Built-in dashboard for quick visibility without complex setup. - **09 - Community Support:** Active Discord community with responsive support and regular updates. ### Enterprise Features - **01 - Governance:** SAML support for SSO and role-based access control with policy enforcement for team collaboration. [Governance resource](https://www.getmaxim.ai/resources/governance.md). - **02 - Adaptive Load Balancing:** Automatically optimizes traffic distribution across provider keys and models based on real-time performance metrics. - **03 - Cluster Mode:** High availability deployment with automatic failover and load balancing. Peer-to-peer clustering where every instance is equal. - **04 - Alerts:** Real-time notifications for budget limits, failures, and performance issues on Email, Slack, PagerDuty, Teams, Webhook, and more. - **05 - Log Exports:** Export and analyze request logs, traces, and telemetry data from Bifrost with enterprise-grade data export for compliance, monitoring, and analytics. - **06 - Audit Logs:** Comprehensive logging and audit trails for compliance and debugging. - **07 - Vault Support:** Secure API key management with HashiCorp Vault, AWS Secrets Manager, Google Secret Manager, and Azure Key Vault integration. - **08 - VPC Deployment:** Deploy Bifrost within your private cloud infrastructure with VPC isolation, custom networking, and enhanced security controls. [Enterprise deployment resource](https://www.getmaxim.ai/resources/enterprise-deployment.md). - **09 - Guardrails:** Automatically detect and block unsafe model outputs with real-time policy enforcement and content moderation across all agents. [Guardrails resource](https://www.getmaxim.ai/resources/guardrails.md). ## Drop-in replacement for compatible AI SDKs Change one line of code to point compatible SDKs at Bifrost. Works with OpenAI, Anthropic, LiteLLM, Google GenAI, LangChain, and Vercel AI SDK. ### OpenAI _openai.py_ ``` import os from openai import OpenAI client = OpenAI( api_key=os.environ.get("OPENAI_API_KEY"), base_url="https:///openai", ) response = client.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": "Hello!"}], ) ``` ### Anthropic _anthropic.py_ ``` import os from anthropic import Anthropic anthropic = Anthropic( api_key=os.environ.get("ANTHROPIC_API_KEY"), base_url="https:///anthropic", ) message = anthropic.messages.create( model="claude-3-5-sonnet-20241022", max_tokens=1024, messages=[{"role": "user", "content": "Hello, Claude"}], ) ``` ### LiteLLM _litellm.py_ ``` import litellm # Set the base URL to your Bifrost deployment litellm.api_base = "https://" response = litellm.completion( model="gpt-4o", messages=[{"role": "user", "content": "Hello!"}], ) ``` ### Google GenAI _genai.py_ ``` import google.generativeai as genai genai.configure( api_key="YOUR_API_KEY", transport="rest", client_options={"api_endpoint": "/google"}, ) model = genai.GenerativeModel("gemini-pro") response = model.generate_content("Hello!") ``` - Point the SDK base URL at your Bifrost deployment. - Keep API keys in your environment or secret manager. - See the docs for provider-specific configuration and deployment steps. - [Gateway setup docs](https://docs.getbifrost.ai/quickstart/gateway/setting-up) - [Drop-in replacement docs](https://docs.getbifrost.ai/features/drop-in-replacement) ## Trust And Deployment - **Open Source:** Bifrost is open source under the Apache 2.0 License. [GitHub](https://github.com/maximhq/bifrost). - **Publisher:** Bifrost is published by H3 Labs Inc. and Maxim AI. - **Compliance:** The site references SOC 2 Type II, GDPR, HIPAA, and ISO 27001 signals. [Enterprise deployment](https://www.getmaxim.ai/resources/enterprise-deployment.md). - **Deployment:** Enterprise resources cover VPC, on-premise, air-gapped, and multi-cloud use. [Enterprise deployment](https://www.getmaxim.ai/resources/enterprise-deployment.md). ## FAQ ### What is Bifrost? Bifrost is the fastest AI and MCP Gateway built for enterprise-grade reliability, governance, and scale. [Docs](https://docs.getbifrost.ai), [GitHub](https://github.com/maximhq/bifrost). ### How is my data protected? Bifrost offers zero-touch in-VPC deployments, so no data ever leaves your environment or passes through Bifrost/Maxim servers. [Governance](https://www.getmaxim.ai/resources/governance.md), [Enterprise deployment](https://www.getmaxim.ai/resources/enterprise-deployment.md). ### Can Bifrost integrate with my existing AI stack? Yes. Bifrost works with major LLM SDKs and frameworks. Compatible SDKs include OpenAI, Anthropic, Mistral, LangChain, LangGraph, and LiteLLM. [Drop-in replacement docs](https://docs.getbifrost.ai/features/drop-in-replacement). ### How much does Bifrost cost? You can get started with the open-source version in seconds for free. Book a demo to learn more about Bifrost Enterprise pricing. [Book a Demo](https://www.getmaxim.ai/book-a-demo). ### How can I get started with Bifrost? You can get started with the open-source version in seconds: npx @maximhq/bifrost [Docs](https://docs.getbifrost.ai), [GitHub](https://github.com/maximhq/bifrost). ## Related Resources - [Docs: Bifrost docs](https://docs.getbifrost.ai) - [GitHub: maximhq/bifrost](https://github.com/maximhq/bifrost) - [Pricing: Bifrost pricing](https://www.getmaxim.ai/pricing) - [Enterprise: Bifrost enterprise](https://www.getmaxim.ai/enterprise) - [Book a Demo: Bifrost demo](https://www.getmaxim.ai/book-a-demo) - [Resources: Bifrost resources](https://www.getmaxim.ai/resources.md) --- *This is a markdown version of [https://www.getmaxim.ai/bifrost/resources](https://www.getmaxim.ai/bifrost/resources) for AI/LLM consumption.* --- title: "Bifrost benchmark results" description: "Performance measurements for Bifrost and LiteLLM under sustained load, gateway overhead tests, and high-throughput stress tests." url: "https://www.getmaxim.ai/bifrost/resources/benchmarks" markdown: "https://www.getmaxim.ai/bifrost/resources/benchmarks.md" --- # Bifrost benchmark results > Performance measurements for Bifrost and LiteLLM under sustained load, gateway overhead tests, and high-throughput stress tests. ## Important Links - [View MCP Gateway](https://www.getmaxim.ai/resources/mcp-gateway.md) - [Features](https://www.getmaxim.ai/#features) - [Enterprise](https://www.getmaxim.ai/enterprise) - [Pricing](https://www.getmaxim.ai/pricing) - [Docs](https://docs.getbifrost.ai) - [GitHub](https://github.com/maximhq/bifrost) - [Book a Demo](https://www.getmaxim.ai/book-a-demo) ## Benchmark summary The benchmark compares Bifrost and LiteLLM using sustained load tests, gateway overhead measurements, high-throughput stress tests, and architecture-level differences. Interactive charts use generated samples around the published benchmark values. Use the tables below for the fixed benchmark figures. - **Throughput:** Bifrost processed 424 req/s and LiteLLM processed 44.84 req/s in the 500 RPS load test on AWS EC2. This is 9.5x higher throughput in the published data. - **Success rate:** Bifrost completed 100% of requests and LiteLLM completed 88.78% of requests in the 500 RPS load test. - **P50 latency:** Bifrost measured 804ms P50 latency and LiteLLM measured 38.65s P50 latency in the 500 RPS load test. - **P99 latency:** Bifrost measured 1.68s P99 latency and LiteLLM measured 90.72s P99 latency in the 500 RPS load test. This is 54x lower P99 latency in the published data. - **Memory:** Bifrost used 120 MB peak memory and LiteLLM used 372 MB peak memory in the 500 RPS load test. This is 68% lower peak memory in the published data. - **Gateway overhead:** Bifrost measured 0.99ms gateway overhead and LiteLLM measured 40ms gateway overhead with a 60ms mock OpenAI upstream. This is 40x lower gateway overhead in the published data. ## Performance at a glance | Signal | Result | Condition | | --- | --- | --- | | Throughput | 9.5x higher throughput | 424 req/s for Bifrost and 44.84 req/s for LiteLLM in the 500 RPS load test. | | P99 latency | 54x lower P99 latency | 1.68s for Bifrost and 90.72s for LiteLLM in the 500 RPS load test. | | Peak memory | 68% lower peak memory | 120 MB for Bifrost and 372 MB for LiteLLM in the 500 RPS load test. | | Gateway overhead | 40x lower gateway overhead | 0.99ms for Bifrost and 40ms for LiteLLM with a 60ms mock OpenAI upstream. | ## Test environment These fields describe the 500 RPS Bifrost vs LiteLLM load test environment. | Field | Value | | --- | --- | | Instance | t3.medium | | CPU | 2 vCPU | | Memory | 4GB RAM | | Provider | AWS EC2 | | Region | us-east-1 | | OpenAI tier | Tier 5 | | Duration | 60 seconds | | Concurrent users | 500 VUs | ## 500 RPS load test | Metric | Bifrost | LiteLLM | Description | | --- | --- | --- | --- | | Success Rate | 100% | 88.78% | Percentage of requests completed successfully | | P50 Latency | 804ms | 38.65s | Median response time | | P99 Latency | 1.68s | 90.72s | 99th percentile response time | | Max Latency | 6.13s | 92.67s | Maximum observed response time | | Throughput | 424 req/s | 44.84 req/s | Requests processed per second | | Peak Memory | 120MB | 372MB | Maximum memory consumption | ## Gateway overhead Gateway overhead excludes the 60ms mock OpenAI response and measures the proxy processing time added by the gateway. | Metric | Bifrost | LiteLLM | Description | | --- | --- | --- | --- | | Median Latency | 60.99ms | 100ms | Median end-to-end latency | | Gateway Overhead | 0.99ms | 40ms | Internal processing time (excluding 60ms mock OpenAI call) | | RPS Capacity | 500 req/s | 475 req/s | Maximum sustainable requests per second | ## High-throughput stress test The 5,000 RPS stress test records Bifrost-only internal overhead on two AWS EC2 instance sizes. - **t3.medium:** 59µs internal overhead at 5,000 RPS with 100% success rate. - **t3.xlarge:** 11µs internal overhead at 5,000 RPS with 100% success rate. ## Architecture comparison | Feature | Bifrost | LiteLLM | | --- | --- | --- | | Language | Go | Python | | Async Runtime | Goroutines | asyncio | | HTTP Server | Fast http | FastAPI/Uvicorn | | Memory Model | Efficient GC | GC-managed | | Concurrency | Native goroutines | GIL-limited | | Binary Size | ~80MB | ~500MB+ (with deps) | | Open Source | Yes (Apache 2.0) | Yes (MIT) | ## Key insights - **Optimized architecture:** Bifrost uses Go, efficient parsing, and memory-optimized data structures to reduce allocations. - **Native concurrency:** Bifrost uses Go goroutines to handle concurrent connections without the Python GIL bottleneck. - **Efficient memory model:** Bifrost uses Go memory management to maintain consistent performance under load while using less memory in the benchmark. ## Full analysis Read the full benchmark analysis for methodology, test configuration, and detailed performance discussion. - [Read full benchmark analysis](https://www.getmaxim.ai/blog/bifrost-a-drop-in-llm-proxy-40x-faster-than-litellm/) ## Features ### OSS Features - **01 - Model Catalog:** Access 8+ providers and 1000+ AI models through a unified interface. Also supports custom deployed models. - **02 - Budgeting:** Set spending limits and track costs across teams, projects, and models. - **03 - Provider Fallback:** Automatic failover between providers ensures 99.99% uptime for your applications. - **04 - MCP Gateway:** Centralize all MCP tool connections, governance, security, and auth. Your AI can safely use MCP tools with centralized policy enforcement. [MCP Gateway resource](https://www.getmaxim.ai/resources/mcp-gateway.md). - **05 - Virtual Key Management:** Create different virtual keys for different use cases with independent budgets and access control. - **06 - Unified Interface:** One consistent API for all providers. Switch models without changing code. - **07 - Drop-in Replacement:** Replace your existing SDK with just one line change. Compatible with OpenAI, Anthropic, LiteLLM, Google GenAI, LangChain, and more. [Drop-in replacement docs](https://docs.getbifrost.ai/features/drop-in-replacement). - **08 - Built-in Observability:** Out-of-the-box OpenTelemetry support. Built-in dashboard for quick visibility without complex setup. - **09 - Community Support:** Active Discord community with responsive support and regular updates. ### Enterprise Features - **01 - Governance:** SAML support for SSO and role-based access control with policy enforcement for team collaboration. [Governance resource](https://www.getmaxim.ai/resources/governance.md). - **02 - Adaptive Load Balancing:** Automatically optimizes traffic distribution across provider keys and models based on real-time performance metrics. - **03 - Cluster Mode:** High availability deployment with automatic failover and load balancing. Peer-to-peer clustering where every instance is equal. - **04 - Alerts:** Real-time notifications for budget limits, failures, and performance issues on Email, Slack, PagerDuty, Teams, Webhook, and more. - **05 - Log Exports:** Export and analyze request logs, traces, and telemetry data from Bifrost with enterprise-grade data export for compliance, monitoring, and analytics. - **06 - Audit Logs:** Comprehensive logging and audit trails for compliance and debugging. - **07 - Vault Support:** Secure API key management with HashiCorp Vault, AWS Secrets Manager, Google Secret Manager, and Azure Key Vault integration. - **08 - VPC Deployment:** Deploy Bifrost within your private cloud infrastructure with VPC isolation, custom networking, and enhanced security controls. [Enterprise deployment resource](https://www.getmaxim.ai/resources/enterprise-deployment.md). - **09 - Guardrails:** Automatically detect and block unsafe model outputs with real-time policy enforcement and content moderation across all agents. [Guardrails resource](https://www.getmaxim.ai/resources/guardrails.md). ## Drop-in replacement for compatible AI SDKs Change one line of code to point compatible SDKs at Bifrost. Works with OpenAI, Anthropic, LiteLLM, Google GenAI, LangChain, and Vercel AI SDK. ### OpenAI _openai.py_ ``` import os from openai import OpenAI client = OpenAI( api_key=os.environ.get("OPENAI_API_KEY"), base_url="https:///openai", ) response = client.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": "Hello!"}], ) ``` ### Anthropic _anthropic.py_ ``` import os from anthropic import Anthropic anthropic = Anthropic( api_key=os.environ.get("ANTHROPIC_API_KEY"), base_url="https:///anthropic", ) message = anthropic.messages.create( model="claude-3-5-sonnet-20241022", max_tokens=1024, messages=[{"role": "user", "content": "Hello, Claude"}], ) ``` ### LiteLLM _litellm.py_ ``` import litellm # Set the base URL to your Bifrost deployment litellm.api_base = "https://" response = litellm.completion( model="gpt-4o", messages=[{"role": "user", "content": "Hello!"}], ) ``` ### Google GenAI _genai.py_ ``` import google.generativeai as genai genai.configure( api_key="YOUR_API_KEY", transport="rest", client_options={"api_endpoint": "/google"}, ) model = genai.GenerativeModel("gemini-pro") response = model.generate_content("Hello!") ``` - Point the SDK base URL at your Bifrost deployment. - Keep API keys in your environment or secret manager. - See the docs for provider-specific configuration and deployment steps. - [Gateway setup docs](https://docs.getbifrost.ai/quickstart/gateway/setting-up) - [Drop-in replacement docs](https://docs.getbifrost.ai/features/drop-in-replacement) ## Trust And Deployment - **Benchmark methodology:** The full analysis documents methodology, test configuration, and detailed performance discussion for the benchmark data. [Full benchmark analysis](https://www.getmaxim.ai/blog/bifrost-a-drop-in-llm-proxy-40x-faster-than-litellm/). ## FAQ ### How were the Bifrost vs LiteLLM benchmarks conducted? The 500 RPS load test used AWS EC2 t3.medium instances in us-east-1 with 2 vCPU, 4GB RAM, Tier 5 OpenAI access, a 60-second test duration, and 500 concurrent virtual users. Gateway overhead tests used a 60ms mock OpenAI response. The 5,000 RPS stress test reports Bifrost internal overhead on t3.medium and t3.xlarge instances. ### Why is Bifrost so much faster than LiteLLM? Bifrost is built in Go, which compiles to native machine code and uses goroutines for lightweight concurrency. LiteLLM is Python-based, which means it's subject to the Global Interpreter Lock (GIL), asyncio overhead, and higher memory consumption from dynamic typing and garbage collection. ### Does switching from LiteLLM to Bifrost require code changes? No. Bifrost provides an OpenAI-compatible API, so you only need to change the base URL in your application. The same SDKs, request formats, and response structures work without modification. ### What does "gateway overhead" mean in these benchmarks? Gateway overhead measures the additional latency the proxy adds on top of the actual LLM provider response time. In the gateway overhead test with a 60ms mock OpenAI response, Bifrost measured 0.99ms overhead. In the 5,000 RPS stress test on t3.xlarge, Bifrost measured 11µs internal overhead. ### Can Bifrost handle higher throughput than what the benchmarks show? The published benchmark data covers a 500 RPS Bifrost vs LiteLLM load test and Bifrost-only 5,000 RPS stress tests on t3.medium and t3.xlarge instances. Bifrost maintains a 100% success rate in the published 5,000 RPS stress tests. Teams should validate higher targets on their own instance sizes, provider mix, and payload profile. ## Related Resources - [Blog: Full benchmark analysis](https://www.getmaxim.ai/blog/bifrost-a-drop-in-llm-proxy-40x-faster-than-litellm/) - [Migration: Migrating from LiteLLM](https://www.getmaxim.ai/resources/migrating-from-litellm.md) - [Alternative: LiteLLM Alternative](https://www.getmaxim.ai/resources/litellm-alternative.md) - [Docs: Bifrost docs](https://docs.getbifrost.ai) - [GitHub: maximhq/bifrost](https://github.com/maximhq/bifrost) - [Pricing: Bifrost pricing](https://www.getmaxim.ai/pricing) - [Enterprise: Bifrost enterprise](https://www.getmaxim.ai/enterprise) - [Book a Demo: Bifrost demo](https://www.getmaxim.ai/book-a-demo) - [Resources: Bifrost resources](https://www.getmaxim.ai/resources.md) --- *This is a markdown version of [https://www.getmaxim.ai/bifrost/resources/benchmarks](https://www.getmaxim.ai/bifrost/resources/benchmarks) for AI/LLM consumption.* --- title: "Bifrost LiteLLM alternative" description: "LLM gateway comparison between Bifrost and LiteLLM. Covers OpenAI-compatible routing, Go vs Python gateway architecture, setup steps, observability, governance, provider coverage, and published benchmark context." url: "https://www.getmaxim.ai/bifrost/resources/litellm-alternative" markdown: "https://www.getmaxim.ai/bifrost/resources/litellm-alternative.md" --- # Bifrost LiteLLM alternative > LLM gateway comparison between Bifrost and LiteLLM. Covers OpenAI-compatible routing, Go vs Python gateway architecture, setup steps, observability, governance, provider coverage, and published benchmark context. ## Important Links - [View MCP Gateway](https://www.getmaxim.ai/resources/mcp-gateway.md) - [Features](https://www.getmaxim.ai/#features) - [Enterprise](https://www.getmaxim.ai/enterprise) - [Pricing](https://www.getmaxim.ai/pricing) - [Docs](https://docs.getbifrost.ai) - [GitHub](https://github.com/maximhq/bifrost) - [Book a Demo](https://www.getmaxim.ai/book-a-demo) ## Evaluation summary Bifrost is a Go-native LLM gateway for production workloads that need low gateway overhead, simple deployment, native observability, and built-in governance. Bifrost and LiteLLM both provide an OpenAI-compatible API. The practical migration path is usually a base URL change plus provider and key configuration in Bifrost. > **Main distinction.** Bifrost is implemented in Go as a gateway runtime with built-in state, metrics, logging, configuration, and routing. LiteLLM is Python-based and emphasizes broad provider abstraction. - [Migration guide](https://www.getmaxim.ai/resources/migrating-from-litellm.md) ## Performance context Bifrost benchmark results are published with sustained-load test conditions. The benchmark page documents methodology, test environment, and detailed metric context. Benchmarked on production infrastructure under sustained load, Bifrost holds a 100% success rate with sub-11µs overhead at 5,000 RPS. - **11µs Gateway overhead.** for Bifrost on a t3.xlarge at 5,000 RPS in the published benchmark. [Benchmark details](https://www.getmaxim.ai/resources/benchmarks.md). - **100% Success rate.** for Bifrost even under extreme load at 5,000 RPS in the published benchmark. - **<30 seconds Setup time.** for the quick-start path using npx or Docker. - **20+ providers Provider coverage.** and 1,000+ models in Bifrost. LiteLLM is listed as 100+ LLM APIs. - **Zero required Runtime dependencies.** for the basic Bifrost gateway path. Redis is recommended for some LiteLLM production setups. ## Why teams compare Bifrost and LiteLLM | Challenge | Bifrost framing | | --- | --- | | High latency at scale | Bifrost uses Go-native concurrency for high-throughput gateway workloads. | | Infrastructure bottlenecks | Bifrost uses connection pooling and avoids Python GIL limitations in the gateway runtime. | | Memory consumption | Bifrost uses Go memory management and lightweight goroutines. | | Complex self-hosting | Bifrost can start through npx or Docker without Redis or Postgres for the basic path. | | Limited observability | Bifrost includes native Prometheus metrics and OpenTelemetry built-in, not bolted on. | | Production reliability | Bifrost holds a 100% success rate at 5,000 RPS with <11µs overhead. | ## Architecture comparison | Factor | Bifrost | LiteLLM | | --- | --- | --- | | Language | Go (compiled) | Python (interpreted) | | Gateway overhead | 11µs | ~40ms | | Concurrency model | Native goroutines | Async/await with GIL | | State and configuration | Built-in state management | Files, env vars, or admin UI | | External dependencies | Zero | Redis recommended | | Deployment asset | Single binary, Docker, npx | Python package or Docker | | Plugin extension model | Go-based plugins | Python callbacks | | License | Apache 2.0 | MIT | ## Capability comparison | Capability | Bifrost | LiteLLM | | --- | --- | --- | | Provider support | 20+ providers and 1,000+ models | 100+ LLM APIs | | OpenAI-compatible API | Supported | Supported | | Automatic failover | Adaptive load balancing and fallback routing | Retry logic and configuration | | Semantic caching | Built-in | Available through external integration | | Web UI | Built-in dashboard | No built-in dashboard in the compared LiteLLM setup | | Prometheus metrics | Native | Available | | OpenTelemetry and distributed tracing | Built in | Available through integrations | | Request logging | Built-in logging backend | Available through configuration | | Budgets and rate limits | Virtual keys with spending limits, model restrictions, and rate limits | Team and user budget controls | | Audit logs | Built in | Available | ## Production controls - **Cost control - Virtual keys with budgets:** Create keys with spend limits, model restrictions, and rate limits per team or use case. - **Observability - Native Prometheus metrics:** Expose gateway metrics for requests, latency, provider health, and memory usage. - **Tracing - OpenTelemetry tracing:** Send traces to Jaeger or an OTEL collector without adding a separate gateway-side tracing layer. - **Web UI - Real-time dashboard:** Monitor spend by key, model, and team through the Bifrost web UI. - **Routing - Adaptive load balancing:** Distribute requests based on current success rates, latency patterns, and provider capacity. - **Reliability - Automatic failover:** Route to configured fallback providers when the primary provider fails or degrades. ## Setup Steps 1. **01 - Install Bifrost.** One command. No configuration files, no Redis, no databases required. ### Install Bifrost ``` # Option 1: NPX npx -y @maximhq/bifrost # Option 2: Docker docker run -p 8080:8080 maximhq/bifrost # Option 3: Go SDK go get github.com/maximhq/bifrost/core@latest ``` 2. **02 - Configure via Web UI.** Add provider keys, configure models, set up fallback chains, all from the browser. ### Configure via Web UI ``` # open the dashboard open http://localhost:8080 # add API keys for providers # configure models and weights # set up fallback chains ``` 3. **03 - Update your endpoint.** Change the base URL in your code. Everything else stays the same. ### Update your endpoint ``` # just update the base URL # before: http://localhost:4000 # after: http://localhost:8080 curl http://localhost:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model":"openai/gpt-4o-mini","messages":[{"role":"user","content":"Hello!"}]}' ``` ## Decision guide | Question | Choose Bifrost when | LiteLLM may fit when | | --- | --- | --- | | Performance profile | You need low gateway overhead for high-throughput production traffic. | Your workloads are not gateway-latency sensitive. | | Operations model | You want a single gateway binary with no required Redis or database dependency for the basic path. | Your team already operates a Python service stack and related dependencies. | | Provider strategy | You need the providers and models supported by Bifrost plus routing and governance controls. | You primarily need the broadest provider abstraction list. | | Configuration workflow | You want web UI, API, or file-based configuration with hot changes. | You are comfortable maintaining LiteLLM config files and environment variables. | | Governance needs | You need virtual keys, budgets, rate limits, audit logs, and observability in one gateway. | You already have equivalent governance layers outside the gateway. | ## Features ### OSS Features - **01 - Model Catalog:** Access 8+ providers and 1000+ AI models through a unified interface. Also supports custom deployed models. - **02 - Budgeting:** Set spending limits and track costs across teams, projects, and models. - **03 - Provider Fallback:** Automatic failover between providers ensures 99.99% uptime for your applications. - **04 - MCP Gateway:** Centralize all MCP tool connections, governance, security, and auth. Your AI can safely use MCP tools with centralized policy enforcement. [MCP Gateway resource](https://www.getmaxim.ai/resources/mcp-gateway.md). - **05 - Virtual Key Management:** Create different virtual keys for different use cases with independent budgets and access control. - **06 - Unified Interface:** One consistent API for all providers. Switch models without changing code. - **07 - Drop-in Replacement:** Replace your existing SDK with just one line change. Compatible with OpenAI, Anthropic, LiteLLM, Google GenAI, LangChain, and more. [Drop-in replacement docs](https://docs.getbifrost.ai/features/drop-in-replacement). - **08 - Built-in Observability:** Out-of-the-box OpenTelemetry support. Built-in dashboard for quick visibility without complex setup. - **09 - Community Support:** Active Discord community with responsive support and regular updates. ### Enterprise Features - **01 - Governance:** SAML support for SSO and role-based access control with policy enforcement for team collaboration. [Governance resource](https://www.getmaxim.ai/resources/governance.md). - **02 - Adaptive Load Balancing:** Automatically optimizes traffic distribution across provider keys and models based on real-time performance metrics. - **03 - Cluster Mode:** High availability deployment with automatic failover and load balancing. Peer-to-peer clustering where every instance is equal. - **04 - Alerts:** Real-time notifications for budget limits, failures, and performance issues on Email, Slack, PagerDuty, Teams, Webhook, and more. - **05 - Log Exports:** Export and analyze request logs, traces, and telemetry data from Bifrost with enterprise-grade data export for compliance, monitoring, and analytics. - **06 - Audit Logs:** Comprehensive logging and audit trails for compliance and debugging. - **07 - Vault Support:** Secure API key management with HashiCorp Vault, AWS Secrets Manager, Google Secret Manager, and Azure Key Vault integration. - **08 - VPC Deployment:** Deploy Bifrost within your private cloud infrastructure with VPC isolation, custom networking, and enhanced security controls. [Enterprise deployment resource](https://www.getmaxim.ai/resources/enterprise-deployment.md). - **09 - Guardrails:** Automatically detect and block unsafe model outputs with real-time policy enforcement and content moderation across all agents. [Guardrails resource](https://www.getmaxim.ai/resources/guardrails.md). ## FAQ ### Is Bifrost a drop-in replacement for LiteLLM? Yes. Bifrost provides an OpenAI-compatible API, so migrating from LiteLLM typically requires changing only the base URL. Your existing SDKs, request formats, and integrations continue to work without code changes. ### Why is Bifrost faster than LiteLLM? Bifrost is built in Go, a compiled language with native concurrency via goroutines. LiteLLM is Python-based, which introduces interpreter overhead, GIL limitations, and higher memory consumption. This architectural difference results in Bifrost delivering 11µs gateway overhead compared to LiteLLM's approximately 40 milliseconds. ### Does Bifrost require Redis or external databases? No. Bifrost handles configuration, caching, logging, and state management internally with zero external dependencies. You can start with a single command (npx or Docker) and have a fully functional gateway in under 30 seconds. ### How does Bifrost compare to LiteLLM for provider support? Bifrost supports 20+ providers and 1,000+ models out of the box including OpenAI, Anthropic, AWS Bedrock, Google Vertex AI, Azure OpenAI, and more. LiteLLM supports 100+ LLM APIs. Both cover the major providers teams use in production. ### Is Bifrost open source? Yes. Bifrost is fully open source under the Apache 2.0 license with the complete source code available on GitHub. There is also an enterprise tier with additional features like SSO, clustering, and premium support. ## Related Resources - [Benchmark: Bifrost vs LiteLLM](https://www.getmaxim.ai/resources/benchmarks.md) - [Migration: Migrating from LiteLLM](https://www.getmaxim.ai/resources/migrating-from-litellm.md) - [Source: Litellm Alternative](https://www.getmaxim.ai/resources/litellm-alternative.md) - [Docs: Bifrost docs](https://docs.getbifrost.ai) - [GitHub: maximhq/bifrost](https://github.com/maximhq/bifrost) - [Pricing: Bifrost pricing](https://www.getmaxim.ai/pricing) - [Enterprise: Bifrost enterprise](https://www.getmaxim.ai/enterprise) - [Book a Demo: Bifrost demo](https://www.getmaxim.ai/book-a-demo) - [Resources: Bifrost resources](https://www.getmaxim.ai/resources.md) --- *This is a markdown version of [https://www.getmaxim.ai/bifrost/resources/litellm-alternative](https://www.getmaxim.ai/bifrost/resources/litellm-alternative) for AI/LLM consumption.* --- title: "Migrating from LiteLLM to Bifrost" description: "Migration guide for moving LiteLLM-compatible traffic to Bifrost. Covers base URL changes, provider setup, virtual key migration, and benchmark context from the 500 RPS LiteLLM comparison." url: "https://www.getmaxim.ai/bifrost/resources/migrating-from-litellm" markdown: "https://www.getmaxim.ai/bifrost/resources/migrating-from-litellm.md" --- # Migrating from LiteLLM to Bifrost > Migration guide for moving LiteLLM-compatible traffic to Bifrost. Covers base URL changes, provider setup, virtual key migration, and benchmark context from the 500 RPS LiteLLM comparison. ## Important Links - [View MCP Gateway](https://www.getmaxim.ai/resources/mcp-gateway.md) - [Features](https://www.getmaxim.ai/#features) - [Enterprise](https://www.getmaxim.ai/enterprise) - [Pricing](https://www.getmaxim.ai/pricing) - [Docs](https://docs.getbifrost.ai) - [GitHub](https://github.com/maximhq/bifrost) - [Book a Demo](https://www.getmaxim.ai/book-a-demo) ## Performance at a Glance - **54x Lower P99 Latency.** Consistently fast response times - **99.999% Uptime SLA.** Automatic failover & circuit breakers - **1000+ Providers.** models supported natively - **15 min Migration Time.** Drop-in OpenAI-compatible API ## Why Migrate to Bifrost? While LiteLLM works well for prototyping, teams scaling to production need infrastructure that doesn't become a bottleneck. - **Performance - 54x Faster Performance:** Built in Go with just 1.68s overhead at 500 RPS compared to 90.72s for Python-based solutions (a ~54x lower P99 latency). Your gateway stops being the bottleneck. - **Reliability - Production-Ready Reliability:** 99.999% uptime SLA with automatic failover, circuit breakers, and intelligent retry logic. No more multi-minute latency spikes at high load. - **Cost control - Cost Optimization:** Semantic caching reduces costs and latency on repeated queries. Adaptive load balancing ensures efficient resource utilization. - **Security - Enterprise Security:** Virtual keys with budgets, RBAC, audit logs, and in-VPC deployments. Full control over your AI infrastructure. - **Observability - Native Observability:** Built-in Prometheus metrics, OpenTelemetry support, and integration with Maxim's evaluation platform. No sidecars needed. - **Migration - Drop-in Replacement:** OpenAI-compatible API means zero code changes. Point your existing LiteLLM integration to Bifrost and you're done. ## Performance Comparison Tested on identical AWS t3.xlarge instances. Bifrost delivers consistent, predictable performance under load. | Metric | Bifrost | Litellm | | --- | --- | --- | | Overhead per Request (500 RPS) | 11µs | ~40ms | | P99 Latency at 500 RPS | 1.68s | 90.72s | | Maximum Sustained RPS | 5,000+ stable | Fails at high load | > **Read the Bifrost vs LiteLLM Deep Dive.** Learn how Bifrost stacks up against LiteLLM on throughput, latency, and gateway overhead. [Read Full Article](https://www.getmaxim.ai/blog/bifrost-a-drop-in-llm-proxy-40x-faster-than-litellm/). ## Feature Comparison | Title | Rows | | --- | --- | | Performance | Feature: Overhead at 500 RPS; Bifrost: 11µs (Go-native); Litellm: 40ms (Python GIL); Feature: Concurrent Request Handling; Bifrost: Native Go concurrency; Litellm: Async overhead | | Reliability | Feature: Automatic Failover; Bifrost: Zero-config; Litellm: Manual config; Feature: Circuit Breakers; Bifrost: Available; Litellm: N/A; Feature: Health Monitoring; Bifrost: Real-time; Litellm: Basic | | Governance & Security | Feature: Virtual Keys; Bifrost: With budgets & rate limits; Litellm: Available; Feature: RBAC; Bifrost: Fine-grained access management; Litellm: Available; Feature: Audit Logs; Bifrost: Available; Litellm: Available; Feature: Guardrails; Bifrost: Available; Litellm: Available; Feature: In-VPC Deployment; Bifrost: Available; Litellm: Available | | Observability | Feature: Prometheus Metrics; Bifrost: Native, no sidecars; Litellm: Via callbacks; Feature: OpenTelemetry; Bifrost: OTel compatible; Litellm: OTel compatible; Feature: Request Logging; Bifrost: Multiple backends; Litellm: Multiple backends | | Developer Experience | Feature: Setup Time; Bifrost: 30 seconds (NPX or Docker); Litellm: 5-10 minute setup; Feature: Web UI; Bifrost: Real-time config; Litellm: Admin panel available; Feature: Configuration; Bifrost: Web UI, API, or file-based; Litellm: Web UI, API, or file-based; Feature: MCP Support; Bifrost: Native gateway; Litellm: Beta integration; Feature: Deployment Asset; Bifrost: Single binary, Docker, K8s; Litellm: Python package, Docker; Feature: Docker Size; Bifrost: 80 MB; Litellm: > 700 MB | | Architecture | Feature: Language; Bifrost: Go; Litellm: Python; Feature: Clustering; Bifrost: Available; Litellm: N/A; Feature: Adaptive load Balancing; Bifrost: Dynamic weight adjustment; Litellm: N/A; Feature: Usage-Based Routing Rules; Bifrost: Yes; Litellm: N/A; Feature: Plugin System; Bifrost: Go-based; Litellm: Python callbacks; Feature: License; Bifrost: Apache 2.0; Litellm: MIT | ## Setup Steps - **Zero code changes:** OpenAI-compatible API means your existing integrations work as-is. - **LiteLLM SDK compatible:** You can even point the LiteLLM Python SDK at Bifrost as a proxy. - **Provider prefix routing:** Use the openai/gpt-4o format for explicit provider control. 1. **01 - Install Bifrost.** Choose your preferred method. Bifrost starts immediately with zero configuration needed. ### Install Bifrost ``` # Option 1: NPX (fastest) npx -y @maximhq/bifrost # Option 2: Docker docker pull maximhq/bifrost docker run -p 8080:8080 maximhq/bifrost ``` 2. **02 - Configure providers.** Add your LLM provider API keys via the web UI at localhost:8080 or a configuration file. ### Configure providers ``` # navigate to http://localhost:8080 # click "Providers" in the sidebar # add API keys for OpenAI, Anthropic, etc. # configure models and fallback chains ``` 3. **03 - Update base URL.** Change one line in your application. Bifrost's OpenAI-compatible API means zero other code changes. ### Update base URL ``` # Before (LiteLLM) # base_url="http://localhost:4000" # After (Bifrost) base_url="http://localhost:8080" ``` ## One Line Change ### Before (LiteLLM) ```python import openai client = openai.OpenAI( api_key="your-litellm-key", base_url="http://localhost:4000" ) response = client.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": "Hello!"}] ) ``` ### After (Bifrost) ```python import openai client = openai.OpenAI( api_key="your-bifrost-key", base_url="http://localhost:8080" ) response = client.chat.completions.create( model="openai/gpt-4o", messages=[{"role": "user", "content": "Hello!"}] ) ``` > Bifrost uses the provider/model format (e.g., openai/gpt-4o) for explicit routing control. ## Common Migration Scenarios ### Migrating Virtual Keys ```bash # LiteLLM virtual keys for team budgets map directly # to Bifrost's equivalent functionality. curl -X POST http://localhost:8080/api/keys \ -H "Content-Type: application/json" \ -d '{ "name": "team-engineering", "budget": 1000, "rate_limit": 100, "models": ["openai/gpt-4o", "anthropic/claude-sonnet-4-20250514"] }' ``` ### Drop-in Replacement ```python # Use the standard OpenAI SDK pointed at Bifrost. import openai client = openai.OpenAI( base_url="http://localhost:8080", api_key="your-key" ) ``` ### LiteLLM SDK Compatibility ```python # Use the LiteLLM Python SDK with Bifrost as the proxy backend. import litellm litellm.api_base = "http://localhost:8080/litellm" response = litellm.completion( model="openai/gpt-4o", messages=[{"role": "user", "content": "Hello!"}] ) ``` ## You Should Migrate If - **Scaling beyond prototyping:** Performance matters at production traffic levels. - **Building multi-step agent architectures:** Overhead compounds with each LLM call. - **Need enterprise governance:** Budget management, access control, and audit trails. - **Want integrated observability:** Maxim platform provides unmatched visibility. - **Experiencing reliability issues:** Timeout spikes, memory issues, or unpredictable latency. - **Need better cost control:** Semantic caching and adaptive load balancing. ## Features ### OSS Features - **01 - Model Catalog:** Access 8+ providers and 1000+ AI models through a unified interface. Also supports custom deployed models. - **02 - Budgeting:** Set spending limits and track costs across teams, projects, and models. - **03 - Provider Fallback:** Automatic failover between providers ensures 99.99% uptime for your applications. - **04 - MCP Gateway:** Centralize all MCP tool connections, governance, security, and auth. Your AI can safely use MCP tools with centralized policy enforcement. [MCP Gateway resource](https://www.getmaxim.ai/resources/mcp-gateway.md). - **05 - Virtual Key Management:** Create different virtual keys for different use cases with independent budgets and access control. - **06 - Unified Interface:** One consistent API for all providers. Switch models without changing code. - **07 - Drop-in Replacement:** Replace your existing SDK with just one line change. Compatible with OpenAI, Anthropic, LiteLLM, Google GenAI, LangChain, and more. [Drop-in replacement docs](https://docs.getbifrost.ai/features/drop-in-replacement). - **08 - Built-in Observability:** Out-of-the-box OpenTelemetry support. Built-in dashboard for quick visibility without complex setup. - **09 - Community Support:** Active Discord community with responsive support and regular updates. ### Enterprise Features - **01 - Governance:** SAML support for SSO and role-based access control with policy enforcement for team collaboration. [Governance resource](https://www.getmaxim.ai/resources/governance.md). - **02 - Adaptive Load Balancing:** Automatically optimizes traffic distribution across provider keys and models based on real-time performance metrics. - **03 - Cluster Mode:** High availability deployment with automatic failover and load balancing. Peer-to-peer clustering where every instance is equal. - **04 - Alerts:** Real-time notifications for budget limits, failures, and performance issues on Email, Slack, PagerDuty, Teams, Webhook, and more. - **05 - Log Exports:** Export and analyze request logs, traces, and telemetry data from Bifrost with enterprise-grade data export for compliance, monitoring, and analytics. - **06 - Audit Logs:** Comprehensive logging and audit trails for compliance and debugging. - **07 - Vault Support:** Secure API key management with HashiCorp Vault, AWS Secrets Manager, Google Secret Manager, and Azure Key Vault integration. - **08 - VPC Deployment:** Deploy Bifrost within your private cloud infrastructure with VPC isolation, custom networking, and enhanced security controls. [Enterprise deployment resource](https://www.getmaxim.ai/resources/enterprise-deployment.md). - **09 - Guardrails:** Automatically detect and block unsafe model outputs with real-time policy enforcement and content moderation across all agents. [Guardrails resource](https://www.getmaxim.ai/resources/guardrails.md). ## FAQ ### How long does it take to migrate from LiteLLM to Bifrost? Most migrations take 15-30 minutes. Since Bifrost provides an OpenAI-compatible API, you typically only need to change the base URL in your application. Provider configuration can be done through the web UI without editing config files. ### Will my existing LiteLLM virtual keys work with Bifrost? LiteLLM virtual keys need to be recreated in Bifrost, but the concepts map directly. Bifrost virtual keys support the same functionality including team budgets, rate limits, and model restrictions. You can configure them via the web UI or API. ### Can I use the LiteLLM Python SDK with Bifrost? Yes. You can point the LiteLLM Python SDK at Bifrost by setting the api_base to your Bifrost URL. This allows a gradual migration where you swap the backend without changing application code. ### What happens to my LiteLLM model routing configuration? Bifrost uses a provider/model format (e.g., openai/gpt-4o) for explicit routing. You can configure fallback chains, load balancing weights, and routing rules through Bifrost's web UI or configuration files. ### Is there any downtime during migration? No. You can run Bifrost alongside LiteLLM during migration and switch traffic gradually. Both gateways can operate simultaneously, allowing you to validate Bifrost performance before fully cutting over. ## Related Resources - [Benchmark: Bifrost vs LiteLLM](https://www.getmaxim.ai/resources/benchmarks.md) - [Alternative: LiteLLM Alternative](https://www.getmaxim.ai/resources/litellm-alternative.md) - [Source: Migrating From Litellm](https://www.getmaxim.ai/resources/migrating-from-litellm.md) - [Docs: Bifrost docs](https://docs.getbifrost.ai) - [GitHub: maximhq/bifrost](https://github.com/maximhq/bifrost) - [Pricing: Bifrost pricing](https://www.getmaxim.ai/pricing) - [Enterprise: Bifrost enterprise](https://www.getmaxim.ai/enterprise) - [Book a Demo: Bifrost demo](https://www.getmaxim.ai/book-a-demo) - [Resources: Bifrost resources](https://www.getmaxim.ai/resources.md) --- *This is a markdown version of [https://www.getmaxim.ai/bifrost/resources/migrating-from-litellm](https://www.getmaxim.ai/bifrost/resources/migrating-from-litellm) for AI/LLM consumption.* --- title: "LLM gateway buyer's guide" description: "Evaluation guide for LLM gateways covering deployment model, data control, routing, observability, governance, cost controls, and Bifrost benchmark context." url: "https://www.getmaxim.ai/bifrost/resources/buyers-guide" markdown: "https://www.getmaxim.ai/bifrost/resources/buyers-guide.md" --- # LLM gateway buyer's guide > Evaluation guide for LLM gateways covering deployment model, data control, routing, observability, governance, cost controls, and Bifrost benchmark context. ## Important Links - [View MCP Gateway](https://www.getmaxim.ai/resources/mcp-gateway.md) - [Features](https://www.getmaxim.ai/#features) - [Enterprise](https://www.getmaxim.ai/enterprise) - [Pricing](https://www.getmaxim.ai/pricing) - [Docs](https://docs.getbifrost.ai) - [GitHub](https://github.com/maximhq/bifrost) - [Book a Demo](https://www.getmaxim.ai/book-a-demo) ## What an LLM gateway does An LLM gateway sits between applications and model providers. It gives teams one control layer for provider access, routing, failover, cost management, observability, guardrails, governance, and tool access. Example providers include OpenAI, Anthropic, AWS Bedrock, and Google Vertex AI. The gateway standardizes access through one API surface instead of forcing every application to manage each provider directly. Production AI teams use an LLM gateway when direct provider integrations become hard to operate across teams, environments, budgets, and compliance boundaries. - **Unified API:** Expose a consistent API surface across model providers and applications. - **Provider routing:** Route requests across providers and models by policy, availability, latency, cost, or team requirements. - **Reliability layer:** Use health checks, retries, circuit breakers, and fallback providers when a primary provider fails or rate limits. - **Cost control:** Track token usage, enforce budgets, and reduce repeated spend through caching and routing rules. - **Governance:** Use virtual keys, RBAC, SSO, audit logs, guardrails, and deployment controls to manage AI access. - **MCP and tools:** Expose tool execution through a controlled gateway layer instead of giving every application direct tool access. ## Buying criteria | Decision area | What to check | Why it matters | | --- | --- | --- | | Data path | Self-hosted, in-VPC, on-premise, or SaaS routing | Prompts, completions, and keys may need to stay inside your network. | | Latency overhead | Published benchmark data with test conditions | Gateway overhead compounds across high-volume agent and API workloads. | | Routing behavior | Fallbacks, retries, circuit breakers, health checks, and load balancing | Provider outages and quota limits should not require application changes. | | Governance | Virtual keys, RBAC, SSO, budgets, rate limits, guardrails, and audit logs | Platform teams need policy enforcement across users, teams, models, and environments. | | Observability | Request logs, token usage, latency, provider health, traces, and alerts | Teams need attribution for failures, cost spikes, and quality regressions. | | Cost model | Gateway markup, provider passthrough, caching, and budget controls | Gateway cost should not hide model cost or make token spend harder to control. | | Tooling | MCP support, webhook workflows, REST API management, and infrastructure manifests | Agent workloads need governed access to tools and internal systems. | ## Gateway evaluation landscape Compare LLM gateway platforms by operating model, data path, pricing model, latency signal, and production fit. The comparison covers hosted traffic platforms, observability-focused gateways, API gateway extensions, and model routing marketplaces as categories. | Category | Deployment | Pricing pattern | Latency signal | Primary fit | | --- | --- | --- | --- | --- | | Bifrost | Self-hosted, in-VPC, on-premise | Zero markup | 11µs gateway overhead at 5,000 RPS | Production teams that need low overhead, governance, MCP, and deployment control | | LiteLLM | Self-hosted | Zero markup | ~40ms in the compared gateway setup | Python teams that need a customizable multi-provider proxy | | Hosted AI traffic platform | SaaS | Platform plan | 10-50ms in the compared setup | Teams already standardizing on a hosted traffic layer | | Observability-focused gateway | SaaS or self-hosted | Zero markup | Not published in the compared data | Teams prioritizing request traces, usage analytics, and gateway observability | | API gateway extension | SaaS or on-premise | Enterprise contract | Not published in the compared data | Organizations extending an existing API gateway estate into AI traffic | | Model routing marketplace | SaaS only | Usage markup | 25-40ms in the compared setup | Prototype access to many models when third-party routing is acceptable | ## Production pain points Moving generative AI from prototype to production exposes infrastructure gaps that direct provider integrations do not handle well. - **Provider fragmentation:** Different APIs, credentials, and usage patterns across providers make scaling brittle. - **Limited visibility:** Without centralized logs and metrics, teams cannot trace errors or attribute token spend. - **Inconsistent reliability:** Provider outages and quota limits disrupt workflows. Individual providers rarely exceed 99.7% uptime. - **Security and governance:** API keys shared across environments create compliance vulnerabilities difficult to audit. ## Core gateway capabilities A production LLM gateway should cover routing, API unification, observability, reliability, access control, cost optimization, guardrails, and framework integration. - **Model routing and load balancing:** Route requests across LLM providers using governance rules and intelligent load distribution. - **Unified API:** Connect to multiple LLM providers with a single OpenAI-compatible API interface. - **Observability and analytics:** Monitor requests in real-time. Track token usage and enforce limits at multiple levels. - **Fallback and reliability:** Health monitoring, circuit breakers, automatic retries, and failover to alternative providers. - **Access control and security:** Virtual keys to manage permissions, rate limiting, budgets, and team-based access. - **Cost optimization:** Semantic caching, budget limits, and intelligent routing to reduce costs and latency. - **Governance and guardrails:** Policy controls on requests and responses with real-time content moderation. - **Integration and extensibility:** Compatible with OpenAI, Anthropic SDKs, LangChain, and popular frameworks. ## LLM gateway feature matrix | Feature | Bifrost | LiteLLM | Hosted AI traffic platform | Observability gateway | API gateway extension | Model routing marketplace | | --- | --- | --- | --- | --- | --- | --- | | Runtime | Go | Python | Not applicable | TypeScript | Lua/plugin stack | TypeScript | | Latency overhead | 11µs at 5,000 RPS | ~40ms | 10-50ms | Not published | Not published | 25-40ms | | Peak throughput | 5,000 RPS | Not published | Not published | Not published | Not published | High | | Open source | Yes | Yes | No | Partial | Partial | No | | Zero markup | Yes | Yes | Yes | Yes | Custom | Usage markup | | Auto failover | Yes | Yes | Yes | Yes | Yes | Yes | | Adaptive load balancing | Yes | No | No | Health-aware | Basic | No | | P2P clustering | Yes | No | No | No | No | No | | Semantic caching | Yes | No | Yes | Yes | No | No | | MCP support | Yes | No | No | No | Yes | No | | Built-in observability | Native | Via integrations | Basic | Native | Basic | No | | Real-time alerts | Yes | No | No | No | Via plugins | No | | Guardrails | Yes | No | No | No | No | No | | RBAC and governance | Yes | No | No | No | Yes | No | | SSO with SAML or OIDC | Yes | No | No | No | Yes | No | | Budget management | Yes | Yes | No | No | No | No | | Evaluation integration | Native Maxim AI integration | No | No | No | No | No | | VPC deployment | Yes | Yes | No | Yes | Yes | No | | Multi-cloud support | AWS, GCP, Azure, edge platforms, Vercel | Self-managed | Single hosted platform | Self-managed | Multi-cloud | No | ## Performance and architecture The gateway runtime affects concurrency, memory usage, and latency under load. Bifrost uses Go for native concurrency and predictable gateway overhead. - **11µs Bifrost overhead.** at 5,000 RPS in the published benchmark. [Bifrost benchmarks](https://www.getmaxim.ai/resources/benchmarks.md). - **5,000 RPS Bifrost throughput.** sustained on a single-node benchmark path. [Benchmark conditions](https://www.getmaxim.ai/resources/benchmarks.md). - **~40ms Python gateway reference.** latency overhead in the compared LiteLLM setup. - **50x faster P95 speedup.** than Python-based gateways at P95 in the compared setup. - **10-50ms Hosted traffic category.** latency overhead in the compared hosted gateway setup. - **99.999% Failover availability.** uptime posture associated with automatic multi-provider failover and configured fallback providers. [Enterprise scalability](https://www.getmaxim.ai/resources/enterprise-scalability.md). ## Bifrost integrations - **Maxim AI Platform:** Native evaluation platform; Continuous quality monitoring; Real-time observability; Agent simulation testing - **Agent Frameworks:** LangChain compatibility; LlamaIndex integration; CrewAI support; OpenAI SDK drop-in - **Tool & Protocol:** MCP support; Webhook workflows; REST API management; Terraform & K8s manifests - **Authentication:** Google & GitHub SSO; SAML/OIDC support; API key management; Virtual key generation - **Infrastructure:** Docker & Compose; Kubernetes + Helm; Multi-cloud deployment; CI/CD integration - **Monitoring:** Prometheus metrics; OpenTelemetry tracing; Custom logging; Alert webhooks ## When Bifrost fits - **Regulated deployment:** Bifrost supports self-hosted, in-VPC, on-premise, and enterprise deployment patterns for teams with data boundary requirements. [Enterprise deployment](https://www.getmaxim.ai/resources/enterprise-deployment.md). - **Low-overhead gateway path:** Bifrost benchmark data reports 11µs gateway overhead at 5,000 RPS. [Benchmarks](https://www.getmaxim.ai/resources/benchmarks.md). - **Governed model access:** Virtual keys, budgets, rate limits, RBAC, SSO, audit logs, and guardrails provide a central control layer. [Governance](https://www.getmaxim.ai/resources/governance.md), [Guardrails](https://www.getmaxim.ai/resources/guardrails.md). - **Agent and tool workloads:** Native MCP support lets teams route tool access through the same gateway layer used for models, logs, and governance. [MCP gateway](https://www.getmaxim.ai/resources/mcp-gateway.md). ## Features ### OSS Features - **01 - Model Catalog:** Access 8+ providers and 1000+ AI models through a unified interface. Also supports custom deployed models. - **02 - Budgeting:** Set spending limits and track costs across teams, projects, and models. - **03 - Provider Fallback:** Automatic failover between providers ensures 99.99% uptime for your applications. - **04 - MCP Gateway:** Centralize all MCP tool connections, governance, security, and auth. Your AI can safely use MCP tools with centralized policy enforcement. [MCP Gateway resource](https://www.getmaxim.ai/resources/mcp-gateway.md). - **05 - Virtual Key Management:** Create different virtual keys for different use cases with independent budgets and access control. - **06 - Unified Interface:** One consistent API for all providers. Switch models without changing code. - **07 - Drop-in Replacement:** Replace your existing SDK with just one line change. Compatible with OpenAI, Anthropic, LiteLLM, Google GenAI, LangChain, and more. [Drop-in replacement docs](https://docs.getbifrost.ai/features/drop-in-replacement). - **08 - Built-in Observability:** Out-of-the-box OpenTelemetry support. Built-in dashboard for quick visibility without complex setup. - **09 - Community Support:** Active Discord community with responsive support and regular updates. ### Enterprise Features - **01 - Governance:** SAML support for SSO and role-based access control with policy enforcement for team collaboration. [Governance resource](https://www.getmaxim.ai/resources/governance.md). - **02 - Adaptive Load Balancing:** Automatically optimizes traffic distribution across provider keys and models based on real-time performance metrics. - **03 - Cluster Mode:** High availability deployment with automatic failover and load balancing. Peer-to-peer clustering where every instance is equal. - **04 - Alerts:** Real-time notifications for budget limits, failures, and performance issues on Email, Slack, PagerDuty, Teams, Webhook, and more. - **05 - Log Exports:** Export and analyze request logs, traces, and telemetry data from Bifrost with enterprise-grade data export for compliance, monitoring, and analytics. - **06 - Audit Logs:** Comprehensive logging and audit trails for compliance and debugging. - **07 - Vault Support:** Secure API key management with HashiCorp Vault, AWS Secrets Manager, Google Secret Manager, and Azure Key Vault integration. - **08 - VPC Deployment:** Deploy Bifrost within your private cloud infrastructure with VPC isolation, custom networking, and enhanced security controls. [Enterprise deployment resource](https://www.getmaxim.ai/resources/enterprise-deployment.md). - **09 - Guardrails:** Automatically detect and block unsafe model outputs with real-time policy enforcement and content moderation across all agents. [Guardrails resource](https://www.getmaxim.ai/resources/guardrails.md). ## FAQ ### How do I choose the right AI gateway for enterprise AI production? Choosing an enterprise AI gateway solves the complexity of managing multiple providers while ensuring AI applications remain fast and secure even at scale. The main objective is to provide a unified layer that handles high-volume traffic with governance and reliability. To meet production standards, a gateway should offer: • Minimal Latency: Sub-microsecond overhead, like Bifrost's 11µs latency, prevents infrastructure bottlenecks. • Operational Reliability: Multi-model routing and automated failover ensure constant uptime. • Enterprise Governance: RBAC and token budgeting are essential for cost and access control. • Security at Scale: VPC or on-premise deployment ensures data privacy while handling thousands of concurrent requests. ### How do I choose between a self-hosted and SaaS LLM gateway? Self-hosted gateways like Bifrost and LiteLLM give you full data control and in-VPC deployment, which regulated industries often require. SaaS options like OpenRouter offer faster setup but route data through third-party infrastructure. Consider your compliance requirements, data sensitivity, and operational capacity. ### What makes Bifrost different from other LLM gateways? Bifrost is built in Go for production-grade performance with 11 µs latency overhead at 5,000 RPS. It includes native MCP support, adaptive load balancing, built-in observability, and integrates with the Maxim AI evaluation platform. It's fully open source under Apache 2.0. ### Do LLM gateways add significant latency to API requests? It depends on the gateway architecture. Python-based gateways typically add 10-50ms of overhead. Go-based gateways like Bifrost add around 11µs, which is negligible compared to LLM response times of hundreds of milliseconds to seconds. ### How does Bifrost handle LLM provider downtime? Bifrost ensures 99.999% uptime through automatic multi-provider failover. If a primary provider (like OpenAI) experiences an outage or rate limit, Bifrost instantly routes traffic to a pre-configured fallback (like Anthropic or AWS Bedrock) without requiring any code changes, ensuring your application stays live. ## Related Resources - [Buyer guide page](https://www.getmaxim.ai/resources/buyers-guide.md) - [Benchmark: Bifrost benchmarks](https://www.getmaxim.ai/resources/benchmarks.md) - [Governance: Bifrost governance](https://www.getmaxim.ai/resources/governance.md) - [Guardrails: Bifrost guardrails](https://www.getmaxim.ai/resources/guardrails.md) - [MCP: Bifrost MCP gateway](https://www.getmaxim.ai/resources/mcp-gateway.md) - [Enterprise: Deployment options](https://www.getmaxim.ai/resources/enterprise-deployment.md) - [Docs: Bifrost docs](https://docs.getbifrost.ai) - [GitHub: maximhq/bifrost](https://github.com/maximhq/bifrost) - [Pricing: Bifrost pricing](https://www.getmaxim.ai/pricing) - [Enterprise: Bifrost enterprise](https://www.getmaxim.ai/enterprise) - [Book a Demo: Bifrost demo](https://www.getmaxim.ai/book-a-demo) - [Resources: Bifrost resources](https://www.getmaxim.ai/resources.md) --- *This is a markdown version of [https://www.getmaxim.ai/bifrost/resources/buyers-guide](https://www.getmaxim.ai/bifrost/resources/buyers-guide) for AI/LLM consumption.* --- title: "Bifrost OSS for Startups | Free Production-Ready LLM Gateway" description: "Free, open-source LLM gateway with automatic failover, load balancing, semantic caching, and 20+ provider support. Production-ready features for startups and SMEs at zero cost." url: "https://www.getmaxim.ai/bifrost/resources/oss-for-startups" markdown: "https://www.getmaxim.ai/bifrost/resources/oss-for-startups.md" --- # Bifrost OSS for Startups | Free Production-Ready LLM Gateway > Free, open-source LLM gateway with automatic failover, load balancing, semantic caching, and 20+ provider support. Production-ready features for startups and SMEs at zero cost. ## Important Links - [View MCP Gateway](https://www.getmaxim.ai/resources/mcp-gateway.md) - [Features](https://www.getmaxim.ai/#features) - [Enterprise](https://www.getmaxim.ai/enterprise) - [Pricing](https://www.getmaxim.ai/pricing) - [Docs](https://docs.getbifrost.ai) - [GitHub](https://github.com/maximhq/bifrost) - [Book a Demo](https://www.getmaxim.ai/book-a-demo) ## Performance at a Glance - **11µs Gateway Overhead.** Per request at 5K RPS - **5K Requests/Second.** Sustained throughput - **1000+ AI Models.** Unified API access - **0 Configuration.** Deploy in seconds ## OSS Features - **99.999% uptime - Automatic failover:** Seamless provider switching when primary systems fail. Zero downtime when OpenAI, Anthropic, or other providers experience outages or hit rate limits. - **Weighted routing - Intelligent load balancing:** Weighted API key distribution with model-specific filtering. Route requests across multiple keys and providers for higher throughput and reliability. - **Cost savings - Semantic caching:** Intelligent response caching based on semantic similarity. Reduce costs and latency by serving similar queries from cache instead of hitting the model. - **Access control - Virtual keys & budgets:** Create separate credentials for each team or project with independent rate limits and budgets. Hierarchical cost management across virtual keys, teams, users, and customers. - **OTEL native - Real-time observability:** Built-in request tracking, performance analysis, Prometheus metrics, and OpenTelemetry integration. Monitor every AI request in real-time. - **Zero refactor - Drop-in replacement:** Replace existing AI SDK connections by changing just the base URL. Works with OpenAI SDK, Anthropic SDK, LangChain, and all major frameworks. ## Why Startups Choose Bifrost OSS - **Zero cost to start:** Free, open-source, and production-ready. No enterprise license required for core functionality. Deploy on your infrastructure and scale without vendor lock-in. - **Scale without breaking:** Built in Go for high-performance workloads. Handles 5,000 requests per second with only 11µs overhead. Scales horizontally as your startup grows. - **Avoid provider lock-in:** Switch between 1000+ models without code changes. Compare model quality, cost, and latency across OpenAI, Anthropic, Google, and others from day one. - **Cut costs immediately:** Semantic caching reduces redundant model calls. Load balancing optimizes API key usage. Virtual key budgets prevent runaway spending. - **Ship faster:** No need to build custom routing, failover, or monitoring infrastructure. Focus on product features while Bifrost handles LLM reliability and observability. - **Community-driven:** Open source with active GitHub community. Contribute features, report issues, and learn from other startups building with Bifrost. ## Core Capabilities - **Multi-provider routing:** Direct requests to specific models and providers with weighted strategies. Support for OpenAI, Anthropic, AWS Bedrock, Google Vertex, Azure, Groq, Mistral, Cohere, and 12+ additional providers. - **MCP tool filtering:** Allow-list controls determining which Model Context Protocol tools are available per virtual key. Fine-grained permissions for AI agent capabilities. - **Custom Plugins:** Custom plugins via Go or WebAssembly for bespoke business logic. Includes Mocker plugin for simulating provider responses during development. - **Built-in web UI:** Visual configuration dashboard with real-time monitoring. Configure virtual keys, view request logs, track costs, and analyze performance without code. - **Prometheus & OTEL:** Native metrics collection via Prometheus scraping or Push Gateway. OpenTelemetry integration with Grafana, New Relic, Honeycomb, and any observability stack. - **Go SDK integration:** Direct application integration for maximum performance and control. Deploy as HTTP gateway or embed in your Go application. ## Deployment Options - **Docker:** Pull and run the official Docker image - **Kubernetes:** Deploy via Helm chart with auto-scaling - **Binary:** Download and run the standalone binary - **Go SDK:** Embed directly in your application ### Docker ``` docker pull bifrost-gateway docker run -p 8080:8080 bifrost ``` ### Kubernetes ``` helm repo add bifrost https://charts.getbifrost.ai helm install bifrost bifrost/bifrost ``` ### Binary ``` wget https://github.com/maximhq/bifrost/releases/latest ./bifrost --port 8080 ``` ### Go SDK ``` import "github.com/maximhq/bifrost" gateway := bifrost.NewGateway() gateway.Start(":8080") ``` ## OSS vs Enterprise OSS provides production-ready core features. Enterprise adds compliance, advanced governance, and priority support. | Category | Open Source (Free) | Enterprise | | --- | --- | --- | | Core Features | Automatic failover; Load balancing; Semantic caching; Virtual keys & budgets; 1000+ models support; MCP tool filtering | Everything in OSS; Adaptive load balancing; Advanced routing strategies; Adaptive multi-region failover | | Observability | Real-time request logs; Prometheus metrics; OpenTelemetry integration; Built-in web UI; Cost tracking | Everything in OSS; Datadog/BigQuery connector; Log exports; Audit logging; Compliance reports | | Security & Governance | Virtual key access control; Rate limiting; Budget enforcement; MCP tool permissions | Everything in OSS; Guardrails (PII, content moderation); SSO (like Okta, Entra); RBAC; Vault integration; In-VPC deployment | | Extensibility | Go plugins; WASM plugins; Mocker plugin; Custom plugins | Everything in OSS; Priority support | ## Use Cases - **Early-stage startups:** Launch AI features without infrastructure overhead. Free OSS provides production-ready failover, caching, and multi-provider access from day one. - **Rapid prototyping:** Test models across OpenAI, Anthropic, Google, and others without large code changes. Compare quality and cost before committing to one provider. - **Cost-conscious teams:** Semantic caching and load balancing reduce redundant LLM spend. Virtual key budgets prevent surprise bills. Free OSS means no gateway license fees. - **SME AI adoption:** Small and medium enterprises can deploy enterprise-grade LLM infrastructure without enterprise budgets. Self-host on existing infrastructure. - **Developer tools:** Integrate AI features into developer tools, IDEs, or CLI applications. Go SDK provides direct embedding. HTTP gateway works with any language. - **Open source projects:** Add AI capabilities to open source software without vendor dependencies. Community-driven development with transparent roadmap and governance. ## Start Free, Scale as You Grow Begin with OSS and upgrade to Enterprise only when you need compliance features. 1. **Phase 1 - Start Free.** Deploy OSS with core features: Failover, Load balancing, Caching, Virtual keys, Observability. 2. **Phase 2 - Scale Up.** Add features as you grow: Horizontal scaling, Custom plugins, Advanced routing, Community support. 3. **Phase 3 - Go Enterprise.** Upgrade when you need compliance: Guardrails, SSO, RBAC, Audit logs, Priority support. ## Features ### OSS Features - **01 - Model Catalog:** Access 8+ providers and 1000+ AI models through a unified interface. Also supports custom deployed models. - **02 - Budgeting:** Set spending limits and track costs across teams, projects, and models. - **03 - Provider Fallback:** Automatic failover between providers ensures 99.99% uptime for your applications. - **04 - MCP Gateway:** Centralize all MCP tool connections, governance, security, and auth. Your AI can safely use MCP tools with centralized policy enforcement. [MCP Gateway resource](https://www.getmaxim.ai/resources/mcp-gateway.md). - **05 - Virtual Key Management:** Create different virtual keys for different use cases with independent budgets and access control. - **06 - Unified Interface:** One consistent API for all providers. Switch models without changing code. - **07 - Drop-in Replacement:** Replace your existing SDK with just one line change. Compatible with OpenAI, Anthropic, LiteLLM, Google GenAI, LangChain, and more. [Drop-in replacement docs](https://docs.getbifrost.ai/features/drop-in-replacement). - **08 - Built-in Observability:** Out-of-the-box OpenTelemetry support. Built-in dashboard for quick visibility without complex setup. - **09 - Community Support:** Active Discord community with responsive support and regular updates. ### Enterprise Features - **01 - Governance:** SAML support for SSO and role-based access control with policy enforcement for team collaboration. [Governance resource](https://www.getmaxim.ai/resources/governance.md). - **02 - Adaptive Load Balancing:** Automatically optimizes traffic distribution across provider keys and models based on real-time performance metrics. - **03 - Cluster Mode:** High availability deployment with automatic failover and load balancing. Peer-to-peer clustering where every instance is equal. - **04 - Alerts:** Real-time notifications for budget limits, failures, and performance issues on Email, Slack, PagerDuty, Teams, Webhook, and more. - **05 - Log Exports:** Export and analyze request logs, traces, and telemetry data from Bifrost with enterprise-grade data export for compliance, monitoring, and analytics. - **06 - Audit Logs:** Comprehensive logging and audit trails for compliance and debugging. - **07 - Vault Support:** Secure API key management with HashiCorp Vault, AWS Secrets Manager, Google Secret Manager, and Azure Key Vault integration. - **08 - VPC Deployment:** Deploy Bifrost within your private cloud infrastructure with VPC isolation, custom networking, and enhanced security controls. [Enterprise deployment resource](https://www.getmaxim.ai/resources/enterprise-deployment.md). - **09 - Guardrails:** Automatically detect and block unsafe model outputs with real-time policy enforcement and content moderation across all agents. [Guardrails resource](https://www.getmaxim.ai/resources/guardrails.md). ## FAQ ### Is Bifrost OSS truly free for production use? Yes. Bifrost OSS is Apache 2.0 licensed and includes production-ready features like automatic failover, load balancing, semantic caching, and multi-provider routing at zero cost. No enterprise license required for core functionality. ### What are the limitations of the OSS version? OSS includes all core LLM gateway features. Enterprise adds advanced governance (guardrails, RBAC, SSO), compliance features (audit logging, in-VPC deployment), and priority support. Most startups and SMEs find OSS sufficient for production workloads. ### Can I self-host Bifrost OSS on my infrastructure? Yes. Bifrost OSS deploys via Docker, Kubernetes (Helm), standalone binary, or embedded Go SDK. Self-host on AWS, GCP, Azure, or any infrastructure. No data leaves your environment unless routing to external LLM providers. ### How does Bifrost OSS compare to hosted LLM gateways? Bifrost OSS is self-hosted (zero gateway fees), open source (full transparency), and community-driven. Hosted alternatives like Helicone or Portkey charge per request or require enterprise contracts. OSS gives you control and flexibility. ### Does OSS include the web UI and monitoring dashboard? Yes. The built-in web UI provides visual configuration, real-time request logs, cost tracking, and performance analysis. Prometheus metrics and OpenTelemetry integration are also included in OSS. ### When should I upgrade from OSS to Enterprise? Upgrade to Enterprise when you need compliance features (SOC 2 Type II, HIPAA, GDPR), advanced security (guardrails, SSO, RBAC), audit logging for regulated industries, or in-VPC deployment with priority support. OSS is production-ready for most use cases. [Get 14 days free enterprise trial](https://www.getmaxim.ai/enterprise). ## Related Resources - [Source: OSS For Startups](https://www.getmaxim.ai/resources/oss-for-startups.md) - [Docs: Bifrost docs](https://docs.getbifrost.ai) - [GitHub: maximhq/bifrost](https://github.com/maximhq/bifrost) - [Pricing: Bifrost pricing](https://www.getmaxim.ai/pricing) - [Enterprise: Bifrost enterprise](https://www.getmaxim.ai/enterprise) - [Book a Demo: Bifrost demo](https://www.getmaxim.ai/book-a-demo) - [Resources: Bifrost resources](https://www.getmaxim.ai/resources.md) --- *This is a markdown version of [https://www.getmaxim.ai/bifrost/resources/oss-for-startups](https://www.getmaxim.ai/bifrost/resources/oss-for-startups) for AI/LLM consumption.* --- title: "Claude Code + Bifrost | Enterprise LLM Gateway for Claude Code" description: "Add multi-provider routing, cost control, guardrails, and governance to Claude Code at scale with Bifrost - the fastest enterprise LLM gateway." url: "https://www.getmaxim.ai/bifrost/resources/claude-code" markdown: "https://www.getmaxim.ai/bifrost/resources/claude-code.md" --- # Claude Code + Bifrost | Enterprise LLM Gateway for Claude Code > Add multi-provider routing, cost control, guardrails, and governance to Claude Code at scale with Bifrost - the fastest enterprise LLM gateway. ## Important Links - [View MCP Gateway](https://www.getmaxim.ai/resources/mcp-gateway.md) - [Features](https://www.getmaxim.ai/#features) - [Enterprise](https://www.getmaxim.ai/enterprise) - [Pricing](https://www.getmaxim.ai/pricing) - [Docs](https://docs.getbifrost.ai) - [GitHub](https://github.com/maximhq/bifrost) - [Book a Demo](https://www.getmaxim.ai/book-a-demo) ## Performance at a Glance - **11µs Mean Latency.** Gateway overhead per request - **5K RPS Throughput.** Requests per second sustained - **50x Faster.** Than Python-based gateways - **20+ Providers.** Model APIs supported ## Pain Points - **No cost visibility:** Teams using Claude Code directly do not get centralized request logs, token attribution, or team-level spend controls. - **Single provider dependency:** A direct Claude Code setup depends on one provider path. Bifrost adds multi-provider routing and fallback policies behind the same developer workflow. - **No guardrails at scale:** Developer prompts and generated code can include sensitive context. Central guardrails help enforce PII redaction and content policies before requests reach models. - **Limited observability:** Without a gateway, platform teams cannot consistently trace Claude Code requests by user, team, route, model, token count, and latency. ## Core Features - **LLM cost control + budgets - Cost management and optimization:** Track LLM spend per request with breakdowns by provider, model, team, and developer. Virtual keys enforce team-level budgets. Semantic caching reduces costs on repeat queries. - **99.999% uptime target - Multi-provider routing with failover:** Automatic failover across Anthropic, AWS Bedrock, and Google Vertex AI when rate limits or outages hit. Adaptive load balancing keeps throughput stable even under heavy load. - **AWS Bedrock + Azure AI - Guardrails and governance:** Enforce content policies, PII redaction, and safety checks before requests reach the model. Role-based access controls and rate limits per team provide fine-grained LLM governance across the organization. - **11µs @ 5K RPS - Reduce latency, high throughput:** Built in Go for production workloads. Bifrost adds only 11µs mean overhead at 5,000 requests per second, making it 50x faster than Python-based gateways. Coding workflows stay fast at scale. - **OTEL native - Centralized LLM API observability:** Every Claude Code request is logged with full metadata including user, team, provider, route, token count, and latency. Filter and export through the dashboard or push to any observability stack via OpenTelemetry. - **Vault + SSO ready - Centralized credential management:** API keys for all providers live in one place. Integrate with HashiCorp Vault for secure key storage or manage them directly in Bifrost. SSO support for Google, GitHub, and enterprise identity providers. ## Setup Steps 1. **01 - Deploy Bifrost.** Bifrost runs as a standalone Go service. Teams deploy it in-VPC or via managed hosting. No agent installation on developer machines. ### Deploy Bifrost ``` # pull and start bifrost docker pull bifrost-gateway docker run -p 8080:8080 bifrost ``` 2. **02 - Point Claude Code at it.** Developers set one environment variable. Claude Code sends all requests through Bifrost without any code changes or plugin installation. ### Point Claude Code at it ``` export ANTHROPIC_BASE_URL="http://localhost:8080" # that's it - claude code just works ``` 3. **03 - Configure from the dashboard.** Set team budgets, apply guardrails, configure provider fallbacks, and view real-time analytics, all from Bifrost's web interface. No code required. ### Configure from the dashboard ``` # dashboard available at localhost:8080/logs # virtual keys, budgets, guardrails ``` ## Enterprise Features - **Automatic failovers:** Requests reroute seamlessly when a provider fails or hits rate limits. - **Adaptive load balancing:** Traffic distributes intelligently based on real-time health signals. - **Semantic caching:** Repeat or near-identical queries resolve instantly, cutting costs and reducing latency. - **Virtual keys & budgets:** Create separate virtual API keys for each team with independent limits. - **Guardrails:** Enforce content policies, PII redaction, and safety checks. - **Audit logs:** Complete, tamper-evident record of every request for compliance. - **SSO integration:** Authenticate via Google, GitHub, Okta, or any SAML/OIDC provider. - **Vault support:** API keys stored in HashiCorp Vault, never touch developer machines. - **Cluster mode:** Horizontal scaling with zero downtime across multiple nodes. - **Code Mode (MCP):** AI generates Python to orchestrate multiple MCP tools in one execution. - **Alerts:** Threshold-based alerts for cost overruns, rate limits, and errors. - **MCP Gateway:** Inject filesystem tools, database connectors, and custom integrations. ## Comparison Data | Feature | Standalone | With Bifrost | | --- | --- | --- | | Multi-model support | No | 20+ providers | | MCP tool gateway | No | Full MCP injection | | Cost tracking | No | Real-time per-request | | Provider failover | No | Automatic across providers | | Semantic caching | No | Reduce costs and latency | | Team budgets | No | Virtual keys + limits | | Request observability | No | Full log trail + OTEL export | | Gateway latency | - | 11µs at 5,000 RPS | ## Native MCP Tool Support for Agentic Workflows Bifrost connects Claude Code to filesystem tools, databases, web search, and custom integrations via Model Context Protocol without modifying the Claude Code client or adding configuration steps on the developer side. - **Multi-provider development:** Teams test code across Claude Sonnet, GPT-4, and Gemini from the same Claude Code workspace. Model performance and cost comparisons happen in real time inside Bifrost's dashboard. - **Agentic coding pipelines:** Claude Code combines with MCP-connected tools for database queries, API testing, deployment scripts, and custom integrations all routed and monitored through a single gateway. - **Semantic caching at scale:** Repeat or near-identical queries across developers resolve instantly from cache. Teams running large codebases see cost savings on common operations like code explanations and documentation generation. ## Use Cases - **Enterprise cost management:** Platform teams set department-level budgets for Claude Code usage. Real-time cost tracking surfaces which teams, projects, or developers are driving LLM spend. Automated alerts fire when budgets approach limits. - **Multi-model testing and comparison:** Engineering teams route the same Claude Code workflow through Claude Sonnet, GPT-4, and Gemini to compare code quality, latency, and cost. Bifrost logs performance metrics for each provider. - **Regulatory compliance and governance:** Organizations in healthcare, finance, or government use Bifrost's guardrails to enforce PII redaction and content policies. Audit logs provide tamper-evident records for SOC 2, HIPAA, and GDPR compliance. - **High-availability production deployments:** Teams running Claude Code at scale rely on Bifrost's automatic failover and load balancing to maintain 99.999% uptime. When Anthropic hits rate limits, requests automatically route to Bedrock or Vertex AI. - **Startups scaling AI development:** Early-stage teams use Bifrost's LLM gateway to experiment with multiple providers without vendor lock-in. Semantic caching cuts costs and latency during rapid prototyping. - **Agentic coding with MCP tools:** Developers connect Claude Code to databases, APIs, and deployment pipelines via MCP. Bifrost handles tool injection transparently, enabling automated database migrations and cloud deployment scripts. ## Governance Features - **Role-based access control:** Define teams, roles, and environment-specific access at the organization level. Developers, platform engineers, and finance teams each get appropriate visibility and control. - **Comprehensive audit trails:** Every request, policy enforcement action, and configuration change is logged with full context. Export audit trails to your SIEM or compliance platform. - **Content filtering and PII redaction:** Bifrost's guardrails detect and redact sensitive information like SSNs, credit card numbers, and API keys before requests reach the model. - **In-VPC deployment:** Deploy Bifrost entirely within your VPC for maximum security and data control. All LLM requests stay within your network perimeter. ## Compliance Badges - **Audited quarterly - SOC 2 Type II:** Compliance signal for audit-ready enterprise deployments. - **EU data residency - GDPR:** Data protection signal for privacy and residency reviews. - **BAA available - HIPAA:** Healthcare compliance signal for protected health information workflows. - **Certified - ISO 27001:** Security management signal for enterprise risk reviews. ## Features ### OSS Features - **01 - Model Catalog:** Access 8+ providers and 1000+ AI models through a unified interface. Also supports custom deployed models. - **02 - Budgeting:** Set spending limits and track costs across teams, projects, and models. - **03 - Provider Fallback:** Automatic failover between providers ensures 99.99% uptime for your applications. - **04 - MCP Gateway:** Centralize all MCP tool connections, governance, security, and auth. Your AI can safely use MCP tools with centralized policy enforcement. [MCP Gateway resource](https://www.getmaxim.ai/resources/mcp-gateway.md). - **05 - Virtual Key Management:** Create different virtual keys for different use cases with independent budgets and access control. - **06 - Unified Interface:** One consistent API for all providers. Switch models without changing code. - **07 - Drop-in Replacement:** Replace your existing SDK with just one line change. Compatible with OpenAI, Anthropic, LiteLLM, Google GenAI, LangChain, and more. [Drop-in replacement docs](https://docs.getbifrost.ai/features/drop-in-replacement). - **08 - Built-in Observability:** Out-of-the-box OpenTelemetry support. Built-in dashboard for quick visibility without complex setup. - **09 - Community Support:** Active Discord community with responsive support and regular updates. ### Enterprise Features - **01 - Governance:** SAML support for SSO and role-based access control with policy enforcement for team collaboration. [Governance resource](https://www.getmaxim.ai/resources/governance.md). - **02 - Adaptive Load Balancing:** Automatically optimizes traffic distribution across provider keys and models based on real-time performance metrics. - **03 - Cluster Mode:** High availability deployment with automatic failover and load balancing. Peer-to-peer clustering where every instance is equal. - **04 - Alerts:** Real-time notifications for budget limits, failures, and performance issues on Email, Slack, PagerDuty, Teams, Webhook, and more. - **05 - Log Exports:** Export and analyze request logs, traces, and telemetry data from Bifrost with enterprise-grade data export for compliance, monitoring, and analytics. - **06 - Audit Logs:** Comprehensive logging and audit trails for compliance and debugging. - **07 - Vault Support:** Secure API key management with HashiCorp Vault, AWS Secrets Manager, Google Secret Manager, and Azure Key Vault integration. - **08 - VPC Deployment:** Deploy Bifrost within your private cloud infrastructure with VPC isolation, custom networking, and enhanced security controls. [Enterprise deployment resource](https://www.getmaxim.ai/resources/enterprise-deployment.md). - **09 - Guardrails:** Automatically detect and block unsafe model outputs with real-time policy enforcement and content moderation across all agents. [Guardrails resource](https://www.getmaxim.ai/resources/guardrails.md). ## FAQ ### How does Bifrost integrate with Claude Code? Bifrost integrates through a single environment variable (ANTHROPIC_BASE_URL). Developers point Claude Code at the Bifrost gateway, and all requests are routed, logged, and governed transparently. No plugins, no code changes, no workflow disruption. ### Can I use Claude Code with other LLM providers through Bifrost? Yes. Bifrost supports 20+ providers including Anthropic, AWS Bedrock, Google Vertex AI, OpenAI, and more. You can configure automatic failover so if Anthropic hits rate limits, requests transparently route to Bedrock or Vertex AI. ### How do enterprises manage and track costs for Claude Code usage? Managing AI spend across multiple teams requires the granular governance tools provided by an LLM gateway. Bifrost provides virtual keys with per-team budgets, real-time cost tracking per request, and automated alerts when budgets approach limits. Enterprise teams get a centralized dashboard showing spend by team, project, and developer. ### Does Bifrost add latency to Claude Code requests? Bifrost adds approximately 11µs of gateway overhead per request at 5,000 RPS. This is effectively invisible in the context of LLM response times that typically range from hundreds of milliseconds to several seconds. ### What security features does Bifrost provide for Claude Code? Bifrost includes PII redaction, content policy enforcement, role-based access control, SSO integration, complete audit trails, and in-VPC deployment. All requests are logged with metadata for SOC 2, HIPAA, and GDPR compliance. ## Related Resources - [Source: Claude Code](https://www.getmaxim.ai/resources/claude-code.md) - [Docs: Bifrost docs](https://docs.getbifrost.ai) - [GitHub: maximhq/bifrost](https://github.com/maximhq/bifrost) - [Pricing: Bifrost pricing](https://www.getmaxim.ai/pricing) - [Enterprise: Bifrost enterprise](https://www.getmaxim.ai/enterprise) - [Book a Demo: Bifrost demo](https://www.getmaxim.ai/book-a-demo) - [Resources: Bifrost resources](https://www.getmaxim.ai/resources.md) --- *This is a markdown version of [https://www.getmaxim.ai/bifrost/resources/claude-code](https://www.getmaxim.ai/bifrost/resources/claude-code) for AI/LLM consumption.* --- title: "CLI Coding Agents + Bifrost | Claude Code, Codex, Gemini, OpenCode" description: "Run Claude Code, Codex CLI, Gemini CLI, and OpenCode through Bifrost for multi-provider routing, enterprise controls, cost tracking, and centralized observability." url: "https://www.getmaxim.ai/bifrost/resources/cli-agents" markdown: "https://www.getmaxim.ai/bifrost/resources/cli-agents.md" --- # CLI Coding Agents + Bifrost | Claude Code, Codex, Gemini, OpenCode > Run Claude Code, Codex CLI, Gemini CLI, and OpenCode through Bifrost for multi-provider routing, enterprise controls, cost tracking, and centralized observability. ## Important Links - [View MCP Gateway](https://www.getmaxim.ai/resources/mcp-gateway.md) - [Features](https://www.getmaxim.ai/#features) - [Enterprise](https://www.getmaxim.ai/enterprise) - [Pricing](https://www.getmaxim.ai/pricing) - [Docs](https://docs.getbifrost.ai) - [GitHub](https://github.com/maximhq/bifrost) - [Book a Demo](https://www.getmaxim.ai/book-a-demo) ## Supported CLI Agents Each CLI agent brings unique capabilities. Bifrost provides the enterprise controls, multi-provider access, and observability that all of them need at scale. - **Provider: Anthropic - Claude Code:** Official CLI Agent for Anthropic Models. - **Provider: OpenAI - Codex CLI:** OpenAI's powerful code generation CLI. - **Provider: Google - Gemini CLI:** Google Gemini powered coding agent. - **Provider: Multi-provider - OpenCode:** Open source coding agent framework. ## Core Capabilities - **1000+ models - Multi-provider model access:** Every CLI agent gets access to 1000+ models through Bifrost. Switch between Anthropic, OpenAI, Google, AWS Bedrock, and more without reconfiguring your agent. - **SOC 2 Type II ready - Enterprise security & governance:** All agent requests route through Bifrost, applying guardrails, PII redaction, content policies, and role-based access controls transparently. - **Per-request costs - Unified cost tracking:** Track LLM spend across all CLI agents from a single dashboard. Virtual keys enforce per-team budgets. Export via the logs API or OTEL for custom cost analysis. - **OTEL native - Centralized observability:** Every request from Claude Code, Codex CLI, Gemini CLI, and OpenCode is logged with full metadata. Export to any observability stack via OpenTelemetry. - **99.999% uptime - Automatic failover & load balancing:** When a provider hits rate limits or experiences an outage, Bifrost automatically routes agent requests to healthy providers. Adaptive load balancing keeps throughput stable. - **Cost savings - Semantic caching:** Repeat or near-identical queries across agents resolve instantly from cache, cutting costs and reducing latency for common coding operations. ## Setup Steps 1. **01 - Deploy Bifrost Gateway.** Run Bifrost as a standalone service in your VPC or use managed hosting. Configure your LLM providers and optional guardrails through the dashboard. ### Deploy Bifrost Gateway ``` # pull and start bifrost docker pull bifrost-gateway docker run -p 8080:8080 bifrost ``` 2. **02 - Point your CLI agent at Bifrost.** Set the API base URL environment variable for your chosen agent. Each agent has slightly different configuration requirements. ### Point your CLI agent at Bifrost ``` # Claude Code export ANTHROPIC_BASE_URL="http://localhost:8080" # Codex CLI (must end in /v1) export OPENAI_API_BASE="http://localhost:8080/openai/v1" # Gemini CLI export GEMINI_API_BASE="http://localhost:8080" # OpenCode export API_BASE_URL="http://localhost:8080" ``` 3. **03 - Start coding with enterprise controls.** Launch your agent and start coding. All requests automatically route through Bifrost with full observability, cost tracking, and guardrails applied. ### Start coding with enterprise controls ``` # Use Bifrost CLI for guided setup npx -y @maximhq/bifrost-cli # Or launch agents directly claude code codex generate gemini chat opencode ``` ## Comparison Data | Feature | Standalone | With Bifrost | | --- | --- | --- | | Multi-provider access | No | 20+ providers | | Automatic failover | No | Across all providers | | Cost tracking | No | Real-time per request | | Team budgets | No | Virtual keys + limits | | Security guardrails | No | PII redaction + policies | | Request observability | No | Full logs + OTEL export | | Semantic caching | No | Reduce costs + latency | | MCP tool integration | Claude Code only | All agents via gateway | ## Use Cases - **Multi-agent development workflows:** Teams use different CLI agents for different tasks: Claude Code for architecture, Codex for quick scripts, Gemini for code review, OpenCode for deployments. Bifrost provides unified governance. - **Model performance comparison:** Run the same coding task through Claude Code, Codex CLI, and Gemini CLI to compare quality, speed, and cost. Bifrost logs performance metrics for each provider. - **Enterprise security compliance:** Organizations route all CLI agent traffic through Bifrost to enforce PII redaction, content policies, and audit trails for SOC 2 Type II, HIPAA, and GDPR compliance. - **Department-level cost management:** Platform teams set budgets for each engineering department. Real-time dashboards show which teams, projects, and agents are driving LLM spend. - **High-availability production coding:** When Anthropic hits rate limits, Claude Code requests automatically route to Bedrock. When OpenAI hits rate limits or errors, Codex CLI switches to Azure OpenAI. - **Centralized AI governance:** Security teams enforce organization-wide policies across all CLI agents from a single control plane. No need to configure guardrails per agent. ## Enterprise Features - **Virtual keys per team:** Create separate credentials for each team with independent rate limits and budgets. - **Content filtering:** Detect and redact PII, sensitive data, and API keys before requests reach models. - **Audit trails:** Complete tamper-evident logs of every request for compliance and security. - **SSO integration:** Authenticate via Google, GitHub, Okta, or any SAML/OIDC provider. - **Adaptive routing:** Traffic distributes intelligently based on real-time health and performance. - **In-VPC deployment:** Deploy entirely within your network perimeter for maximum security. ## Features ### OSS Features - **01 - Model Catalog:** Access 8+ providers and 1000+ AI models through a unified interface. Also supports custom deployed models. - **02 - Budgeting:** Set spending limits and track costs across teams, projects, and models. - **03 - Provider Fallback:** Automatic failover between providers ensures 99.99% uptime for your applications. - **04 - MCP Gateway:** Centralize all MCP tool connections, governance, security, and auth. Your AI can safely use MCP tools with centralized policy enforcement. [MCP Gateway resource](https://www.getmaxim.ai/resources/mcp-gateway.md). - **05 - Virtual Key Management:** Create different virtual keys for different use cases with independent budgets and access control. - **06 - Unified Interface:** One consistent API for all providers. Switch models without changing code. - **07 - Drop-in Replacement:** Replace your existing SDK with just one line change. Compatible with OpenAI, Anthropic, LiteLLM, Google GenAI, LangChain, and more. [Drop-in replacement docs](https://docs.getbifrost.ai/features/drop-in-replacement). - **08 - Built-in Observability:** Out-of-the-box OpenTelemetry support. Built-in dashboard for quick visibility without complex setup. - **09 - Community Support:** Active Discord community with responsive support and regular updates. ### Enterprise Features - **01 - Governance:** SAML support for SSO and role-based access control with policy enforcement for team collaboration. [Governance resource](https://www.getmaxim.ai/resources/governance.md). - **02 - Adaptive Load Balancing:** Automatically optimizes traffic distribution across provider keys and models based on real-time performance metrics. - **03 - Cluster Mode:** High availability deployment with automatic failover and load balancing. Peer-to-peer clustering where every instance is equal. - **04 - Alerts:** Real-time notifications for budget limits, failures, and performance issues on Email, Slack, PagerDuty, Teams, Webhook, and more. - **05 - Log Exports:** Export and analyze request logs, traces, and telemetry data from Bifrost with enterprise-grade data export for compliance, monitoring, and analytics. - **06 - Audit Logs:** Comprehensive logging and audit trails for compliance and debugging. - **07 - Vault Support:** Secure API key management with HashiCorp Vault, AWS Secrets Manager, Google Secret Manager, and Azure Key Vault integration. - **08 - VPC Deployment:** Deploy Bifrost within your private cloud infrastructure with VPC isolation, custom networking, and enhanced security controls. [Enterprise deployment resource](https://www.getmaxim.ai/resources/enterprise-deployment.md). - **09 - Guardrails:** Automatically detect and block unsafe model outputs with real-time policy enforcement and content moderation across all agents. [Guardrails resource](https://www.getmaxim.ai/resources/guardrails.md). ## FAQ ### Can I use multiple CLI agents with the same Bifrost gateway? Yes. Bifrost provides 100% compatible endpoints for OpenAI, Anthropic, and Gemini APIs, allowing you to route Claude Code, Codex CLI, Gemini CLI, and OpenCode through the same gateway simultaneously. Each agent gets access to all configured providers. [Read more about Bifrost CLI](https://docs.getbifrost.ai/quickstart/cli/getting-started). ### How do I switch models or providers for a specific CLI agent? You can configure model routing through Bifrost virtual keys, which specify which models each key can access. Alternatively, use Bifrost CLI (npx @maximhq/bifrost-cli) for guided setup that lets you select models before launching agents. The gateway handles provider routing automatically. ### Does routing CLI agents through Bifrost add latency? Bifrost adds approximately 11µs of gateway overhead per request at 5,000 RPS. This is effectively invisible compared to LLM response times that typically range from hundreds of milliseconds to several seconds. [Read more about Bifrost benchmarks](https://www.getmaxim.ai/bifrost/resources/benchmarks). ### Can I use MCP tools with all CLI agents through Bifrost? Claude Code has native MCP support that integrates directly with Bifrost MCP endpoints. Other agents can access MCP-connected tools through Bifrost's gateway, though the integration may vary by agent. The Bifrost CLI automatically registers MCP servers for Claude Code. ### How are costs tracked across different CLI agents? Bifrost logs every request with model, provider, token counts, and cost. Use virtual keys to tag requests by agent, team, or environment. Export via the logs API or OTEL to build cost breakdowns in your BI tool of choice. ### What happens if one provider goes down while using CLI agents? Bifrost automatically fails over to healthy providers. If Claude Code is configured to use Anthropic but hits rate limits, requests transparently route to AWS Bedrock or Google Vertex AI. Developers experience zero downtime. ## Related Resources - [Source: CLI Agents](https://www.getmaxim.ai/resources/cli-agents.md) - [Docs: Bifrost docs](https://docs.getbifrost.ai) - [GitHub: maximhq/bifrost](https://github.com/maximhq/bifrost) - [Pricing: Bifrost pricing](https://www.getmaxim.ai/pricing) - [Enterprise: Bifrost enterprise](https://www.getmaxim.ai/enterprise) - [Book a Demo: Bifrost demo](https://www.getmaxim.ai/book-a-demo) - [Resources: Bifrost resources](https://www.getmaxim.ai/resources.md) --- *This is a markdown version of [https://www.getmaxim.ai/bifrost/resources/cli-agents](https://www.getmaxim.ai/bifrost/resources/cli-agents) for AI/LLM consumption.* --- title: "Bifrost CLI | Launch Claude Code & Coding Agents Through Bifrost" description: "Interactive terminal tool to launch Claude Code, Codex CLI, Gemini CLI, and Opencode through Bifrost gateway with automatic configuration, MCP integration, and secure credential storage." url: "https://www.getmaxim.ai/bifrost/resources/bifrost-cli" markdown: "https://www.getmaxim.ai/bifrost/resources/bifrost-cli.md" --- # Bifrost CLI | Launch Claude Code & Coding Agents Through Bifrost > Interactive terminal tool to launch Claude Code, Codex CLI, Gemini CLI, and Opencode through Bifrost gateway with automatic configuration, MCP integration, and secure credential storage. ## Important Links - [View MCP Gateway](https://www.getmaxim.ai/resources/mcp-gateway.md) - [Features](https://www.getmaxim.ai/#features) - [Enterprise](https://www.getmaxim.ai/enterprise) - [Pricing](https://www.getmaxim.ai/pricing) - [Docs](https://docs.getbifrost.ai) - [GitHub](https://github.com/maximhq/bifrost) - [Book a Demo](https://www.getmaxim.ai/book-a-demo) ## Key Features - **Automatic - Zero-config setup:** Bifrost CLI handles all technical setup automatically, including base URLs, API keys, and model settings. No manual configuration files or environment variables to manage. - **Multi-session - Tabbed session management:** Work with multiple coding agents simultaneously in a persistent tabbed interface. Switch between sessions, open new tabs, or close existing ones using simple keyboard shortcuts. - **MCP ready - Native MCP integration:** When launching Claude Code, Bifrost CLI automatically registers Bifrost's MCP server, making configured tools immediately available without manual setup. - **OS keyring - Secure credential storage:** Virtual keys are stored securely in your OS keyring (macOS Keychain, Windows Credential Manager, or Linux Secret Service) rather than plaintext files. ## Setup Steps 1. **01 - Install Bifrost CLI.** Launch Bifrost CLI with a single npx command. No global installation required. Works with Node.js 18+ and connects to your running Bifrost gateway. ### Install Bifrost CLI ``` npx -y @maximhq/bifrost-cli ``` 2. **02 - Configure gateway.** Enter your Bifrost gateway URL (default: localhost:8080), optionally provide a virtual key, and the CLI will fetch available models and configure everything automatically. ### Configure gateway ``` # CLI prompts you for: # 1. Gateway URL # 2. Virtual key (optional) # 3. Select agent (Claude Code, etc) # 4. Choose model # 5. Review and launch ``` 3. **03 - Launch and code.** Select your preferred coding agent and model through the guided setup. Bifrost CLI installs missing agents via npm automatically and launches your session with full MCP integration. ### Launch and code ``` # Session starts with: # - Tabbed terminal interface # - Activity badges (active/idle status) # - MCP tools auto-registered # - Keyboard shortcuts enabled ``` ## CLI Features - **Smart agent switching:** Switch between Claude Code, Codex CLI, Gemini CLI, and Opencode before launching. The CLI remembers your previous selections across sessions. - **Virtual key management:** Securely store and manage virtual keys for different teams or projects. Keys are encrypted in your OS keyring and never touch plaintext files. - **Session activity badges:** Visual indicators show session status at a glance with clear markers for active processing, idle sessions, and alerts that need attention. - **Persistent configuration:** Settings stored in ~/.bifrost/config.json include gateway URL, default agent, and model preferences. Use -config flag for custom config paths. - **Git worktree support:** Create isolated git worktrees for Claude Code using the -worktree flag, keeping experimental changes separate from your main branch. - **Auto-install missing agents:** Don't have Claude Code or Codex CLI installed? Bifrost CLI automatically installs missing agents via npm when you select them. ## Enterprise Benefits - **Enterprise-grade security:** All coding agent traffic routes through Bifrost gateway, applying guardrails, PII redaction, and content policies transparently. - **Multi-provider routing:** Access 1000+ models through a single interface. Switch between Anthropic, OpenAI, Google, and more without reconfiguring your agent. - **Centralized observability:** Every request from your coding agents is logged with full metadata including user, team, model, and latency. Export to any observability stack. ## Features ### OSS Features - **01 - Model Catalog:** Access 8+ providers and 1000+ AI models through a unified interface. Also supports custom deployed models. - **02 - Budgeting:** Set spending limits and track costs across teams, projects, and models. - **03 - Provider Fallback:** Automatic failover between providers ensures 99.99% uptime for your applications. - **04 - MCP Gateway:** Centralize all MCP tool connections, governance, security, and auth. Your AI can safely use MCP tools with centralized policy enforcement. [MCP Gateway resource](https://www.getmaxim.ai/resources/mcp-gateway.md). - **05 - Virtual Key Management:** Create different virtual keys for different use cases with independent budgets and access control. - **06 - Unified Interface:** One consistent API for all providers. Switch models without changing code. - **07 - Drop-in Replacement:** Replace your existing SDK with just one line change. Compatible with OpenAI, Anthropic, LiteLLM, Google GenAI, LangChain, and more. [Drop-in replacement docs](https://docs.getbifrost.ai/features/drop-in-replacement). - **08 - Built-in Observability:** Out-of-the-box OpenTelemetry support. Built-in dashboard for quick visibility without complex setup. - **09 - Community Support:** Active Discord community with responsive support and regular updates. ### Enterprise Features - **01 - Governance:** SAML support for SSO and role-based access control with policy enforcement for team collaboration. [Governance resource](https://www.getmaxim.ai/resources/governance.md). - **02 - Adaptive Load Balancing:** Automatically optimizes traffic distribution across provider keys and models based on real-time performance metrics. - **03 - Cluster Mode:** High availability deployment with automatic failover and load balancing. Peer-to-peer clustering where every instance is equal. - **04 - Alerts:** Real-time notifications for budget limits, failures, and performance issues on Email, Slack, PagerDuty, Teams, Webhook, and more. - **05 - Log Exports:** Export and analyze request logs, traces, and telemetry data from Bifrost with enterprise-grade data export for compliance, monitoring, and analytics. - **06 - Audit Logs:** Comprehensive logging and audit trails for compliance and debugging. - **07 - Vault Support:** Secure API key management with HashiCorp Vault, AWS Secrets Manager, Google Secret Manager, and Azure Key Vault integration. - **08 - VPC Deployment:** Deploy Bifrost within your private cloud infrastructure with VPC isolation, custom networking, and enhanced security controls. [Enterprise deployment resource](https://www.getmaxim.ai/resources/enterprise-deployment.md). - **09 - Guardrails:** Automatically detect and block unsafe model outputs with real-time policy enforcement and content moderation across all agents. [Guardrails resource](https://www.getmaxim.ai/resources/guardrails.md). ## FAQ ### What coding agents does Bifrost CLI support? [Bifrost CLI](https://docs.getbifrost.ai/quickstart/cli/getting-started) supports popular coding agents like Claude Code, Codex CLI, Gemini CLI, and Opencode. All agents work seamlessly with the Bifrost gateway for multi-provider routing and enterprise controls. ### How does MCP integration work with Claude Code? When launching Claude Code through Bifrost CLI, the CLI automatically registers Bifrost's [MCP server](https://docs.getbifrost.ai/mcp/overview), making all configured tools immediately available. This includes filesystem tools, database connectors, and custom integrations without any manual configuration on your part. ### Where are credentials and configuration stored? Virtual keys are stored securely in your operating system keyring (macOS Keychain, Windows Credential Manager, or Linux Secret Service). Configuration settings like gateway URL and default preferences are stored in ~/.bifrost/config.json. You can specify custom config paths with the -config flag. ### Can I use Bifrost CLI with multiple projects or teams? Yes. Bifrost CLI supports virtual keys, allowing you to create separate credentials for different teams or projects. You can also use the -config flag to maintain different configuration files, and the -worktree flag to create isolated git environments for each project. [Read more about virtual keys](https://docs.getbifrost.ai/features/governance/virtual-keys). ## Related Resources - [Source: Bifrost CLI](https://www.getmaxim.ai/resources/bifrost-cli.md) - [Docs: Bifrost docs](https://docs.getbifrost.ai) - [GitHub: maximhq/bifrost](https://github.com/maximhq/bifrost) - [Pricing: Bifrost pricing](https://www.getmaxim.ai/pricing) - [Enterprise: Bifrost enterprise](https://www.getmaxim.ai/enterprise) - [Book a Demo: Bifrost demo](https://www.getmaxim.ai/book-a-demo) - [Resources: Bifrost resources](https://www.getmaxim.ai/resources.md) --- *This is a markdown version of [https://www.getmaxim.ai/bifrost/resources/bifrost-cli](https://www.getmaxim.ai/bifrost/resources/bifrost-cli) for AI/LLM consumption.* --- title: "Bifrost for AWS Bedrock | Enterprise AI Gateway with Governance & Guardrails" description: "Add governance, guardrails, multi-region failover, and cost controls to AWS Bedrock. Bifrost is the enterprise AI gateway with native Bedrock SDK compatibility. Zero code changes." url: "https://www.getmaxim.ai/bifrost/resources/aws-bedrock" markdown: "https://www.getmaxim.ai/bifrost/resources/aws-bedrock.md" --- # Bifrost for AWS Bedrock | Enterprise AI Gateway with Governance & Guardrails > Add governance, guardrails, multi-region failover, and cost controls to AWS Bedrock. Bifrost is the enterprise AI gateway with native Bedrock SDK compatibility. Zero code changes. ## Important Links - [View MCP Gateway](https://www.getmaxim.ai/resources/mcp-gateway.md) - [Features](https://www.getmaxim.ai/#features) - [Enterprise](https://www.getmaxim.ai/enterprise) - [Pricing](https://www.getmaxim.ai/pricing) - [Docs](https://docs.getbifrost.ai) - [GitHub](https://github.com/maximhq/bifrost) - [Book a Demo](https://www.getmaxim.ai/book-a-demo) ## Performance at a Glance - **11µs Internal Overhead.** Ultra-low latency at high throughput - **5,000 req/s Throughput.** Sustained with 100% success rate - **0 Code Changes.** Native Bedrock SDK drop-in - **1000+ Models Supported.** Multi-provider escape hatch ## Arch Cards - **Your Applications:** Existing boto3 Bedrock clients; Converse API and Invoke API calls; streaming and batch workloads; SDK-compatible endpoint configuration - **Bifrost Gateway:** Governance and budget enforcement; Bedrock Guardrails integration; Multi-region routing and failover; Observability and audit logs ## Core Capabilities - **Zero code changes - Native Bedrock SDK Drop-in:** Point boto3 at Bifrost's /bedrock endpoint. Converse, Invoke, and streaming just work. IAM role auth supported. - **99.99% uptime - Multi-Region Failover:** Set up weighted routing across Bedrock regions. When us-east-1 throttles, Bifrost shifts traffic to us-west-2. Add Anthropic or OpenAI as a fallback tier. - **Gateway-level enforcement - Bedrock Guardrails Enforcement:** Attach Bedrock Guardrail ARNs at the gateway level. Layer with Azure Content Safety or Patronus AI for defense-in-depth. - **Real-time enforcement - Per-Team Budget Controls:** Virtual keys give every team, project, or environment its own spending cap, rate limit, and model allowlist, all enforced before requests reach Bedrock. - **Full data isolation - In-VPC Deployment:** Bifrost runs entirely inside your AWS VPC. API keys stay in Secrets Manager with automatic sync and zero-downtime rotation. - **Built-in tracing - Full Request Observability:** Every request logged with prompt/response metadata, latency, tokens, cost, and guardrail actions. Built-in dashboard, OpenTelemetry, Datadog, or BigQuery. ## Setup Steps 1. **01 - Deploy Bifrost.** Run Bifrost inside your VPC with Docker or Kubernetes. Zero configuration needed to start. ### Deploy Bifrost ``` # pull and start bifrost docker pull maximhq/bifrost docker run -p 8080:8080 maximhq/bifrost ``` 2. **02 - Configure Bedrock Provider.** Add your Bedrock regions and IAM credentials in the Bifrost config. Supports multiple regions with weighted routing. ### Configure Bedrock Provider ``` # config.json - Bedrock provider # region: us-east-1, weight: 1.0 # arn: inference-profile ARN # models: claude-3-sonnet, etc. ``` 3. **03 - Update endpoint_url.** Point your existing boto3 client at Bifrost. One line change. Your Converse and Invoke API calls work as-is. ### Update endpoint_url ``` # Python - one line change # endpoint_url="http://bifrost:8080/bedrock" # that's it - boto3 just works ``` ## Guardrail Features - **PII Detection:** Connect your existing Bedrock Guardrail ARN for automatic PII filtering across all requests. - **Content Filtering:** Block harmful outputs across all teams and models at the gateway level. - **Prompt Injection Prevention:** Real-time input validation before requests reach Bedrock. - **Image Scanning:** PNG and JPEG content scanning through Bedrock Guardrails. - **Defense-in-Depth:** Stack Bedrock + Azure Content Safety + Patronus AI for layered protection. - **Usage Metrics:** Detailed guardrail invocation metrics for cost tracking and monitoring. ## Comparison Rows | Capability | Native | Bifrost | | --- | --- | --- | | Multi-region failover | CRI profiles | Automatic with health monitoring | | Cross-provider fallback | Not available | 20+ providers as escape hatch | | Per-team budget caps | IAM-based, no real-time limits | Virtual keys with real-time enforcement | | Guardrails enforcement | Native | Gateway-level, global enforcement | | Semantic caching | Not available | Built-in, configurable per key | | Request observability | CloudWatch logs | Full tracing, OTEL, Datadog, BigQuery | | MCP tool governance | Not available | Per-key filtering with audit trails | | API key rotation | Manual | Auto via Secrets Manager sync | | Added latency | - | 11µs | ## Security Cards - **VPC Isolation:** Full In-VPC deployment with private networking and zero external data exposure. - **AWS Secrets Manager:** Native integration with automatic key sync and zero-downtime rotation. - **Vault Support:** Also works with HashiCorp Vault, Google Secret Manager, and Azure Key Vault. - **Immutable Audit Trails:** Every request logged for SOC 2 Type II, GDPR, HIPAA, and ISO 27001 compliance. - **RBAC:** OpenID Connect integration with user-level governance and team sync. - **Log Exports:** Automated export to S3, data lakes, or your SIEM. ## Use Cases - **Multi-team AI platform:** Central platform team deploys Bifrost as the AI gateway. Each product team gets a virtual key with independent budgets, model allowlists, and guardrail policies. - **Cross-region resilience:** Production agents route through Bifrost with Bedrock us-east-1 as primary and us-west-2 as fallback. If the primary throttles, Bifrost switches automatically. Anthropic Direct serves as a third-tier escape hatch. - **Regulated industry compliance:** Healthcare or financial services teams needing HIPAA/SOC 2 Type II for all AI interactions. Bifrost deploys in-VPC, enforces Bedrock Guardrails for PII detection, logs immutable audit trails, and exports to SIEM. - **Cost optimization at scale:** Customer support agents on Bedrock with semantic caching for repetitive queries. Similar questions return cached responses instantly, cutting token spend. Per-team budgets prevent overspend. - **MCP tool governance:** Bifrost centralizes MCP tool access with per-key filtering and audit trails. Code Mode cuts token usage by 50% by orchestrating tools via Python. - **Multi-provider flexibility:** Start on Bedrock for compliance, but route specific workloads to OpenAI for cost or Groq for speed. Your application code never changes when you add providers. ## Why Teams Choose Bifrost for Bedrock - **11µs overhead at 5,000 requests per second:** - **Native Bedrock SDK compatibility (Converse + Invoke API):** - **Zero code changes, one endpoint_url update:** - **In-VPC deployment with AWS Secrets Manager integration:** - **Bedrock Guardrails enforced globally at the gateway:** - **Multi-region failover with adaptive load balancing:** - **Per-team budgets, rate limits, and model allowlists:** - **Complete audit trails with OpenTelemetry and Datadog/BigQuery export:** - **Open source (Apache 2.0) with enterprise support:** - **Go-native with zero Python GIL bottleneck:** ## Compliance Badges - **Audited quarterly - SOC 2 Type II:** - **EU data residency - GDPR:** - **BAA available - HIPAA:** - **Certified - ISO 27001:** ## Features ### OSS Features - **01 - Model Catalog:** Access 8+ providers and 1000+ AI models through a unified interface. Also supports custom deployed models. - **02 - Budgeting:** Set spending limits and track costs across teams, projects, and models. - **03 - Provider Fallback:** Automatic failover between providers ensures 99.99% uptime for your applications. - **04 - MCP Gateway:** Centralize all MCP tool connections, governance, security, and auth. Your AI can safely use MCP tools with centralized policy enforcement. [MCP Gateway resource](https://www.getmaxim.ai/resources/mcp-gateway.md). - **05 - Virtual Key Management:** Create different virtual keys for different use cases with independent budgets and access control. - **06 - Unified Interface:** One consistent API for all providers. Switch models without changing code. - **07 - Drop-in Replacement:** Replace your existing SDK with just one line change. Compatible with OpenAI, Anthropic, LiteLLM, Google GenAI, LangChain, and more. [Drop-in replacement docs](https://docs.getbifrost.ai/features/drop-in-replacement). - **08 - Built-in Observability:** Out-of-the-box OpenTelemetry support. Built-in dashboard for quick visibility without complex setup. - **09 - Community Support:** Active Discord community with responsive support and regular updates. ### Enterprise Features - **01 - Governance:** SAML support for SSO and role-based access control with policy enforcement for team collaboration. [Governance resource](https://www.getmaxim.ai/resources/governance.md). - **02 - Adaptive Load Balancing:** Automatically optimizes traffic distribution across provider keys and models based on real-time performance metrics. - **03 - Cluster Mode:** High availability deployment with automatic failover and load balancing. Peer-to-peer clustering where every instance is equal. - **04 - Alerts:** Real-time notifications for budget limits, failures, and performance issues on Email, Slack, PagerDuty, Teams, Webhook, and more. - **05 - Log Exports:** Export and analyze request logs, traces, and telemetry data from Bifrost with enterprise-grade data export for compliance, monitoring, and analytics. - **06 - Audit Logs:** Comprehensive logging and audit trails for compliance and debugging. - **07 - Vault Support:** Secure API key management with HashiCorp Vault, AWS Secrets Manager, Google Secret Manager, and Azure Key Vault integration. - **08 - VPC Deployment:** Deploy Bifrost within your private cloud infrastructure with VPC isolation, custom networking, and enhanced security controls. [Enterprise deployment resource](https://www.getmaxim.ai/resources/enterprise-deployment.md). - **09 - Guardrails:** Automatically detect and block unsafe model outputs with real-time policy enforcement and content moderation across all agents. [Guardrails resource](https://www.getmaxim.ai/resources/guardrails.md). ## FAQ ### Do I need to change my application code? No. Point your boto3 client's endpoint_url to Bifrost. Your existing Bedrock SDK code works as-is. No SDK swap, no API format changes. ### Does Bifrost support IAM role authentication? Yes. When both access_key and secret_key are empty, Bifrost automatically uses the IAM role attached to the host. No credentials need to be exposed. ### Can I use my existing Bedrock Guardrails? Yes. Bifrost integrates directly with your Bedrock Guardrail ARNs. Enforce them globally at the gateway without per-request application code. ### How much latency does Bifrost add? 11 microseconds per request at 5,000 req/s sustained throughput. [Read more about Bifrost benchmarks](https://www.getmaxim.ai/bifrost/resources/benchmarks). ### Where does Bifrost deploy? Inside your VPC, next to your Bedrock endpoints. Supports Kubernetes, Docker, and direct binary deployment. No data leaves your private network. ### Can I fall back to non-Bedrock providers? Yes. Configure fallback chains like Bedrock us-east-1 → Bedrock us-west-2 → Anthropic Direct → OpenAI. Your applications never go down even if an entire region is unavailable. ### How does Bifrost handle API key rotation? Native AWS Secrets Manager integration with periodic sync. Keys rotate with zero downtime. Also supports HashiCorp Vault, Google Secret Manager, and Azure Key Vault. ### Is Bifrost open source? Yes. Apache 2.0 license. Enterprise features including guardrails, adaptive load balancing, clustering, in-VPC deployments, and audit logs are available with a 14-day free trial. [Get 14 days free enterprise trial](https://www.getmaxim.ai/enterprise). ## Related Resources - [Source: AWS Bedrock](https://www.getmaxim.ai/resources/aws-bedrock.md) - [Docs: Bifrost docs](https://docs.getbifrost.ai) - [GitHub: maximhq/bifrost](https://github.com/maximhq/bifrost) - [Pricing: Bifrost pricing](https://www.getmaxim.ai/pricing) - [Enterprise: Bifrost enterprise](https://www.getmaxim.ai/enterprise) - [Book a Demo: Bifrost demo](https://www.getmaxim.ai/book-a-demo) - [Resources: Bifrost resources](https://www.getmaxim.ai/resources.md) --- *This is a markdown version of [https://www.getmaxim.ai/bifrost/resources/aws-bedrock](https://www.getmaxim.ai/bifrost/resources/aws-bedrock) for AI/LLM consumption.* --- title: "MCP Gateway | High-Performance Tool Execution for AI Agents" description: "Connect AI models to external tools with an open-source MCP gateway that delivers 11µs overhead and complete security control." url: "https://www.getmaxim.ai/bifrost/resources/mcp-gateway" markdown: "https://www.getmaxim.ai/bifrost/resources/mcp-gateway.md" --- # MCP Gateway | High-Performance Tool Execution for AI Agents > Connect AI models to external tools with an open-source MCP gateway that delivers 11µs overhead and complete security control. ## Important Links - [View MCP Gateway](https://www.getmaxim.ai/resources/mcp-gateway.md) - [Features](https://www.getmaxim.ai/#features) - [Enterprise](https://www.getmaxim.ai/enterprise) - [Pricing](https://www.getmaxim.ai/pricing) - [Docs](https://docs.getbifrost.ai) - [GitHub](https://github.com/maximhq/bifrost) - [Book a Demo](https://www.getmaxim.ai/book-a-demo) ## Performance at a Glance - **11µs Internal Overhead.** Ultra-low latency at high throughput - **50%+ Token Savings.** With Code Mode vs classic MCP - **40% Faster Execution.** Code Mode execution pipeline - **1000+ Model Support.** LLM models supported ## Setup Steps 1. **01 - Register MCP servers.** Connect Bifrost to any MCP-compliant server. Bifrost auto-discovers available tools and their schemas at startup. ### Register MCP servers ``` # bifrost config mcp_servers: - name: filesystem transport: stdio command: npx @modelcontextprotocol/server-filesystem ``` 2. **02 - Send a chat request.** Your app sends a standard chat completion request. Bifrost injects discovered MCP tools into the request automatically. ### Send a chat request ``` curl http://localhost:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model": "claude-sonnet", "messages": [...]}' ``` 3. **03 - Execute tool calls.** When the LLM suggests a tool call, your app decides whether to execute it. Bifrost handles the MCP protocol and returns results. ### Execute tool calls ``` # tool call returned in response # your app approves, then Bifrost executes # full audit trail logged automatically ``` ## Core Capabilities - **STDIO + HTTP + SSE - Connect to MCP servers:** Connect to any MCP-compliant server via STDIO, HTTP, or SSE. Bifrost auto-discovers tools and their schemas at runtime so your AI models can use them immediately. - **OAuth 2.0 with automatic token refresh - OAuth Authentication:** Secure OAuth 2.0 authentication with automatic token refresh - **Security-first - Explicit tool execution:** Tool calls from LLMs are suggestions only. Execution requires a separate API call, giving your app full control to validate, filter, and approve every action before it runs. - **Configurable auto-approval - Agent Mode:** Enable autonomous multi-step tool execution with configurable auto-approval. Specify exactly which tools can auto-execute while keeping human oversight for sensitive operations. - **Token efficiency - Code Mode:** AI writes Python to orchestrate multiple tools. Four meta-tools replace 100+ definitions with on-demand schema loading and sandbox execution. Cuts tokens by 50%+ and LLM calls by 3-4x. - **Single gateway URL - MCP Gateway URL:** A single endpoint for tool discovery, execution, and management. ## Security Principles - **Explicit execution:** Tool calls from LLMs are suggestions only. Execution requires a separate API call from your application. - **Granular control:** Filter tools per-request, per-client, or per-virtual-key. Blacklist dangerous tools globally. - **Opt-in auto-execution:** Agent Mode with auto-execution must be explicitly configured. Specify exactly which tools are allowed. - **Stateless design:** Each API call is independent. Your app controls conversation state with full audit trails at every step. ## Comparison Data | Feature | Classic | Bifrost | | --- | --- | --- | | Tool definition overhead | 100+ tool schemas sent every request | AI writes code to call tools | | Token usage | High (all tool schemas in context) | 50%+ reduction | | Execution latency | Multiple round-trips per tool | 40% faster execution | | Multi-tool orchestration | Sequential tool calls only | Python orchestrates in one pass | | Scalability with servers | Degrades with 3+ servers | Scales to any number | | Error handling | LLM retries each tool call | Python try/catch in sandbox | ## Virtual Keys and Virtual MCP Servers Production MCP needs governance at the gateway. - **Per-consumer scoping - Virtual keys:** Issue scoped credentials for every user, team, or customer integration. Each virtual key carries a tool-level allowlist, not just server access, so the model only receives definitions for granted tools. - **MCP Tool Groups - Virtual MCP server:** MCP Tool Groups bundle selected tools from any connected server into one curated virtual MCP server for a team or customer. Define a group once, attach it to virtual keys, and Bifrost exposes only those tools at request time. - **Per-tool visibility - Audit and cost tracking:** Every tool execution is logged with tool name, server, latency, virtual key, and parent LLM request. Per-tool pricing surfaces MCP tool spend alongside LLM token costs in one view. ## Connection Types - **Local tools - STDIO:** Local process execution via stdin/stdout. - **Microservices - HTTP:** Remote MCP servers via HTTP requests. - **Live data - SSE:** Persistent streaming for real-time data. ## Use Cases - **Agentic coding pipelines:** Connect AI coding agents to filesystem tools, databases, and deployment pipelines. Bifrost handles tool injection transparently with full audit trails for every operation. - **Regulated enterprise environments:** Deploy in healthcare, finance, or government with explicit approval workflows, PII redaction, and tamper-evident audit logs for SOC 2 and HIPAA compliance. - **Multi-tool orchestration:** Coordinate filesystem operations, database queries, and API calls in a single request using Code Mode. Reduce token waste and latency when using 3+ MCP servers. - **DevOps & infrastructure automation:** Supervised infrastructure actions and deployments with role-based tool access. Only approved tools execute, with complete visibility into every automated step. - **Centralized tool governance:** Manage tool access across teams with virtual keys and per-key tool filtering. Set different tool policies for development, staging, and production environments. - **Claude Desktop & MCP clients:** Expose your entire tool ecosystem through a single Bifrost gateway URL. Claude Desktop and other MCP clients connect once and discover all available tools automatically. ## Architecture Features - **External MCP server connections:** Bifrost connects to external MCP servers such as filesystem tools, web search, databases, and custom APIs. It discovers their capabilities automatically. - **Single gateway endpoint:** Bifrost exposes all connected tools through a single gateway URL. MCP clients like Claude Desktop connect to Bifrost and access everything. ## The Fastest Open-Source MCP Gateway - **11µs overhead at 5,000 requests per second:** - **Stateless architecture with explicit approval:** - **Code Mode: 50% fewer tokens, 40% faster execution:** - **Dual role: MCP Client and MCP Server:** - **Built-in OAuth 2.0 with automatic token refresh:** - **Production-proven at millions of requests/day:** - **Complete audit trails and OpenTelemetry export:** - **Open source (Apache 2.0) with enterprise support:** - **Go-native with zero Python GIL bottleneck:** ## Read the MCP Gateway Deep Dive Virtual keys, MCP Tool Groups, Code Mode benchmarks, and how production teams govern tool access while cutting context cost at scale. - [Read Full Article: Bifrost MCP Gateway deep dive](https://www.getmaxim.ai/blog/bifrost-mcp-gateway-access-control-cost-governance-and-92-lower-token-costs-at-scale) ## Features ### OSS Features - **01 - Model Catalog:** Access 8+ providers and 1000+ AI models through a unified interface. Also supports custom deployed models. - **02 - Budgeting:** Set spending limits and track costs across teams, projects, and models. - **03 - Provider Fallback:** Automatic failover between providers ensures 99.99% uptime for your applications. - **04 - MCP Gateway:** Centralize all MCP tool connections, governance, security, and auth. Your AI can safely use MCP tools with centralized policy enforcement. [MCP Gateway resource](https://www.getmaxim.ai/resources/mcp-gateway.md). - **05 - Virtual Key Management:** Create different virtual keys for different use cases with independent budgets and access control. - **06 - Unified Interface:** One consistent API for all providers. Switch models without changing code. - **07 - Drop-in Replacement:** Replace your existing SDK with just one line change. Compatible with OpenAI, Anthropic, LiteLLM, Google GenAI, LangChain, and more. [Drop-in replacement docs](https://docs.getbifrost.ai/features/drop-in-replacement). - **08 - Built-in Observability:** Out-of-the-box OpenTelemetry support. Built-in dashboard for quick visibility without complex setup. - **09 - Community Support:** Active Discord community with responsive support and regular updates. ### Enterprise Features - **01 - Governance:** SAML support for SSO and role-based access control with policy enforcement for team collaboration. [Governance resource](https://www.getmaxim.ai/resources/governance.md). - **02 - Adaptive Load Balancing:** Automatically optimizes traffic distribution across provider keys and models based on real-time performance metrics. - **03 - Cluster Mode:** High availability deployment with automatic failover and load balancing. Peer-to-peer clustering where every instance is equal. - **04 - Alerts:** Real-time notifications for budget limits, failures, and performance issues on Email, Slack, PagerDuty, Teams, Webhook, and more. - **05 - Log Exports:** Export and analyze request logs, traces, and telemetry data from Bifrost with enterprise-grade data export for compliance, monitoring, and analytics. - **06 - Audit Logs:** Comprehensive logging and audit trails for compliance and debugging. - **07 - Vault Support:** Secure API key management with HashiCorp Vault, AWS Secrets Manager, Google Secret Manager, and Azure Key Vault integration. - **08 - VPC Deployment:** Deploy Bifrost within your private cloud infrastructure with VPC isolation, custom networking, and enhanced security controls. [Enterprise deployment resource](https://www.getmaxim.ai/resources/enterprise-deployment.md). - **09 - Guardrails:** Automatically detect and block unsafe model outputs with real-time policy enforcement and content moderation across all agents. [Guardrails resource](https://www.getmaxim.ai/resources/guardrails.md). ## FAQ ### What is an MCP gateway and why do I need one? An MCP (Model Context Protocol) gateway connects AI models to external tools like filesystems, databases, and APIs. Without a gateway, each AI client needs individual tool configurations. Bifrost centralizes tool management, adds security controls, and provides audit trails for every tool execution. ### Does Bifrost automatically execute tool calls from AI models? No. By default, Bifrost treats tool calls from LLMs as suggestions only. Your application must explicitly approve and trigger execution via a separate API call. This security-first design prevents unintended actions. Agent Mode with auto-execution is available but requires explicit opt-in configuration. ### How do I manage rate limits and resource usage across MCP servers? An MCP gateway solves the problem of runaway tool-calling that can overload internal systems or hit provider API limits. The main objective is to regulate resource consumption while maintaining a smooth developer experience. Key features for resource management include: • Token & Request Budgeting: Set hard limits on tool calls per team to prevent backend system overloads and control costs. • Automated Failover: Reroute traffic to secondary servers or models if an MCP connection times out or fails. • Scale-Ready Architecture: Built to handle thousands of tool-calls concurrently without degrading performance or reliability. ### What is Code Mode and how does it reduce costs? Code Mode replaces traditional tool calling with AI-generated Python code that orchestrates multiple tools in a single round-trip. Instead of sending 100+ tool schemas in every request, Code Mode uses four meta-tools for on-demand schema loading. This cuts token usage by 50%+ and reduces LLM calls by 3-4x. ### What MCP transport protocols does Bifrost support? Bifrost supports all three MCP transport types: STDIO for local process execution, HTTP for remote MCP servers, and SSE (Server-Sent Events) for real-time streaming connections. OAuth 2.0 authentication with automatic token refresh is built in. ### Can I use Bifrost as an MCP server for Claude Desktop? Yes. Bifrost acts as both an MCP client (connecting to external tool servers) and an MCP server (exposing tools to clients). Claude Desktop and other MCP-compatible clients can connect to a single Bifrost gateway URL to discover and use all registered tools. ### What are virtual keys in Bifrost MCP Gateway? Virtual keys are scoped credentials for each consumer of your MCP gateway: a user, team, or customer integration. Each key defines which tools it may call at the tool level, not just per server, so customer-facing agents cannot reach internal admin tooling. ### What is a Virtual MCP server? A Virtual MCP server is a curated toolkit your agents see at the gateway, built from MCP Tool Groups. Each group is a named collection of tools from one or more backend MCP servers. Define a group once, attach it to virtual keys, teams, or customers, and Bifrost resolves allowed tools in memory at request time without duplicates. ## Related Resources - [Source: MCP Gateway](https://www.getmaxim.ai/resources/mcp-gateway.md) - [MCP documentation: docs.getbifrost.ai/mcp/overview](https://docs.getbifrost.ai/mcp/overview) - [Blog: MCP Gateway deep dive](https://www.getmaxim.ai/blog/bifrost-mcp-gateway-access-control-cost-governance-and-92-lower-token-costs-at-scale) - [Docs: Bifrost docs](https://docs.getbifrost.ai) - [GitHub: maximhq/bifrost](https://github.com/maximhq/bifrost) - [Pricing: Bifrost pricing](https://www.getmaxim.ai/pricing) - [Enterprise: Bifrost enterprise](https://www.getmaxim.ai/enterprise) - [Book a Demo: Bifrost demo](https://www.getmaxim.ai/book-a-demo) - [Resources: Bifrost resources](https://www.getmaxim.ai/resources.md) --- *This is a markdown version of [https://www.getmaxim.ai/bifrost/resources/mcp-gateway](https://www.getmaxim.ai/bifrost/resources/mcp-gateway) for AI/LLM consumption.* --- title: "Bifrost Governance | Virtual Keys, Budgets & Enterprise RBAC" description: "Comprehensive governance for LLM access with virtual keys, budgets, rate limits, routing, MCP tool filtering, and enterprise RBAC with SSO integration." url: "https://www.getmaxim.ai/bifrost/resources/governance" markdown: "https://www.getmaxim.ai/bifrost/resources/governance.md" --- # Bifrost Governance | Virtual Keys, Budgets & Enterprise RBAC > Comprehensive governance for LLM access with virtual keys, budgets, rate limits, routing, MCP tool filtering, and enterprise RBAC with SSO integration. ## Important Links - [View MCP Gateway](https://www.getmaxim.ai/resources/mcp-gateway.md) - [Features](https://www.getmaxim.ai/#features) - [Enterprise](https://www.getmaxim.ai/enterprise) - [Pricing](https://www.getmaxim.ai/pricing) - [Docs](https://docs.getbifrost.ai) - [GitHub](https://github.com/maximhq/bifrost) - [Book a Demo](https://www.getmaxim.ai/book-a-demo) ## OSS Governance Features - **OSS - Virtual keys:** Primary governance entity providing authentication, access control, budgets, and rate limits per consumer. Support for multiple header formats (OpenAI, Anthropic, Gemini-compatible). - **OSS - Intelligent routing:** Weighted load balancing across providers with automatic failover. Model and provider restrictions per virtual key with API key-level binding for environment separation. - **OSS - Hierarchical budgets:** Independent cost tracking at Business Units → Team → Virtual Key → Provider levels. Costs calculated from the model catalog and deducted across all applicable tiers automatically. - **OSS - Rate limits:** Token and request-based throttling at provider and virtual key levels. Flexible reset durations from 1 minute to 1 month with automatic enforcement. - **OSS - MCP tool filtering:** Allow-list controls for Model Context Protocol tools per virtual key. Fine-grained permissions with wildcard support and automatic header generation. - **OSS - Required headers:** Enforce custom HTTP headers on every request for tenant isolation, audit trails, and routing metadata. Case-insensitive validation with 400 rejection. ## Enterprise Governance Features - **Enterprise - Role-based access control:** Fine-grained permissions with custom roles across Bifrost resources. Pre-configured Admin, Developer, and Viewer roles plus custom roles. - **Enterprise - SSO integration:** OpenID Connect with Okta, Zitadel, Keycloak, and Microsoft Entra. Automatic role assignment from IdP groups with highest-privilege logic for multi-group users. - **Enterprise - Team synchronization:** Automatic team creation from identity provider groups. Hierarchical structure from Business Units through team to individual user levels. - **Enterprise - User-level governance:** Individual authentication and budget allocation. Personal access tracking with per-user budget enforcement. - **Enterprise - Comprehensive audit logs:** Access, usage, data, and compliance-focused reporting. Audit trails and report generation for security posture and regulatory health. - **Enterprise - Compliance frameworks:** SOC 2 Type II, GDPR, ISO 27001, and HIPAA compliance. Automated controls with risk assessment and regulatory reporting. ## Virtual Key Capabilities - **Model filtering:** Restrict which AI models users can access - **Provider control:** Limit access to specific AI providers - **Budget management:** Independent cost tracking per virtual key - **Rate limiting:** Token and request-based throttling - **API key restrictions:** Bind to specific provider keys - **Status control:** Instantly enable or disable access | Header | Format | Provider | | --- | --- | --- | | x-bf-vk | sk-bf-* | Native Bifrost | | Authorization | Bearer | OpenAI-compatible | | x-api-key | sk-ant-* | Anthropic-compatible | | x-goog-api-key | AI* | Gemini-compatible | ## Routing Features - **Adaptive load balancing:** Automatically optimizes traffic distribution across providers and keys based on real-time performance metrics. - **Automatic failover:** Create fallback chains ordered by weight when primary providers fail or hit rate limits - **Provider restrictions:** Whitelist specific provider-model combinations with empty array defaulting to catalog detection - **API key binding:** Restrict VKs to specific provider API keys for environment separation (dev/test/prod) ## Governance Deep Dive Virtual keys, hierarchical budgets, weighted routing with automatic failover, and how production teams enforce per-consumer controls without slowing developers. - [Read Full Article](https://www.getmaxim.ai/articles/enterprise-ai-gateway-per-user-rate-limiting-budget-controls-and-automatic-fallbacks/) ## Hierarchical Budgets Independent cost tracking at Customer, Team, Virtual Key, and Provider levels with automatic deduction across all tiers. - **Level 1: Customer:** Top-level organization with independent budget - **Level 2: Team:** Department-level budget within customer - **Level 3: Virtual Key:** Individual access token budget - **Level 4: Provider Config:** Per-provider spending limits > **All Applicable Budgets Must Pass.** When a transaction occurs, the same cost deducts from every relevant level simultaneously. A single exhausted budget at any tier blocks the entire request. ## Role-Based Access Control Benefits - **Principle of least privilege:** Users receive only the permissions they need for their job function, reducing security vulnerabilities and preventing accidental misconfigurations. - **Simplified user management:** Assign roles once instead of configuring individual permissions. New team members inherit appropriate access automatically through role assignment. - **Audit-ready access tracking:** Demonstrate to auditors exactly who has what access. Audit logs track permission changes over time for compliance frameworks like SOC 2 Type II and GDPR. - **Custom roles for specialized teams:** Create tailored roles for QA teams, security auditors, or compliance officers. Custom roles adapt to your organizational structure. ## Three Pre-Configured Roles - **Admin:** Full control over all Bifrost resources and configurations. Use case: Platform engineers, security admins. - **Developer:** Manage technical resources without administrative privileges. Use case: Engineering teams, DevOps. - **Viewer:** Read-only access for monitoring and compliance. Use case: Finance, compliance, executives. ## SSO Providers - **Okta:** OIDC web application; Custom bifrostRole attribute; Group-to-role mappings; Authorization Code + Refresh Token - **Microsoft Entra:** App registration in Azure Portal; Three-tier app roles (Admin/Developer/Viewer); Group claims for team sync; Client secret authentication ## Use Cases - **Multi-tenant SaaS platforms:** Isolate tenants with virtual keys, enforce per-tenant budgets, and track usage with required headers. Automatic cost allocation across customers. - **Enterprise team management:** Department-level budgets with team-specific provider access. SSO integration syncs teams from Okta/Entra with automatic role assignment. - **Cost control & optimization:** Hierarchical budgets prevent runaway spending. Weighted routing sends 80% of traffic to cost-effective providers with automatic failover to premium options. - **AI agent security:** MCP tool filtering restricts which tools agents can access. Virtual key permissions ensure agents only call approved models and providers. - **Regulatory compliance:** Required headers enforce audit trails. RBAC controls who can configure guardrails. Comprehensive logs support SOC 2 Type II, HIPAA, GDPR requirements. - **Environment separation:** Bind virtual keys to dev/staging/prod API keys. Developers use test keys with lower budgets while production gets dedicated high-limit keys. ## Configuration Methods - **Web UI:** Visual dashboard for configuring virtual keys, budgets, routing, and RBAC - **REST API:** Programmatic management via endpoints at /api/governance/* - **config.json:** Declarative file-based configuration for GitOps workflows - **Bifrost CLI:** Interactive terminal setup for managing governance settings from your workflow ## Features ### OSS Features - **01 - Model Catalog:** Access 8+ providers and 1000+ AI models through a unified interface. Also supports custom deployed models. - **02 - Budgeting:** Set spending limits and track costs across teams, projects, and models. - **03 - Provider Fallback:** Automatic failover between providers ensures 99.99% uptime for your applications. - **04 - MCP Gateway:** Centralize all MCP tool connections, governance, security, and auth. Your AI can safely use MCP tools with centralized policy enforcement. [MCP Gateway resource](https://www.getmaxim.ai/resources/mcp-gateway.md). - **05 - Virtual Key Management:** Create different virtual keys for different use cases with independent budgets and access control. - **06 - Unified Interface:** One consistent API for all providers. Switch models without changing code. - **07 - Drop-in Replacement:** Replace your existing SDK with just one line change. Compatible with OpenAI, Anthropic, LiteLLM, Google GenAI, LangChain, and more. [Drop-in replacement docs](https://docs.getbifrost.ai/features/drop-in-replacement). - **08 - Built-in Observability:** Out-of-the-box OpenTelemetry support. Built-in dashboard for quick visibility without complex setup. - **09 - Community Support:** Active Discord community with responsive support and regular updates. ### Enterprise Features - **01 - Governance:** SAML support for SSO and role-based access control with policy enforcement for team collaboration. [Governance resource](https://www.getmaxim.ai/resources/governance.md). - **02 - Adaptive Load Balancing:** Automatically optimizes traffic distribution across provider keys and models based on real-time performance metrics. - **03 - Cluster Mode:** High availability deployment with automatic failover and load balancing. Peer-to-peer clustering where every instance is equal. - **04 - Alerts:** Real-time notifications for budget limits, failures, and performance issues on Email, Slack, PagerDuty, Teams, Webhook, and more. - **05 - Log Exports:** Export and analyze request logs, traces, and telemetry data from Bifrost with enterprise-grade data export for compliance, monitoring, and analytics. - **06 - Audit Logs:** Comprehensive logging and audit trails for compliance and debugging. - **07 - Vault Support:** Secure API key management with HashiCorp Vault, AWS Secrets Manager, Google Secret Manager, and Azure Key Vault integration. - **08 - VPC Deployment:** Deploy Bifrost within your private cloud infrastructure with VPC isolation, custom networking, and enhanced security controls. [Enterprise deployment resource](https://www.getmaxim.ai/resources/enterprise-deployment.md). - **09 - Guardrails:** Automatically detect and block unsafe model outputs with real-time policy enforcement and content moderation across all agents. [Guardrails resource](https://www.getmaxim.ai/resources/guardrails.md). ## FAQ ### What is the difference between OSS and Enterprise governance? OSS includes virtual keys, budgets, rate limits, routing, MCP tool filtering, and required headers - sufficient for most production workloads. Enterprise adds RBAC with SSO (Okta/Entra), user-level governance, team synchronization, comprehensive audit logs, and compliance frameworks (SOC 2 Type II, HIPAA, GDPR, ISO 27001). ### How do hierarchical budgets work? Budgets cascade from Customer → Team → Virtual Key → Provider. All applicable budgets must pass for a request to proceed. When a transaction occurs, the same cost deducts from every relevant level simultaneously. A single exhausted budget at any tier blocks the entire request. [Read more about Budgets](https://docs.getbifrost.ai/features/governance/budget-and-limits). ### Can I enforce different provider access for dev vs. production? Yes. Virtual keys support API key restrictions, allowing you to bind VKs to specific provider API keys. Create separate VKs for dev (using test keys with low budgets) and production (using dedicated high-limit keys) for complete environment separation. ### How does weighted routing work with automatic failover? Configure weights per provider (e.g., 80% Azure, 20% OpenAI). Bifrost normalizes weights to sum 1.0 and distributes traffic proportionally. When a provider fails or hits rate limits, Bifrost creates fallback chains ordered by weight for [automatic failover](https://docs.getbifrost.ai/features/fallbacks#automatic-provider-failover). ### What happens when a virtual key exceeds its budget or rate limit? Requests return specific HTTP status codes: 402 for budget exceeded, 429 for rate limits exceeded. Virtual keys remain functional for other operations but block LLM requests until budgets reset (based on configured duration) or rate limit windows expire. ### How does SSO integration handle users in multiple groups? When a user belongs to multiple Okta/Entra groups mapped to different Bifrost roles, the system automatically assigns the highest privilege role. For example, if a user is in both viewer and admin groups, they receive admin permissions. ## Related Resources - [Source: Governance](https://www.getmaxim.ai/resources/governance.md) - [Docs: Bifrost docs](https://docs.getbifrost.ai) - [GitHub: maximhq/bifrost](https://github.com/maximhq/bifrost) - [Pricing: Bifrost pricing](https://www.getmaxim.ai/pricing) - [Enterprise: Bifrost enterprise](https://www.getmaxim.ai/enterprise) - [Book a Demo: Bifrost demo](https://www.getmaxim.ai/book-a-demo) - [Resources: Bifrost resources](https://www.getmaxim.ai/resources.md) --- *This is a markdown version of [https://www.getmaxim.ai/bifrost/resources/governance](https://www.getmaxim.ai/bifrost/resources/governance) for AI/LLM consumption.* --- title: "Bifrost Guardrails | Enterprise AI Safety & Policy Enforcement" description: "Real-time validation of LLM inputs and outputs with PII detection, content moderation, prompt injection defense, and multi-provider guardrail support for enterprise compliance." url: "https://www.getmaxim.ai/bifrost/resources/guardrails" markdown: "https://www.getmaxim.ai/bifrost/resources/guardrails.md" --- # Bifrost Guardrails | Enterprise AI Safety & Policy Enforcement > Real-time validation of LLM inputs and outputs with PII detection, content moderation, prompt injection defense, and multi-provider guardrail support for enterprise compliance. ## Important Links - [View MCP Gateway](https://www.getmaxim.ai/resources/mcp-gateway.md) - [Features](https://www.getmaxim.ai/#features) - [Enterprise](https://www.getmaxim.ai/enterprise) - [Pricing](https://www.getmaxim.ai/pricing) - [Docs](https://docs.getbifrost.ai) - [GitHub](https://github.com/maximhq/bifrost) - [Book a Demo](https://www.getmaxim.ai/book-a-demo) ## Two-Tier Architecture: Rules + Profiles Rules decide what to check and when to check it. Profiles decide how to check it and which provider runs the check. Configure both once and apply them anywhere. - **Rules: When to validate:** Define validation logic using Common Expression Language (CEL). Rules specify whether to check inputs, outputs, or both, and can reference multiple profiles for defense-in-depth. Example: Apply PII detection + content moderation to customer-facing endpoints. - **Profiles: How to validate:** Configure external guardrail providers with credentials, thresholds, and detection parameters. Reusable profiles eliminate redundant configuration across rules. Example: AWS Bedrock profile with high-sensitivity PII detection. ## Guardrail Providers | Name | Logo Domain | Description | Capabilities | Categories | | --- | --- | --- | --- | --- | | AWS Bedrock Guardrails | aws.amazon.com | Comprehensive protection with content filtering, PII detection, and image analysis | 50+ PII entity types detected; Prompt injection detection; Image content analysis; 13 AWS regions supported | Content Filtering; PII Detection; Prompt Defense; Image Safety | | Azure Content Safety | microsoft.com | Severity-based filtering with multi-category detection and custom blocklists | 4-level severity filtering; Prompt Shield for jailbreaks; Protected material detection; Custom blocklists | Hate Speech; Violence; Sexual Content; Self-Harm | | GraySwan Cygnal | grayswan.ai | Natural language rule definition with continuous violation scoring | No-code rule definition; 0-1 violation scoring; Mutation detection; 3 reasoning modes | Custom Policies; Indirect Injection; Fast/Balanced/Thorough | | Patronus AI | patronus.ai | LLM-specific risk detection with hallucination and toxicity screening | Hallucination detection; Context-aware evaluation; Multi-turn analysis; Toxicity screening | Hallucinations; PII; Toxicity; Prompt Injection | ## Detection Capabilities - **50+ PII types - PII leakage prevention:** Detect and redact 50+ types of sensitive information including SSNs, credit cards, addresses, medical records, and device identifiers before they reach or leave the model. - **Multi-category - Content safety filtering:** Block or redact hate speech, sexual content, violence, self-harm, and profanity across inputs and outputs. Severity-based thresholds enable fine-grained control. - **Multi-layered - Prompt injection defense:** Protect against direct and indirect prompt attacks, jailbreaks, and mutation attempts. Multiple providers offer layered defense against evolving attack vectors. - **Context-aware - Hallucination detection:** Identify when models generate factually incorrect or unsupported information. Patronus AI provides context-aware evaluation for high-stakes applications. - **Copyright detection - Protected material screening:** Azure Content Safety detects copyrighted content and protected intellectual property in model outputs, helping organizations avoid legal exposure. - **No-code rules - Custom organizational policies:** Define business-specific rules in natural language (GraySwan) or through configuration. Enforce brand safety, regulatory requirements, and internal compliance policies. ## Setup Steps 1. **01 - Configure guardrail providers.** Set up guardrail provider profiles through the dashboard or API. Configure credentials, detection thresholds, and category filters for each provider. ### Configure guardrail providers ``` # Via dashboard: Guardrails > Providers # Or via config.json { "guardrail_providers": [{ "id": "bedrock-prod", "type": "aws_bedrock", "region": "us-east-1", "guardrail_id": "your-guardrail-id", "version": "DRAFT" }] } ``` 2. **02 - Create validation rules.** Define when and how to validate requests using CEL expressions. Rules can apply to specific routes, models, virtual keys, or user attributes. ### Create validation rules ``` # Via dashboard: Guardrails > Configuration # Or via config.json { "guardrail_rules": [{ "id": "customer-safety", "condition": "request.path.startsWith('/v1/chat')", "input_profiles": ["bedrock-prod"], "output_profiles": ["patronus-ai"], "action": "BLOCK" }] } ``` 3. **03 - Attach to requests.** Apply guardrails via request headers or inline configuration. Bifrost validates inputs before sending to the model and outputs before returning to the client. ### Attach to requests ``` curl https://your-gateway/v1/chat/completions \ -H "x-bf-guardrail-id: customer-safety" \ -H "Authorization: Bearer vk-..." \ -d '{ "model": "gpt-4", "messages": [{"role": "user", "content": "..."}] }' ``` ## Response Types - **Pass (HTTP 200):** Validation succeeded. Request processed normally with detailed guardrail metadata including processing times and rule results. - **Block (HTTP 446):** Violations detected and request blocked. Response includes violation details, severity levels, and affected content excerpts for audit trails. - **Warning (HTTP 246):** Violations detected but content modified (PII redacted) rather than blocked. Includes redaction counts and modification details. ### Pass ``` 200 ``` ### Block ``` 446 ``` ### Warning ``` 246 ``` ## Enterprise Features - **Sampling control:** Apply guardrails to a percentage of requests for performance optimization while maintaining statistical confidence. - **Async processing:** Choose synchronous or asynchronous validation modes. Async reduces latency for non-critical checks. - **Defense-in-depth:** Link multiple provider profiles to single rules for sequential validation and comprehensive protection. - **Comprehensive logging:** Detailed audit trails capture every validation with timestamps, results, and violation details for compliance. - **Timeout configuration:** Set maximum execution duration per rule to prevent guardrail latency from impacting user experience. - **Multi-region support:** Deploy guardrails across 13 AWS regions. Azure Content Safety profiles can target region-specific endpoints for data residency compliance. ## Use Cases - **Healthcare HIPAA compliance:** Prevent PHI leakage in patient-facing chatbots. AWS Bedrock detects medical record numbers, health plan IDs, and clinical notes before they leave the model. - **Financial PII protection:** Block credit card numbers, SSNs, and account details in banking applications. Multi-provider validation ensures no sensitive data escapes detection. - **Prompt injection defense:** Protect against adversarial inputs attempting to override system instructions. GraySwan and Azure Prompt Shield detect mutation attempts and indirect attacks. - **Content moderation for UGC:** Filter user-generated content in social platforms. Severity-based thresholds allow nuanced handling of hate speech, violence, and sexual content. - **Hallucination prevention:** Validate factual accuracy in high-stakes applications like legal research or medical advice. Patronus AI detects unsupported claims and inconsistencies. - **Brand safety enforcement:** Define custom organizational policies in natural language. Ensure model outputs align with brand voice, values, and regulatory requirements. ## Compliance Frameworks Bifrost Guardrails help organizations meet regulatory requirements with automated detection, redaction, and comprehensive audit trails. - **SOC 2 Type II:** Comprehensive audit trails and access controls for guardrail enforcement. - **GDPR:** Personal data protection and right-to-erasure compliance. - **ISO 27001:** Information security management and certification alignment. - **HIPAA:** PHI detection and redaction for healthcare applications. ## Features ### OSS Features - **01 - Model Catalog:** Access 8+ providers and 1000+ AI models through a unified interface. Also supports custom deployed models. - **02 - Budgeting:** Set spending limits and track costs across teams, projects, and models. - **03 - Provider Fallback:** Automatic failover between providers ensures 99.99% uptime for your applications. - **04 - MCP Gateway:** Centralize all MCP tool connections, governance, security, and auth. Your AI can safely use MCP tools with centralized policy enforcement. [MCP Gateway resource](https://www.getmaxim.ai/resources/mcp-gateway.md). - **05 - Virtual Key Management:** Create different virtual keys for different use cases with independent budgets and access control. - **06 - Unified Interface:** One consistent API for all providers. Switch models without changing code. - **07 - Drop-in Replacement:** Replace your existing SDK with just one line change. Compatible with OpenAI, Anthropic, LiteLLM, Google GenAI, LangChain, and more. [Drop-in replacement docs](https://docs.getbifrost.ai/features/drop-in-replacement). - **08 - Built-in Observability:** Out-of-the-box OpenTelemetry support. Built-in dashboard for quick visibility without complex setup. - **09 - Community Support:** Active Discord community with responsive support and regular updates. ### Enterprise Features - **01 - Governance:** SAML support for SSO and role-based access control with policy enforcement for team collaboration. [Governance resource](https://www.getmaxim.ai/resources/governance.md). - **02 - Adaptive Load Balancing:** Automatically optimizes traffic distribution across provider keys and models based on real-time performance metrics. - **03 - Cluster Mode:** High availability deployment with automatic failover and load balancing. Peer-to-peer clustering where every instance is equal. - **04 - Alerts:** Real-time notifications for budget limits, failures, and performance issues on Email, Slack, PagerDuty, Teams, Webhook, and more. - **05 - Log Exports:** Export and analyze request logs, traces, and telemetry data from Bifrost with enterprise-grade data export for compliance, monitoring, and analytics. - **06 - Audit Logs:** Comprehensive logging and audit trails for compliance and debugging. - **07 - Vault Support:** Secure API key management with HashiCorp Vault, AWS Secrets Manager, Google Secret Manager, and Azure Key Vault integration. - **08 - VPC Deployment:** Deploy Bifrost within your private cloud infrastructure with VPC isolation, custom networking, and enhanced security controls. [Enterprise deployment resource](https://www.getmaxim.ai/resources/enterprise-deployment.md). - **09 - Guardrails:** Automatically detect and block unsafe model outputs with real-time policy enforcement and content moderation across all agents. [Guardrails resource](https://www.getmaxim.ai/resources/guardrails.md). ## FAQ ### How do Bifrost Guardrails differ from provider-native safety features? Bifrost aggregates multiple guardrail providers (AWS Bedrock, Azure, GraySwan, Patronus) into a unified interface with cross-provider rules. This enables defense-in-depth, provider failover, and centralized policy management across all LLM providers, not just the model provider you are using. [Read more about Guardrails](https://docs.getbifrost.ai/enterprise/guardrails). ### Do guardrails add latency to LLM requests? Guardrails add 50-500ms depending on provider and validation complexity. Bifrost offers sampling (validate X% of requests), async processing (validate in background), and timeout controls to balance security and performance. ### Can I use multiple guardrail providers for the same request? Yes. Bifrost supports defense-in-depth by linking multiple provider profiles to a single rule. For example, use AWS Bedrock for PII detection + Azure for content moderation + Patronus for hallucination detection on the same request. ### How are custom organizational policies defined? GraySwan Cygnal allows natural language rule definition without code (e.g., "Block financial advice"). AWS Bedrock and Azure support custom blocklists and topic filters. All providers support CEL-based conditional logic for when rules apply. ### Are guardrails required for all requests or can I selectively apply them? Guardrails are optional and selective. Use CEL expressions to apply rules based on request path, model, virtual key, user attributes, or content. For example, only validate customer-facing endpoints while skipping internal testing traffic. ## Related Resources - [Source: Guardrails](https://www.getmaxim.ai/resources/guardrails.md) - [Docs: Bifrost docs](https://docs.getbifrost.ai) - [GitHub: maximhq/bifrost](https://github.com/maximhq/bifrost) - [Pricing: Bifrost pricing](https://www.getmaxim.ai/pricing) - [Enterprise: Bifrost enterprise](https://www.getmaxim.ai/enterprise) - [Book a Demo: Bifrost demo](https://www.getmaxim.ai/book-a-demo) - [Resources: Bifrost resources](https://www.getmaxim.ai/resources.md) --- *This is a markdown version of [https://www.getmaxim.ai/bifrost/resources/guardrails](https://www.getmaxim.ai/bifrost/resources/guardrails) for AI/LLM consumption.* --- title: "Enterprise AI Gateway for Scalability" description: "Scale AI workloads with Bifrost routing, circuit breakers, semantic caching, cost analytics, RBAC, guardrails, in-VPC deployment, and MCP gateway controls for production agents." url: "https://www.getmaxim.ai/bifrost/resources/enterprise-scalability" markdown: "https://www.getmaxim.ai/bifrost/resources/enterprise-scalability.md" --- # Enterprise AI Gateway for Scalability > Scale AI workloads with Bifrost routing, circuit breakers, semantic caching, cost analytics, RBAC, guardrails, in-VPC deployment, and MCP gateway controls for production agents. ## Important Links - [View MCP Gateway](https://www.getmaxim.ai/resources/mcp-gateway.md) - [Features](https://www.getmaxim.ai/#features) - [Enterprise](https://www.getmaxim.ai/enterprise) - [Pricing](https://www.getmaxim.ai/pricing) - [Docs](https://docs.getbifrost.ai) - [GitHub](https://github.com/maximhq/bifrost) - [Book a Demo](https://www.getmaxim.ai/book-a-demo) ## Performance at a Glance - **11µs Mean Latency.** Gateway overhead per request - **5K RPS Throughput.** Sustained requests per second - **99.999% High Availability.** Uptime with automatic failover - **1000+ Models.** Model APIs through one gateway ## What Enterprises Need Moving from prototype to production-grade AI means solving for performance, security, governance, and agent reliability simultaneously. Each enterprise requirement maps to how Bifrost solves it. - **Performance at scale:** Requirement: Low latency, high throughput, and predictable response times at 1,000+ RPS without degradation. Bifrost: 11µs overhead at 5,000 RPS, Go-native with goroutines, pre-spawned worker pools, circuit breaker, adaptive load balancing, and clustering. - **Security and governance:** Requirement: RBAC, SSO, guardrails, PII redaction, audit logs, and compliance frameworks (SOC 2 Type II, HIPAA, GDPR). Bifrost: 3-tier role hierarchy via Okta/Entra SSO, CEL-based guardrail rules with AWS Bedrock/Azure/Patronus, immutable audit trails. - **Cost management and access control:** Requirement: Per-team budgets, rate limiting, virtual keys with independent limits, and real-time cost analytics. Bifrost: Customer → Team → User → Virtual Key budget hierarchy. Per-VK rate limits (token + request), provider-level budgets. - **Production-grade agents:** Requirement: MCP gateway for tool execution, federated auth for internal APIs, agent mode with parallel execution. Bifrost: MCP gateway with federated auth, centralized tool governance, multi-level filtering, code mode (50% fewer tokens), OAuth 2.0 with PKCE for internal APIs. - **Private networking and deployment:** Requirement: In-VPC deployment, vault support for key storage, no data leaving your infrastructure perimeter. Bifrost: Deploy in AWS/GCP/Azure VPC. HashiCorp Vault, AWS Secrets Manager, Google Secret Manager, Azure Key Vault supported. - **Full observability:** Requirement: Native Datadog, Prometheus, OTEL, Splunk integrations. Log exports to S3, Snowflake, BigQuery. Bifrost: Datadog APM + LLM Observability plugin, OTEL to Grafana/New Relic/Honeycomb, log exports to S3/Snowflake/BigQuery/Redshift. ## Problem Cards - **Provider outages kill uptime:** Without automatic failover or model routing, your engineering team scrambles to hardcode a backup manually. Downtime is measured in hours, not seconds. - **No circuit breaker means cascading failures:** A degraded provider responds slowly instead of failing fast. Without a circuit breaker, requests queue behind timeouts, dragging down throughput for healthy providers too. - **Security and governance are bolted on:** PII flows through LLM APIs without redaction. No RBAC, no audit trail, no guardrails. Compliance teams cannot approve production deployment without centralized controls. - **Agents need governance, not just prompts:** As MCP servers multiply, each agent connects to tools independently. No centralized tool policy, no federated auth for internal APIs, no audit trail for tool executions. - **Cost analytics are invisible at scale:** At thousands of requests per second, you lose visibility into which teams, models, and providers drive spend. No unified cost analytics layer across providers. - **Retries and timeouts are fragile:** Hand-rolled retry logic without centralized control means retry storms amplify load during the exact moments your infrastructure is most stressed. ## Security & Governance Bifrost ships with the security controls, access management, and compliance infrastructure platform teams need before rolling out AI tooling organization-wide. - **Guardrails:** Block PII leakage, prompt injection, and policy violations in real time. CEL-based rules with configurable input/output enforcement. Supports AWS Bedrock, Azure Content Safety, GraySwan, Patronus AI, and CEL Rules. [Guardrails docs](https://docs.getbifrost.ai/enterprise/guardrails). - **Role-Based Access Control:** 3-tier role hierarchy (Admin, Developer, Viewer) mapped from your IdP. Custom roles with resource-level permissions. Okta, Entra ID (OIDC), auto-provisioning, and IdP group sync. [Advanced governance docs](https://docs.getbifrost.ai/enterprise/advanced-governance). - **Audit Logs & Compliance:** Immutable, cryptographically verified trails for auth, config changes, and data access. SOC 2 Type II, GDPR, HIPAA, ISO 27001 ready. Splunk, Datadog, Elastic, Webhook, and auto-archival. [Audit logs docs](https://docs.getbifrost.ai/enterprise/audit-logs). - **Vault Support:** Auto-sync API keys from enterprise secret managers with zero-downtime rotation and periodic sync cycles. HashiCorp Vault, AWS SM, Google SM, Azure Key Vault. [Vault support docs](https://docs.getbifrost.ai/enterprise/vault-support). - **In-VPC Deployment:** Deploy entirely within your VPC on AWS, GCP, or Azure. All requests stay in your network. Full private subnet isolation and data residency. [In-VPC deployments docs](https://docs.getbifrost.ai/enterprise/invpc-deployments). - **Enterprise Observability:** Native Datadog APM + LLM Observability, OTEL export to Grafana/New Relic/Honeycomb, and log exports to S3/Snowflake/BigQuery. Datadog, OTEL, Prometheus, S3, Snowflake, BigQuery. [Datadog connector docs](https://docs.getbifrost.ai/enterprise/datadog-connector). ## Agent Capabilities - **MCP Gateway - Centralized tool governance for AI agents:** Bifrost acts as both MCP client and server. It connects to external tool servers and exposes them to agents with centralized policy enforcement. Three-level tool filtering controls agent tool access, and agent mode supports configurable auto-approval for approved tools. - **MCP Federated Auth - Turn internal APIs into MCP tools with zero code:** MCP Federated Auth turns existing private APIs into LLM-ready tools while preserving existing RBAC, tenant isolation, audit trails, and rate limits. Sensitive data stays in the systems that already enforce policy. > **Read the MCP Gateway Deep Dive.** Virtual keys, MCP Tool Groups, Code Mode benchmarks, and how production teams govern tool access while cutting context cost at scale. [Read Full Article](https://www.getmaxim.ai/blog/bifrost-mcp-gateway-access-control-cost-governance-and-92-lower-token-costs-at-scale). ## Performance Capabilities - **Adaptive load balancing across 1000+ models:** Multi-factor scoring weighing error rates (50%), latency (20%), utilization, and momentum. Provider and key selection at two independent levels. - **Auto-reroute on provider degradation:** More than 2% errors marks a provider Degraded. More than 5% marks it Failed with automatic rerouting. Recovery: 90% penalty reduction in 30s. Sequential fallbacks during full outage. - **11µs overhead at 5,000 RPS:** Go-native gateway architecture with goroutines, pre-spawned worker pools, and sync.Pool memory reuse. The published LiteLLM benchmark reports 54x lower P99 latency and 9.5x higher throughput at 500 RPS. - **Dual-layer hash + vector similarity:** Exact hash matching plus semantic similarity (0.0-1.0 threshold). Weaviate, Redis, Qdrant, Pinecone. Sub-ms retrieval with streaming support. - **4-level hierarchical budgets:** Customer to Team to User to Virtual Key hierarchy with independent limits. Rate limiting by tokens and requests. Alerts via Email, Slack, Webhook. - **Per-provider exponential backoff:** Context-based timeouts so slow providers fail fast. Only transient errors trigger retries. Permanent errors fail immediately. ## Comparison Rows | Capability | Diy | Bifrost | | --- | --- | --- | | Model routing | Manual provider switching | Adaptive routing across 1000+ models | | Circuit breaker | Not available | Configurable thresholds (2%/5% error triggers) | | Provider outage | Manual failover, hours of downtime | Automatic fallbacks, <5second rerouting | | Retries & timeout | Hand-rolled, inconsistent | Centralized per-provider exponential backoff | | Low latency | 40ms+ overhead in the compared Python gateway path | 11µs mean overhead at 5,000 RPS; 54x lower P99 in the 500 RPS LiteLLM benchmark | | Throughput | GIL-bound runtimes (Python gateways) | 5,000 RPS sustained (Go native) | | Semantic caching | Not available | Dual-layer with vector similarity | | Cost analytics | Scattered billing dashboards | Unified tracking by team, model, provider | | RBAC & SSO | Build from scratch | Okta/Entra OIDC, 3-tier role hierarchy | | Guardrails | Not available | AWS Bedrock, Azure, Patronus, GraySwan | | Audit logs | Build from scratch | Immutable trails, SIEM export, compliance reports | | MCP gateway | Per-agent tool connections | Centralized governance, 3-level filtering | | Vault support | Manual key management | HashiCorp, AWS, GCP, Azure vaults | | High availability | Single point of failure | Cluster mode with gossip-based sync | | Observability | Multiple integrations needed | Native Datadog, BigQuery, OTEL, Prometheus, log exports | ## Use Cases - **Surviving provider outages without downtime:** Fallback chains across OpenAI, Bedrock, and Vertex. Circuit breaker reroutes traffic in seconds when a provider fails. Full visibility in Datadog. - **Deploying AI in regulated industries:** In-VPC deployment keeps data in your network. Guardrails redact PII before it reaches any model. Audit logs export to Splunk. RBAC controls who accesses production models. - **Scaling to 5,000 requests per second:** Cluster mode on Kubernetes with gossip-based sync. Throughput scales linearly. Semantic caching absorbs repeat queries, multiplying capacity without provider spend. - **Governing AI agents with MCP tools:** 3-level tool filtering controls which agents use which tools. Federated auth exposes internal APIs as MCP tools without code changes, preserving RBAC and tenant isolation. - **Cost governance across 10 engineering teams:** Virtual keys give each team its own budget through the Customer to Team to Virtual Key hierarchy. Cost analytics slice by team, model, and provider. Semantic caching cuts redundant spend. - **Full observability across the AI stack:** Native Datadog plugin sends APM traces and LLM Observability data with session tracking and W3C tracing. Log exports push daily Parquet files. Prometheus alerts on error spikes. ## Setup Steps 1. **01 - Deploy Bifrost.** Run as a standalone binary or Docker container. For high availability, deploy in cluster mode on Kubernetes with gossip-based discovery. In-VPC deployment for regulated environments. ### Deploy Bifrost ``` # Single node docker pull maximhq/bifrost docker run -p 8080:8080 maximhq/bifrost # Or via NPX npx -y @maximhq/bifrost ``` 2. **02 - Configure security and routing.** Add provider keys, set up RBAC via your IdP, enable guardrails, configure fallback chains and circuit breaker thresholds. Connect vault for key storage. All via dashboard or config file. ### Configure security and routing ``` curl -X POST http://localhost:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -H "x-bf-vk: vk-engineering-main" \ -d '{"model": "openai/gpt-4o", "messages": [{"role":"user","content":"Hello"}], "fallbacks": ["anthropic/claude-3-5-sonnet-20241022", "bedrock/anthropic.claude-3-sonnet"]}' ``` 3. **03 - Monitor, govern, and scale.** Connect Datadog/BigQuery or your OTEL stack. Set team budgets. Enable audit log exports. Scale by adding cluster nodes. Monitor everything from the built-in dashboard or your existing tools. ### Monitor, govern, and scale ``` # Built-in dashboard open http://localhost:8080 # Prometheus metrics curl http://localhost:8080/metrics # Datadog, Grafana, New Relic, # Honeycomb via OTEL plugin ``` ## Features ### OSS Features - **01 - Model Catalog:** Access 8+ providers and 1000+ AI models through a unified interface. Also supports custom deployed models. - **02 - Budgeting:** Set spending limits and track costs across teams, projects, and models. - **03 - Provider Fallback:** Automatic failover between providers ensures 99.99% uptime for your applications. - **04 - MCP Gateway:** Centralize all MCP tool connections, governance, security, and auth. Your AI can safely use MCP tools with centralized policy enforcement. [MCP Gateway resource](https://www.getmaxim.ai/resources/mcp-gateway.md). - **05 - Virtual Key Management:** Create different virtual keys for different use cases with independent budgets and access control. - **06 - Unified Interface:** One consistent API for all providers. Switch models without changing code. - **07 - Drop-in Replacement:** Replace your existing SDK with just one line change. Compatible with OpenAI, Anthropic, LiteLLM, Google GenAI, LangChain, and more. [Drop-in replacement docs](https://docs.getbifrost.ai/features/drop-in-replacement). - **08 - Built-in Observability:** Out-of-the-box OpenTelemetry support. Built-in dashboard for quick visibility without complex setup. - **09 - Community Support:** Active Discord community with responsive support and regular updates. ### Enterprise Features - **01 - Governance:** SAML support for SSO and role-based access control with policy enforcement for team collaboration. [Governance resource](https://www.getmaxim.ai/resources/governance.md). - **02 - Adaptive Load Balancing:** Automatically optimizes traffic distribution across provider keys and models based on real-time performance metrics. - **03 - Cluster Mode:** High availability deployment with automatic failover and load balancing. Peer-to-peer clustering where every instance is equal. - **04 - Alerts:** Real-time notifications for budget limits, failures, and performance issues on Email, Slack, PagerDuty, Teams, Webhook, and more. - **05 - Log Exports:** Export and analyze request logs, traces, and telemetry data from Bifrost with enterprise-grade data export for compliance, monitoring, and analytics. - **06 - Audit Logs:** Comprehensive logging and audit trails for compliance and debugging. - **07 - Vault Support:** Secure API key management with HashiCorp Vault, AWS Secrets Manager, Google Secret Manager, and Azure Key Vault integration. - **08 - VPC Deployment:** Deploy Bifrost within your private cloud infrastructure with VPC isolation, custom networking, and enhanced security controls. [Enterprise deployment resource](https://www.getmaxim.ai/resources/enterprise-deployment.md). - **09 - Guardrails:** Automatically detect and block unsafe model outputs with real-time policy enforcement and content moderation across all agents. [Guardrails resource](https://www.getmaxim.ai/resources/guardrails.md). ## FAQ ### How does Bifrost handle provider outages at scale? Bifrost uses a circuit breaker pattern that detects provider degradation within seconds. Keys exceeding 2% error rate are marked Degraded, and above 5% triggers Failed state with automatic rerouting. Sequential fallbacks try each configured backup provider until one succeeds. Recovery is automatic with 90% penalty reduction in 30 seconds. See [Fallbacks documentation](https://docs.getbifrost.ai/features/fallbacks#fallbacks). ### What latency does Bifrost add to LLM requests? Bifrost adds approximately 11µs mean overhead per request at 5,000 RPS on a t3.xlarge instance. This is effectively invisible compared to typical LLM response times of hundreds of milliseconds to several seconds. Built in Go with native goroutines, it achieves 54x faster P99 latency than Python-based gateways. [Read more about Bifrost benchmarks](https://www.getmaxim.ai/bifrost/resources/benchmarks). ### How does the MCP Gateway work for agent tool governance? Bifrost acts as both MCP client and server, connecting to external tool servers and exposing them to agents with centralized policy enforcement. Three-level tool filtering (client config, request-level headers, virtual key policies) controls which agents access which tools. MCP Federated Auth transforms existing internal APIs into MCP tools without code changes. ### Can Bifrost be deployed entirely within our VPC? Yes. Bifrost supports full In-VPC deployment on AWS, GCP, and Azure. All LLM requests stay within your network perimeter. Combined with vault support (HashiCorp Vault, AWS Secrets Manager, Google Secret Manager, Azure Key Vault), no API keys or data leave your infrastructure. See [In-VPC deployments](https://docs.getbifrost.ai/enterprise/invpc-deployments). ## Related Resources - [Source: Enterprise Scalability](https://www.getmaxim.ai/resources/enterprise-scalability.md) - [Docs: Bifrost docs](https://docs.getbifrost.ai) - [GitHub: maximhq/bifrost](https://github.com/maximhq/bifrost) - [Pricing: Bifrost pricing](https://www.getmaxim.ai/pricing) - [Enterprise: Bifrost enterprise](https://www.getmaxim.ai/enterprise) - [Book a Demo: Bifrost demo](https://www.getmaxim.ai/book-a-demo) - [Resources: Bifrost resources](https://www.getmaxim.ai/resources.md) --- *This is a markdown version of [https://www.getmaxim.ai/bifrost/resources/enterprise-scalability](https://www.getmaxim.ai/bifrost/resources/enterprise-scalability) for AI/LLM consumption.* --- title: "Enterprise AI Gateway Deployment | In-VPC, Air-Gapped, Multi-Cloud" description: "Deployment guide for running Bifrost in a VPC, on-premise, air-gapped, or multi-cloud environment with zero data egress and enterprise compliance controls." url: "https://www.getmaxim.ai/bifrost/resources/enterprise-deployment" markdown: "https://www.getmaxim.ai/bifrost/resources/enterprise-deployment.md" --- # Enterprise AI Gateway Deployment | In-VPC, Air-Gapped, Multi-Cloud > Deployment guide for running Bifrost in a VPC, on-premise, air-gapped, or multi-cloud environment with zero data egress and enterprise compliance controls. ## Important Links - [View MCP Gateway](https://www.getmaxim.ai/resources/mcp-gateway.md) - [Features](https://www.getmaxim.ai/#features) - [Enterprise](https://www.getmaxim.ai/enterprise) - [Pricing](https://www.getmaxim.ai/pricing) - [Docs](https://docs.getbifrost.ai) - [GitHub](https://github.com/maximhq/bifrost) - [Book a Demo](https://www.getmaxim.ai/book-a-demo) ## Performance at a Glance - **11µs Gateway Overhead.** Per request at 5K RPS - **99.999% Uptime SLA.** Enterprise In-VPC deployments - **In-VPC Supported.** AWS, GCP, Azure, Cloudflare, Vercel - **0 Bytes Data Egress.** All processing inside your network ## Problem Cards - **No data sovereignty:** SaaS gateways route every prompt, completion, and API key through a third-party network. Sensitive data leaves your perimeter, and compliance teams block adoption before it starts. - **No air-gapped support:** Classified and regulated environments require zero external network access. SaaS gateways cannot operate offline, and most open-source proxies still phone home for updates or telemetry. - **Multi-cloud fragmentation:** When teams deploy a separate gateway in each cloud, policies drift between environments, configuration is duplicated manually, and rate limits or budgets cannot be enforced across providers from a single control plane. - **No production-grade HA:** Single-instance gateways are single points of failure. Without native clustering, automatic failover, or zero-downtime rolling updates, any outage takes your entire AI stack offline. ## Four Deployment Models, One Gateway - **In-VPC - In-VPC / Private Cloud:** Deploy entirely within your VPC on AWS, GCP, or Azure. Complete network isolation with native IAM integration, private endpoints, and no external dependencies. Platforms: AWS EKS/ECS, GCP GKE, Azure AKS. Proof points: 99.999% SLA, multi-zone HA, zero data egress. [In-VPC deployment docs](https://docs.getbifrost.ai/enterprise/invpc-deployments). - **On-Premise - On-Premise / Bare Metal:** Run on your own hardware via Kubernetes, Docker Compose, or a single binary on bare metal VMs. Full control over compute, storage, and networking with no cloud dependency. Platforms: Kubernetes v1.19+, Docker, single binary. Proof points: private registry mirror, credential rotation, Helm + Compose. [On-premise deployment docs](https://docs.getbifrost.ai/deployment-guides/enterprise/on-premise). - **Air-Gapped - Air-Gapped Environments:** For environments with zero internet access. Export the Bifrost image on a connected machine, transfer via tarball, load into your internal registry. No phone-home, no telemetry. Platforms: Docker save/load, internal registry. Proof points: fully offline operation, no telemetry leakage, image mirroring. [Air-gapped deployment docs](https://docs.getbifrost.ai/deployment-guides/enterprise/overview). - **Edge / Dev - Single-Node & Edge:** Run a single Bifrost instance for dev/test, branch offices, or edge deployments. Minimal footprint with instant setup via Docker Compose, fly.io, or a single Go binary. Platforms: Docker Compose, fly.io, Helm. Proof points: 2 vCPU / 4GB minimum, single binary deploy, 30-second setup. [Edge / single-node deployment docs](https://docs.getbifrost.ai/deployment-guides/enterprise/overview). ## Cloud Rows | Cloud | Targets | Auth | Registry | Iac | | --- | --- | --- | --- | --- | | AWS | EKS, ECS | IRSA | Artifact Registry | Terraform, Helm | | GCP | GKE, Cloud Run | Workload Identity | Artifact Registry | Terraform, Helm | | Azure | AKS | Azure WIF | Artifact Registry | Terraform, Helm | | On-Premise | K8s, Docker, Bare Metal | Basic Auth | Internal mirror | Helm, Compose, Single binary | ### One module, any cloud _main.tf_ ```hcl module "bifrost" { source = "github.com/maximhq/bifrost//terraform/modules/bifrost?ref=terraform/v0.1.0" cloud_provider = "aws" # "aws" | "gcp" | "azure" | "kubernetes" service = "eks" # AWS: "ecs" | "eks", GCP: "gke" | "cloud-run", Azure: "aks" region = "us-east-1" image_tag = "latest" } ``` - [Terraform module: maximhq/bifrost](https://github.com/maximhq/bifrost) ## Core Capabilities - **High Availability & Clustering:** Zero-downtime at scale. Peer-to-peer architecture, no single point of failure; 6 discovery methods: K8s, Consul, etcd, DNS, UDP, mDNS; Gossip-based state sync for rate limits and traffic; 3-node minimum; Zero-downtime rolling updates; Adaptive load balancing with health monitoring - **Security & Compliance:** Zero trust architecture, audit ready. Complete VPC isolation, no external dependencies; Vault: HashiCorp, AWS SM, GCP SM, Azure KV; TLS 1.3 in-transit, KMS encryption at-rest; RBAC + OIDC (Okta, Entra ID) + virtual keys; Audit trails for SOC 2 Type II, GDPR, HIPAA, ISO 27001; Container image signing + vulnerability scanning - **Observability:** Full stack monitoring. Built-in dashboard for real-time monitoring; Native Prometheus metrics (scrape or Push Gateway); OpenTelemetry / OTLP distributed tracing; Native Datadog/BigQuery connectors; Log exports to data lakes and storage; Health endpoints: /health, /ready, /cluster/status ## Setup Steps 1. **01 - Choose deployment model.** VPC, on-prem, air-gapped, or multi-cloud. Pick the model that matches your infrastructure and compliance requirements. ### Choose deployment model ``` # In-VPC, On-Premise, Air-Gapped, Multi-Cloud, or Edge ``` 2. **02 - Provision infrastructure.** Use the Terraform module or Helm chart. AWS, GCP, Azure, and generic Kubernetes all supported out of the box. ### Provision infrastructure ``` terraform apply ``` 3. **03 - Configure Bifrost.** Single JSON config file. Connect config store (Postgres or file), set up providers, define virtual keys and routing rules. ### Configure Bifrost ``` # config.json # providers, virtual keys, routing ``` 4. **04 - Deploy.** Single Go binary with minimal dependencies. Helm install, docker compose up, or terraform apply. ### Deploy ``` helm install bifrost maximhq/bifrost ``` 5. **05 - Integrate your SDKs.** Change one line in your existing OpenAI, Anthropic, or LiteLLM SDK. Point at your Bifrost endpoint. ### Integrate your SDKs ``` base_url = "https://bifrost.internal" ``` 6. **06 - Monitor & scale.** Enable clustering for HA, connect Prometheus or OpenTelemetry, set up auto-scaling. Production ready. ### Monitor & scale ``` curl http://bifrost.internal/cluster/status ``` ## Comparison Rows | Capability | SaaS | OSS | Bifrost | | --- | --- | --- | --- | | In-VPC deployment | No | Manual setup | Terraform + Helm | | Air-gapped support | No | No | Docker save/load | | Cloud-native auth | API key only | API key only | IRSA, Workload Identity, Azure WIF | | Vault integration | No | No | HashiCorp, AWS SM, GCP SM, Azure KV | | RBAC + SSO | Limited | No | Okta, Entra ID, OIDC | | Audit logs (SOC 2 Type II, HIPAA) | Limited | No | Immutable, compliance ready | | Data stays in your network | No | Yes | Yes, zero egress | | P99 latency at 500 RPS | ~50ms | ~90.72s | ~1.68s | | Uptime SLA | Varies | None | 99.999% | ## Use Cases - **Regulated Industries - Healthcare, Finance & Government:** Compliance requires all AI traffic to stay within your VPC. No data can transit third-party proxies. In-VPC deployment with complete network isolation, audit logs for HIPAA/SOC 2 Type II/GDPR, Vault for secrets, and zero data egress. All processing stays inside your controlled environment. - **Defense & Air-Gapped - Zero Internet Access Environments:** Your environment has no internet. You need a gateway that runs entirely offline with no phone-home. Docker image export/import workflow, internal registry mirroring, no telemetry leakage. On-prem credential management with offline operation and manual update cycles. - **Multi-Cloud Enterprise - AWS + GCP + Azure, Unified:** Your org runs AWS for production, GCP for ML, Azure for business units. You need one gateway with consistent governance. Clustered deployment across clouds with gossip-based state sync, cloud-native auth per environment (IRSA, Workload Identity, WIF), and unified rate limiting and budgets. - **Gateway Migration - Migrating from LiteLLM or Other Proxies:** You're hitting performance ceilings, missing enterprise features, or struggling with reliability on your current gateway. Drop-in LiteLLM compatibility, 54x faster P99 latency, 68% less memory. Plus enterprise features like RBAC, guardrails, clustering, and audit logs that alternatives don't offer. ## Compliance Badges - **Audited quarterly - SOC 2 Type II:** Compliance signal for audit-ready enterprise deployments. - **EU data residency - GDPR:** Data protection signal for privacy and residency reviews. - **BAA available - HIPAA:** Healthcare compliance signal for protected health information workflows. - **Certified - ISO 27001:** Security management signal for enterprise risk reviews. ## Features ### OSS Features - **01 - Model Catalog:** Access 8+ providers and 1000+ AI models through a unified interface. Also supports custom deployed models. - **02 - Budgeting:** Set spending limits and track costs across teams, projects, and models. - **03 - Provider Fallback:** Automatic failover between providers ensures 99.99% uptime for your applications. - **04 - MCP Gateway:** Centralize all MCP tool connections, governance, security, and auth. Your AI can safely use MCP tools with centralized policy enforcement. [MCP Gateway resource](https://www.getmaxim.ai/resources/mcp-gateway.md). - **05 - Virtual Key Management:** Create different virtual keys for different use cases with independent budgets and access control. - **06 - Unified Interface:** One consistent API for all providers. Switch models without changing code. - **07 - Drop-in Replacement:** Replace your existing SDK with just one line change. Compatible with OpenAI, Anthropic, LiteLLM, Google GenAI, LangChain, and more. [Drop-in replacement docs](https://docs.getbifrost.ai/features/drop-in-replacement). - **08 - Built-in Observability:** Out-of-the-box OpenTelemetry support. Built-in dashboard for quick visibility without complex setup. - **09 - Community Support:** Active Discord community with responsive support and regular updates. ### Enterprise Features - **01 - Governance:** SAML support for SSO and role-based access control with policy enforcement for team collaboration. [Governance resource](https://www.getmaxim.ai/resources/governance.md). - **02 - Adaptive Load Balancing:** Automatically optimizes traffic distribution across provider keys and models based on real-time performance metrics. - **03 - Cluster Mode:** High availability deployment with automatic failover and load balancing. Peer-to-peer clustering where every instance is equal. - **04 - Alerts:** Real-time notifications for budget limits, failures, and performance issues on Email, Slack, PagerDuty, Teams, Webhook, and more. - **05 - Log Exports:** Export and analyze request logs, traces, and telemetry data from Bifrost with enterprise-grade data export for compliance, monitoring, and analytics. - **06 - Audit Logs:** Comprehensive logging and audit trails for compliance and debugging. - **07 - Vault Support:** Secure API key management with HashiCorp Vault, AWS Secrets Manager, Google Secret Manager, and Azure Key Vault integration. - **08 - VPC Deployment:** Deploy Bifrost within your private cloud infrastructure with VPC isolation, custom networking, and enhanced security controls. [Enterprise deployment resource](https://www.getmaxim.ai/resources/enterprise-deployment.md). - **09 - Guardrails:** Automatically detect and block unsafe model outputs with real-time policy enforcement and content moderation across all agents. [Guardrails resource](https://www.getmaxim.ai/resources/guardrails.md). ## FAQ ### Can Bifrost be deployed entirely within our VPC with zero data egress? Yes. Bifrost deploys entirely within your VPC on AWS, GCP, or Azure with complete network isolation. All LLM requests, API keys, prompts, and completions stay within your network perimeter. Combined with vault support, no secrets leave your infrastructure. ### Does Bifrost support air-gapped environments with no internet? Yes. Export the Bifrost Docker image on a connected machine using docker save, transfer the tarball to your air-gapped environment, and load it into your internal registry. No phone-home, no telemetry, fully offline operation. ### What infrastructure-as-code options are available? Bifrost provides a single Terraform module that targets AWS (EKS/ECS), GCP (GKE/Cloud Run), Azure (AKS), and generic Kubernetes. Helm charts are also available for Kubernetes deployments. Docker Compose and single binary are provided for on-premise and bare metal. ### What is the minimum hardware requirement? A single Bifrost node requires 2 vCPU and 4GB RAM minimum. For production HA deployments, a 3-node cluster is recommended. The Go-native binary has a minimal footprint with 68% less memory usage than Python-based alternatives. ## Related Resources - [Source: Enterprise Deployment](https://www.getmaxim.ai/resources/enterprise-deployment.md) - [Docs: Bifrost docs](https://docs.getbifrost.ai) - [GitHub: maximhq/bifrost](https://github.com/maximhq/bifrost) - [Pricing: Bifrost pricing](https://www.getmaxim.ai/pricing) - [Enterprise: Bifrost enterprise](https://www.getmaxim.ai/enterprise) - [Book a Demo: Bifrost demo](https://www.getmaxim.ai/book-a-demo) - [Resources: Bifrost resources](https://www.getmaxim.ai/resources.md) --- *This is a markdown version of [https://www.getmaxim.ai/bifrost/resources/enterprise-deployment](https://www.getmaxim.ai/bifrost/resources/enterprise-deployment) for AI/LLM consumption.* --- title: "AI gateway alternatives" description: "Compare AI gateway platforms for production workloads across performance, governance, MCP support, cost tracking, scalability, security, and migration fit." url: "https://www.getmaxim.ai/bifrost/alternatives" markdown: "https://www.getmaxim.ai/bifrost/alternatives.md" --- # AI gateway alternatives > Compare AI gateway platforms for production workloads across performance, governance, MCP support, cost tracking, scalability, security, and migration fit. ## Important Links - [View MCP Gateway](https://www.getmaxim.ai/resources/mcp-gateway.md) - [Features](https://www.getmaxim.ai/#features) - [Enterprise](https://www.getmaxim.ai/enterprise) - [Pricing](https://www.getmaxim.ai/pricing) - [Docs](https://docs.getbifrost.ai) - [GitHub](https://github.com/maximhq/bifrost) - [Book a Demo](https://www.getmaxim.ai/book-a-demo) ## Resources - [LiteLLM Alternatives](https://www.getmaxim.ai/alternatives/litellm-alternatives.md): Compare LiteLLM alternatives for production AI gateway performance, reliability, governance, observability, MCP support, and migration path. - [Portkey Alternatives](https://www.getmaxim.ai/alternatives/portkey-alternatives.md): Compare Portkey alternatives for multi-provider routing, SDK compatibility, latency overhead, MCP capabilities, deployment model, and enterprise controls. ## AI Gateway Alternative Evaluation _Comparison hub_ The alternatives hub compares AI gateway platforms for production workloads. It focuses on multi-model routing, MCP support, governance, cost tracking, scalability, enterprise-grade security, and production reliability. Detailed LiteLLM and Portkey comparisons cover architecture, performance, migration fit, governance, SDK compatibility, MCP support, and enterprise controls. - [LiteLLM alternatives](https://www.getmaxim.ai/alternatives/litellm-alternatives.md) - [Portkey alternatives](https://www.getmaxim.ai/alternatives/portkey-alternatives.md) - [Buyer guide](https://www.getmaxim.ai/resources/buyers-guide.md) ## Evaluation Criteria _What production teams compare_ - **Performance:** Latency overhead, throughput, success rate under load, memory usage, and runtime architecture. - **Reliability:** Automatic fallbacks, routing rules, load balancing, backpressure, circuit breaking, and health-aware routing. - **Governance:** Virtual keys, budgets, rate limits, RBAC, SSO, audit logs, guardrails, and enterprise deployment controls. - **Agent tooling:** MCP Gateway support, MCP Code Mode, tool hosting, federated authentication, and CLI coding-agent governance. - **Migration fit:** Drop-in SDK compatibility, base URL changes, setup time, self-hosting complexity, and operational dependencies. ## Features ### OSS Features - **01 - Model Catalog:** Access 8+ providers and 1000+ AI models through a unified interface. Also supports custom deployed models. - **02 - Budgeting:** Set spending limits and track costs across teams, projects, and models. - **03 - Provider Fallback:** Automatic failover between providers ensures 99.99% uptime for your applications. - **04 - MCP Gateway:** Centralize all MCP tool connections, governance, security, and auth. Your AI can safely use MCP tools with centralized policy enforcement. [MCP Gateway resource](https://www.getmaxim.ai/resources/mcp-gateway.md). - **05 - Virtual Key Management:** Create different virtual keys for different use cases with independent budgets and access control. - **06 - Unified Interface:** One consistent API for all providers. Switch models without changing code. - **07 - Drop-in Replacement:** Replace your existing SDK with just one line change. Compatible with OpenAI, Anthropic, LiteLLM, Google GenAI, LangChain, and more. [Drop-in replacement docs](https://docs.getbifrost.ai/features/drop-in-replacement). - **08 - Built-in Observability:** Out-of-the-box OpenTelemetry support. Built-in dashboard for quick visibility without complex setup. - **09 - Community Support:** Active Discord community with responsive support and regular updates. ### Enterprise Features - **01 - Governance:** SAML support for SSO and role-based access control with policy enforcement for team collaboration. [Governance resource](https://www.getmaxim.ai/resources/governance.md). - **02 - Adaptive Load Balancing:** Automatically optimizes traffic distribution across provider keys and models based on real-time performance metrics. - **03 - Cluster Mode:** High availability deployment with automatic failover and load balancing. Peer-to-peer clustering where every instance is equal. - **04 - Alerts:** Real-time notifications for budget limits, failures, and performance issues on Email, Slack, PagerDuty, Teams, Webhook, and more. - **05 - Log Exports:** Export and analyze request logs, traces, and telemetry data from Bifrost with enterprise-grade data export for compliance, monitoring, and analytics. - **06 - Audit Logs:** Comprehensive logging and audit trails for compliance and debugging. - **07 - Vault Support:** Secure API key management with HashiCorp Vault, AWS Secrets Manager, Google Secret Manager, and Azure Key Vault integration. - **08 - VPC Deployment:** Deploy Bifrost within your private cloud infrastructure with VPC isolation, custom networking, and enhanced security controls. [Enterprise deployment resource](https://www.getmaxim.ai/resources/enterprise-deployment.md). - **09 - Guardrails:** Automatically detect and block unsafe model outputs with real-time policy enforcement and content moderation across all agents. [Guardrails resource](https://www.getmaxim.ai/resources/guardrails.md). ## Drop-in replacement for compatible AI SDKs Change one line of code to point compatible SDKs at Bifrost. Works with OpenAI, Anthropic, LiteLLM, Google GenAI, LangChain, and Vercel AI SDK. ### OpenAI _openai.py_ ``` import os from openai import OpenAI client = OpenAI( api_key=os.environ.get("OPENAI_API_KEY"), base_url="https:///openai", ) response = client.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": "Hello!"}], ) ``` ### Anthropic _anthropic.py_ ``` import os from anthropic import Anthropic anthropic = Anthropic( api_key=os.environ.get("ANTHROPIC_API_KEY"), base_url="https:///anthropic", ) message = anthropic.messages.create( model="claude-3-5-sonnet-20241022", max_tokens=1024, messages=[{"role": "user", "content": "Hello, Claude"}], ) ``` ### LiteLLM _litellm.py_ ``` import litellm # Set the base URL to your Bifrost deployment litellm.api_base = "https://" response = litellm.completion( model="gpt-4o", messages=[{"role": "user", "content": "Hello!"}], ) ``` ### Google GenAI _genai.py_ ``` import google.generativeai as genai genai.configure( api_key="YOUR_API_KEY", transport="rest", client_options={"api_endpoint": "/google"}, ) model = genai.GenerativeModel("gemini-pro") response = model.generate_content("Hello!") ``` - Point the SDK base URL at your Bifrost deployment. - Keep API keys in your environment or secret manager. - See the docs for provider-specific configuration and deployment steps. - [Gateway setup docs](https://docs.getbifrost.ai/quickstart/gateway/setting-up) - [Drop-in replacement docs](https://docs.getbifrost.ai/features/drop-in-replacement) ## Trust And Deployment - **Open Source:** Bifrost is open source under the Apache 2.0 License. [GitHub](https://github.com/maximhq/bifrost). - **Publisher:** Bifrost is published by H3 Labs Inc. and Maxim AI. - **Compliance:** The site references SOC 2 Type II, GDPR, HIPAA, and ISO 27001 signals. [Enterprise deployment](https://www.getmaxim.ai/resources/enterprise-deployment.md). - **Deployment:** Enterprise resources cover VPC, on-premise, air-gapped, and multi-cloud use. [Enterprise deployment](https://www.getmaxim.ai/resources/enterprise-deployment.md). ## FAQ ### What is Bifrost? Bifrost is the fastest AI and MCP Gateway built for enterprise-grade reliability, governance, and scale. [Docs](https://docs.getbifrost.ai), [GitHub](https://github.com/maximhq/bifrost). ### How is my data protected? Bifrost offers zero-touch in-VPC deployments, so no data ever leaves your environment or passes through Bifrost/Maxim servers. [Governance](https://www.getmaxim.ai/resources/governance.md), [Enterprise deployment](https://www.getmaxim.ai/resources/enterprise-deployment.md). ### Can Bifrost integrate with my existing AI stack? Yes. Bifrost works with major LLM SDKs and frameworks. Compatible SDKs include OpenAI, Anthropic, Mistral, LangChain, LangGraph, and LiteLLM. [Drop-in replacement docs](https://docs.getbifrost.ai/features/drop-in-replacement). ### How much does Bifrost cost? You can get started with the open-source version in seconds for free. Book a demo to learn more about Bifrost Enterprise pricing. [Book a Demo](https://www.getmaxim.ai/book-a-demo). ### How can I get started with Bifrost? You can get started with the open-source version in seconds: npx @maximhq/bifrost [Docs](https://docs.getbifrost.ai), [GitHub](https://github.com/maximhq/bifrost). ## Related Resources - [LiteLLM alternatives](https://www.getmaxim.ai/alternatives/litellm-alternatives.md) - [Portkey alternatives](https://www.getmaxim.ai/alternatives/portkey-alternatives.md) - [Performance benchmarks](https://www.getmaxim.ai/resources/benchmarks.md) - [LLM gateway buyer's guide](https://www.getmaxim.ai/resources/buyers-guide.md) - [Pricing](https://www.getmaxim.ai/pricing) - [Docs](https://docs.getbifrost.ai) - [GitHub](https://github.com/maximhq/bifrost) --- *This is a markdown version of [https://www.getmaxim.ai/bifrost/alternatives](https://www.getmaxim.ai/bifrost/alternatives) for AI/LLM consumption.* --- title: "LiteLLM alternatives" description: "Compare LiteLLM alternatives for scalable enterprise AI gateways, including performance, deployment complexity, governance, observability, MCP support, and migration path." url: "https://www.getmaxim.ai/bifrost/alternatives/litellm-alternatives" markdown: "https://www.getmaxim.ai/bifrost/alternatives/litellm-alternatives.md" --- # LiteLLM alternatives > Compare LiteLLM alternatives for scalable enterprise AI gateways, including performance, deployment complexity, governance, observability, MCP support, and migration path. ## Important Links - [View MCP Gateway](https://www.getmaxim.ai/resources/mcp-gateway.md) - [Features](https://www.getmaxim.ai/#features) - [Enterprise](https://www.getmaxim.ai/enterprise) - [Pricing](https://www.getmaxim.ai/pricing) - [Docs](https://docs.getbifrost.ai) - [GitHub](https://github.com/maximhq/bifrost) - [Book a Demo](https://www.getmaxim.ai/book-a-demo) ## What LiteLLM Is _Gateway overview_ LiteLLM is an open-source, Python-based LLM proxy that provides a unified OpenAI-compatible API for routing requests across multiple LLM providers. LiteLLM is useful for prototyping and multi-provider experimentation. Higher-scale production workloads can require lower overhead, simpler operations, stronger governance, native MCP controls, and built-in guardrails. - [Migrating from LiteLLM](https://www.getmaxim.ai/resources/migrating-from-litellm.md) - [LiteLLM alternative resource](https://www.getmaxim.ai/resources/litellm-alternative.md) ## Bifrost Performance At A Glance _Published comparison metrics_ - **9.5x Faster throughput.** More requests processed per second in the Bifrost vs LiteLLM comparison. [Benchmarks](https://www.getmaxim.ai/resources/benchmarks.md). - **54x Lower P99 latency.** Lower P99 latency in the Bifrost vs LiteLLM comparison. - **68% Less memory.** Lower memory usage in the Bifrost vs LiteLLM comparison. - **40x Less overhead.** Lower gateway processing overhead in the Bifrost vs LiteLLM comparison. ## LiteLLM Strengths - **Unified provider access:** LiteLLM provides a single API for multiple LLM providers with an OpenAI-compatible interface. - **Self-hosted and open source:** LiteLLM can be self-hosted and gives teams control over deployment and data flow. - **Broad provider catalog:** LiteLLM supports a broad set of LLM APIs across major and niche providers. - **Strong community:** LiteLLM is widely used across developer communities and has active open-source adoption. ## LiteLLM Production Challenges - **Python GIL bottleneck:** Python runtime limits can create concurrency bottlenecks under high load. - **Async overhead:** Asyncio context switching and event-loop management can add overhead as request concurrency grows. - **Database dependency:** Production LiteLLM deployments commonly require PostgreSQL and Redis, adding operational complexity. - **Limited enterprise governance:** Teams may need additional engineering for native RBAC, workspaces, audit logs, and granular budget controls. ## Bifrost vs LiteLLM Feature Comparison | Feature | Bifrost | LiteLLM | | --- | --- | --- | | Language | Go | Python | | Gateway overhead per request | 11 microseconds in stress-test context | ~8ms in this comparison | | High-load success rate | 100% at 5K RPS in published stress tests | Degrades above 500 RPS in this comparison | | Memory usage | 68% less in comparison | Baseline | | Adaptive load balancing | Included | Not included | | Health-aware routing | Included | Fallback only | | MCP Code Mode | Included | Not included | | MCP tool hosting | Included | Not included | | Built-in guardrails | Included | Plugin-based | | Semantic cache | Included | Not included | | Native OpenTelemetry | Included | Not included | | Request/response debug | Included | Not included | | Setup time | 30 seconds with NPX or Docker | 5-10 minute setup | | Deployment asset | Single binary, Docker, Kubernetes | Python package, Docker | | Docker size | 80 MB | Greater than 700 MB | ## Migration Path Bifrost is a migration path for teams that need higher throughput, lower overhead, simpler deployment, native observability, governance, MCP support, and guardrails. 1. **01 - Install Bifrost.** Start Bifrost with NPX, Docker, or the Go SDK. ### Install _terminal_ ``` # Option 1: NPX npx -y @maximhq/bifrost # Option 2: Docker docker run -p 8080:8080 maximhq/bifrost # Option 3: Go SDK go get github.com/maximhq/bifrost/core@latest ``` 2. **02 - Configure providers.** Use the web UI to add provider keys, configure models, set fallback chains, and define routing behavior. ### Open dashboard _terminal_ ``` open http://localhost:8080 ``` 3. **03 - Update the endpoint.** Point your compatible SDK or application base URL at Bifrost. The request format can remain OpenAI-compatible. ### Request through Bifrost _curl_ ``` curl http://localhost:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model":"openai/gpt-4o-mini","messages":[{"role":"user","content":"Hello!"}]}' ``` - [Gateway setup docs](https://docs.getbifrost.ai/quickstart/gateway/setting-up) - [Drop-in replacement docs](https://docs.getbifrost.ai/features/drop-in-replacement) - [Migrating from LiteLLM resource](https://www.getmaxim.ai/resources/migrating-from-litellm.md) ## Features ### OSS Features - **01 - Model Catalog:** Access 8+ providers and 1000+ AI models through a unified interface. Also supports custom deployed models. - **02 - Budgeting:** Set spending limits and track costs across teams, projects, and models. - **03 - Provider Fallback:** Automatic failover between providers ensures 99.99% uptime for your applications. - **04 - MCP Gateway:** Centralize all MCP tool connections, governance, security, and auth. Your AI can safely use MCP tools with centralized policy enforcement. [MCP Gateway resource](https://www.getmaxim.ai/resources/mcp-gateway.md). - **05 - Virtual Key Management:** Create different virtual keys for different use cases with independent budgets and access control. - **06 - Unified Interface:** One consistent API for all providers. Switch models without changing code. - **07 - Drop-in Replacement:** Replace your existing SDK with just one line change. Compatible with OpenAI, Anthropic, LiteLLM, Google GenAI, LangChain, and more. [Drop-in replacement docs](https://docs.getbifrost.ai/features/drop-in-replacement). - **08 - Built-in Observability:** Out-of-the-box OpenTelemetry support. Built-in dashboard for quick visibility without complex setup. - **09 - Community Support:** Active Discord community with responsive support and regular updates. ### Enterprise Features - **01 - Governance:** SAML support for SSO and role-based access control with policy enforcement for team collaboration. [Governance resource](https://www.getmaxim.ai/resources/governance.md). - **02 - Adaptive Load Balancing:** Automatically optimizes traffic distribution across provider keys and models based on real-time performance metrics. - **03 - Cluster Mode:** High availability deployment with automatic failover and load balancing. Peer-to-peer clustering where every instance is equal. - **04 - Alerts:** Real-time notifications for budget limits, failures, and performance issues on Email, Slack, PagerDuty, Teams, Webhook, and more. - **05 - Log Exports:** Export and analyze request logs, traces, and telemetry data from Bifrost with enterprise-grade data export for compliance, monitoring, and analytics. - **06 - Audit Logs:** Comprehensive logging and audit trails for compliance and debugging. - **07 - Vault Support:** Secure API key management with HashiCorp Vault, AWS Secrets Manager, Google Secret Manager, and Azure Key Vault integration. - **08 - VPC Deployment:** Deploy Bifrost within your private cloud infrastructure with VPC isolation, custom networking, and enhanced security controls. [Enterprise deployment resource](https://www.getmaxim.ai/resources/enterprise-deployment.md). - **09 - Guardrails:** Automatically detect and block unsafe model outputs with real-time policy enforcement and content moderation across all agents. [Guardrails resource](https://www.getmaxim.ai/resources/guardrails.md). ## Drop-in replacement for compatible AI SDKs Change one line of code to point compatible SDKs at Bifrost. Works with OpenAI, Anthropic, LiteLLM, Google GenAI, LangChain, and Vercel AI SDK. ### OpenAI _openai.py_ ``` import os from openai import OpenAI client = OpenAI( api_key=os.environ.get("OPENAI_API_KEY"), base_url="https:///openai", ) response = client.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": "Hello!"}], ) ``` ### Anthropic _anthropic.py_ ``` import os from anthropic import Anthropic anthropic = Anthropic( api_key=os.environ.get("ANTHROPIC_API_KEY"), base_url="https:///anthropic", ) message = anthropic.messages.create( model="claude-3-5-sonnet-20241022", max_tokens=1024, messages=[{"role": "user", "content": "Hello, Claude"}], ) ``` ### LiteLLM _litellm.py_ ``` import litellm # Set the base URL to your Bifrost deployment litellm.api_base = "https://" response = litellm.completion( model="gpt-4o", messages=[{"role": "user", "content": "Hello!"}], ) ``` ### Google GenAI _genai.py_ ``` import google.generativeai as genai genai.configure( api_key="YOUR_API_KEY", transport="rest", client_options={"api_endpoint": "/google"}, ) model = genai.GenerativeModel("gemini-pro") response = model.generate_content("Hello!") ``` - Point the SDK base URL at your Bifrost deployment. - Keep API keys in your environment or secret manager. - See the docs for provider-specific configuration and deployment steps. - [Gateway setup docs](https://docs.getbifrost.ai/quickstart/gateway/setting-up) - [Drop-in replacement docs](https://docs.getbifrost.ai/features/drop-in-replacement) ## Trust And Deployment - **Open Source:** Bifrost is open source under the Apache 2.0 License. [GitHub](https://github.com/maximhq/bifrost). - **Publisher:** Bifrost is published by H3 Labs Inc. and Maxim AI. - **Compliance:** The site references SOC 2 Type II, GDPR, HIPAA, and ISO 27001 signals. [Enterprise deployment](https://www.getmaxim.ai/resources/enterprise-deployment.md). - **Deployment:** Enterprise resources cover VPC, on-premise, air-gapped, and multi-cloud use. [Enterprise deployment](https://www.getmaxim.ai/resources/enterprise-deployment.md). ## FAQ ### What is Bifrost? Bifrost is the fastest AI and MCP Gateway built for enterprise-grade reliability, governance, and scale. [Docs](https://docs.getbifrost.ai), [GitHub](https://github.com/maximhq/bifrost). ### How is my data protected? Bifrost offers zero-touch in-VPC deployments, so no data ever leaves your environment or passes through Bifrost/Maxim servers. [Governance](https://www.getmaxim.ai/resources/governance.md), [Enterprise deployment](https://www.getmaxim.ai/resources/enterprise-deployment.md). ### Can Bifrost integrate with my existing AI stack? Yes. Bifrost works with major LLM SDKs and frameworks. Compatible SDKs include OpenAI, Anthropic, Mistral, LangChain, LangGraph, and LiteLLM. [Drop-in replacement docs](https://docs.getbifrost.ai/features/drop-in-replacement). ### How much does Bifrost cost? You can get started with the open-source version in seconds for free. Book a demo to learn more about Bifrost Enterprise pricing. [Book a Demo](https://www.getmaxim.ai/book-a-demo). ### How can I get started with Bifrost? You can get started with the open-source version in seconds: npx @maximhq/bifrost [Docs](https://docs.getbifrost.ai), [GitHub](https://github.com/maximhq/bifrost). ## Related Resources - [Alternatives hub](https://www.getmaxim.ai/alternatives.md) - [Performance benchmarks](https://www.getmaxim.ai/resources/benchmarks.md) - [LLM gateway buyer's guide](https://www.getmaxim.ai/resources/buyers-guide.md) - [LiteLLM migration](https://www.getmaxim.ai/resources/migrating-from-litellm.md) - [MCP Gateway](https://www.getmaxim.ai/resources/mcp-gateway.md) - [Governance](https://www.getmaxim.ai/resources/governance.md) - [Guardrails](https://www.getmaxim.ai/resources/guardrails.md) - [Pricing](https://www.getmaxim.ai/pricing) - [Docs](https://docs.getbifrost.ai) - [GitHub](https://github.com/maximhq/bifrost) --- *This is a markdown version of [https://www.getmaxim.ai/bifrost/alternatives/litellm-alternatives](https://www.getmaxim.ai/bifrost/alternatives/litellm-alternatives) for AI/LLM consumption.* --- title: "Portkey alternatives" description: "Compare Portkey alternatives for production AI gateways, including latency overhead, SDK compatibility, pricing complexity, MCP capabilities, deployment model, and enterprise controls." url: "https://www.getmaxim.ai/bifrost/alternatives/portkey-alternatives" markdown: "https://www.getmaxim.ai/bifrost/alternatives/portkey-alternatives.md" --- # Portkey alternatives > Compare Portkey alternatives for production AI gateways, including latency overhead, SDK compatibility, pricing complexity, MCP capabilities, deployment model, and enterprise controls. ## Important Links - [View MCP Gateway](https://www.getmaxim.ai/resources/mcp-gateway.md) - [Features](https://www.getmaxim.ai/#features) - [Enterprise](https://www.getmaxim.ai/enterprise) - [Pricing](https://www.getmaxim.ai/pricing) - [Docs](https://docs.getbifrost.ai) - [GitHub](https://github.com/maximhq/bifrost) - [Book a Demo](https://www.getmaxim.ai/book-a-demo) ## What Portkey Is _Gateway overview_ Portkey is an AI gateway and LLMOps platform for routing requests across 250+ LLM models. It provides managed gateway capabilities with observability, guardrails, and prompt management. The comparison focuses on where production teams may need lower latency overhead, broader SDK drop-in compatibility, simpler private deployment, advanced MCP capabilities, and predictable pricing. ## Bifrost Performance At A Glance - **11 microseconds Mean latency.** Bifrost gateway overhead per request. [Benchmarks](https://www.getmaxim.ai/resources/benchmarks.md). - **5K RPS Throughput.** Sustained request throughput. - **100% Success rate.** Published success rate under the stress-test context. - **<30s Setup time.** NPX or Docker startup path. ## Portkey Strengths - **Broad provider support:** Portkey provides a unified API across 250+ AI models and modalities. - **Built-in observability:** Portkey offers dashboards for latency, token usage, cost analytics, and request-level tracing. - **Enterprise compliance:** Portkey has SOC 2 Type 2, ISO 27001, HIPAA, GDPR, SSO, and SCIM signals. - **Prompt management:** Portkey includes prompt versioning, a testing playground, and collaborative template management. ## Portkey Production Challenges - **Latency overhead at scale:** TypeScript and Node.js runtime overhead can matter for high-throughput workloads. - **SDK integration gaps:** Native drop-in support is centered on the OpenAI SDK, while Anthropic, Google GenAI, AWS Bedrock, and Go SDKs need workarounds. - **Pricing complexity:** Portkey has a free development tier, paid project pricing starting at $49/month per project, and custom enterprise pricing. - **Complex on-premise deployment:** Private deployment can involve multiple services, databases, and ongoing infrastructure maintenance. ## Bifrost vs Portkey Feature Comparison | Feature | Bifrost | Portkey | | --- | --- | --- | | Language | Go | TypeScript / Node.js | | Gateway overhead per request | 11 microseconds | 30-40ms minimum in this comparison | | Automatic fallbacks | Included | Not included | | Adaptive load balancing | Included | Not included | | Backpressure | Included | Not included | | Geo-aware routing | Included | Not included | | MCP server management | Included | Included | | MCP Agent Mode | Included | Not included | | MCP Code Mode | Included | Not included | | MCP tool hosting | Included | Not included | | OpenAI SDK drop-in | Included | Included | | Anthropic SDK drop-in | Included | Not included | | Google GenAI SDK drop-in | Included | Not included | | AWS Bedrock SDK drop-in | Included | Not included | | LiteLLM SDK compatibility | Included | Not included | | Native OpenTelemetry | Included | Not included | | Cluster mode / HA | Included | Not included | | Helm charts | Included | Not included | | Traffic mirroring | Included | Not included | | Mock responses | Included | Not included | ## Migration Path Bifrost is a fit for teams that need lower gateway overhead, broader SDK drop-in compatibility, advanced MCP capabilities, private deployment, and production reliability controls. 1. **01 - Install Bifrost.** Start Bifrost with NPX, Docker, or the Go SDK. ### Install _terminal_ ``` # Option 1: NPX npx -y @maximhq/bifrost # Option 2: Docker docker run -p 8080:8080 maximhq/bifrost # Option 3: Go SDK go get github.com/maximhq/bifrost/core@latest ``` 2. **02 - Configure providers.** Use the web UI to add provider keys, configure models, set fallback chains, and define routing behavior. ### Open dashboard _terminal_ ``` open http://localhost:8080 ``` 3. **03 - Update the endpoint.** Point your compatible SDK or application base URL at Bifrost. The request format can remain OpenAI-compatible. ### Request through Bifrost _curl_ ``` curl http://localhost:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model":"openai/gpt-4o-mini","messages":[{"role":"user","content":"Hello!"}]}' ``` - [Gateway setup docs](https://docs.getbifrost.ai/quickstart/gateway/setting-up) - [Drop-in replacement docs](https://docs.getbifrost.ai/features/drop-in-replacement) - [MCP Gateway resource](https://www.getmaxim.ai/resources/mcp-gateway.md) ## Features ### OSS Features - **01 - Model Catalog:** Access 8+ providers and 1000+ AI models through a unified interface. Also supports custom deployed models. - **02 - Budgeting:** Set spending limits and track costs across teams, projects, and models. - **03 - Provider Fallback:** Automatic failover between providers ensures 99.99% uptime for your applications. - **04 - MCP Gateway:** Centralize all MCP tool connections, governance, security, and auth. Your AI can safely use MCP tools with centralized policy enforcement. [MCP Gateway resource](https://www.getmaxim.ai/resources/mcp-gateway.md). - **05 - Virtual Key Management:** Create different virtual keys for different use cases with independent budgets and access control. - **06 - Unified Interface:** One consistent API for all providers. Switch models without changing code. - **07 - Drop-in Replacement:** Replace your existing SDK with just one line change. Compatible with OpenAI, Anthropic, LiteLLM, Google GenAI, LangChain, and more. [Drop-in replacement docs](https://docs.getbifrost.ai/features/drop-in-replacement). - **08 - Built-in Observability:** Out-of-the-box OpenTelemetry support. Built-in dashboard for quick visibility without complex setup. - **09 - Community Support:** Active Discord community with responsive support and regular updates. ### Enterprise Features - **01 - Governance:** SAML support for SSO and role-based access control with policy enforcement for team collaboration. [Governance resource](https://www.getmaxim.ai/resources/governance.md). - **02 - Adaptive Load Balancing:** Automatically optimizes traffic distribution across provider keys and models based on real-time performance metrics. - **03 - Cluster Mode:** High availability deployment with automatic failover and load balancing. Peer-to-peer clustering where every instance is equal. - **04 - Alerts:** Real-time notifications for budget limits, failures, and performance issues on Email, Slack, PagerDuty, Teams, Webhook, and more. - **05 - Log Exports:** Export and analyze request logs, traces, and telemetry data from Bifrost with enterprise-grade data export for compliance, monitoring, and analytics. - **06 - Audit Logs:** Comprehensive logging and audit trails for compliance and debugging. - **07 - Vault Support:** Secure API key management with HashiCorp Vault, AWS Secrets Manager, Google Secret Manager, and Azure Key Vault integration. - **08 - VPC Deployment:** Deploy Bifrost within your private cloud infrastructure with VPC isolation, custom networking, and enhanced security controls. [Enterprise deployment resource](https://www.getmaxim.ai/resources/enterprise-deployment.md). - **09 - Guardrails:** Automatically detect and block unsafe model outputs with real-time policy enforcement and content moderation across all agents. [Guardrails resource](https://www.getmaxim.ai/resources/guardrails.md). ## Drop-in replacement for compatible AI SDKs Change one line of code to point compatible SDKs at Bifrost. Works with OpenAI, Anthropic, LiteLLM, Google GenAI, LangChain, and Vercel AI SDK. ### OpenAI _openai.py_ ``` import os from openai import OpenAI client = OpenAI( api_key=os.environ.get("OPENAI_API_KEY"), base_url="https:///openai", ) response = client.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": "Hello!"}], ) ``` ### Anthropic _anthropic.py_ ``` import os from anthropic import Anthropic anthropic = Anthropic( api_key=os.environ.get("ANTHROPIC_API_KEY"), base_url="https:///anthropic", ) message = anthropic.messages.create( model="claude-3-5-sonnet-20241022", max_tokens=1024, messages=[{"role": "user", "content": "Hello, Claude"}], ) ``` ### LiteLLM _litellm.py_ ``` import litellm # Set the base URL to your Bifrost deployment litellm.api_base = "https://" response = litellm.completion( model="gpt-4o", messages=[{"role": "user", "content": "Hello!"}], ) ``` ### Google GenAI _genai.py_ ``` import google.generativeai as genai genai.configure( api_key="YOUR_API_KEY", transport="rest", client_options={"api_endpoint": "/google"}, ) model = genai.GenerativeModel("gemini-pro") response = model.generate_content("Hello!") ``` - Point the SDK base URL at your Bifrost deployment. - Keep API keys in your environment or secret manager. - See the docs for provider-specific configuration and deployment steps. - [Gateway setup docs](https://docs.getbifrost.ai/quickstart/gateway/setting-up) - [Drop-in replacement docs](https://docs.getbifrost.ai/features/drop-in-replacement) ## Trust And Deployment - **Open Source:** Bifrost is open source under the Apache 2.0 License. [GitHub](https://github.com/maximhq/bifrost). - **Publisher:** Bifrost is published by H3 Labs Inc. and Maxim AI. - **Compliance:** The site references SOC 2 Type II, GDPR, HIPAA, and ISO 27001 signals. [Enterprise deployment](https://www.getmaxim.ai/resources/enterprise-deployment.md). - **Deployment:** Enterprise resources cover VPC, on-premise, air-gapped, and multi-cloud use. [Enterprise deployment](https://www.getmaxim.ai/resources/enterprise-deployment.md). ## FAQ ### What is Bifrost? Bifrost is the fastest AI and MCP Gateway built for enterprise-grade reliability, governance, and scale. [Docs](https://docs.getbifrost.ai), [GitHub](https://github.com/maximhq/bifrost). ### How is my data protected? Bifrost offers zero-touch in-VPC deployments, so no data ever leaves your environment or passes through Bifrost/Maxim servers. [Governance](https://www.getmaxim.ai/resources/governance.md), [Enterprise deployment](https://www.getmaxim.ai/resources/enterprise-deployment.md). ### Can Bifrost integrate with my existing AI stack? Yes. Bifrost works with major LLM SDKs and frameworks. Compatible SDKs include OpenAI, Anthropic, Mistral, LangChain, LangGraph, and LiteLLM. [Drop-in replacement docs](https://docs.getbifrost.ai/features/drop-in-replacement). ### How much does Bifrost cost? You can get started with the open-source version in seconds for free. Book a demo to learn more about Bifrost Enterprise pricing. [Book a Demo](https://www.getmaxim.ai/book-a-demo). ### How can I get started with Bifrost? You can get started with the open-source version in seconds: npx @maximhq/bifrost [Docs](https://docs.getbifrost.ai), [GitHub](https://github.com/maximhq/bifrost). ## Related Resources - [Alternatives hub](https://www.getmaxim.ai/alternatives.md) - [Performance benchmarks](https://www.getmaxim.ai/resources/benchmarks.md) - [LLM gateway buyer's guide](https://www.getmaxim.ai/resources/buyers-guide.md) - [LiteLLM migration](https://www.getmaxim.ai/resources/migrating-from-litellm.md) - [MCP Gateway](https://www.getmaxim.ai/resources/mcp-gateway.md) - [Governance](https://www.getmaxim.ai/resources/governance.md) - [Guardrails](https://www.getmaxim.ai/resources/guardrails.md) - [Pricing](https://www.getmaxim.ai/pricing) - [Docs](https://docs.getbifrost.ai) - [GitHub](https://github.com/maximhq/bifrost) --- *This is a markdown version of [https://www.getmaxim.ai/bifrost/alternatives/portkey-alternatives](https://www.getmaxim.ai/bifrost/alternatives/portkey-alternatives) for AI/LLM consumption.* --- title: "AI Model Library" description: "Explore AI models across providers. Discover model capabilities, context limits, supported modes, and pricing for chat, image generation, audio, embeddings, and more." url: "https://www.getmaxim.ai/bifrost/model-library" markdown: "https://www.getmaxim.ai/bifrost/model-library.md" --- # AI Model Library > Explore AI models across providers. Discover model capabilities, context limits, supported modes, and pricing for chat, image generation, audio, embeddings, and more. ## Important Links - [View MCP Gateway](https://www.getmaxim.ai/resources/mcp-gateway.md) - [Features](https://www.getmaxim.ai/#features) - [Enterprise](https://www.getmaxim.ai/enterprise) - [Pricing](https://www.getmaxim.ai/pricing) - [Docs](https://docs.getbifrost.ai) - [GitHub](https://github.com/maximhq/bifrost) - [Book a Demo](https://www.getmaxim.ai/book-a-demo) ## Library Overview - **3,827 Models.** - **96 Providers.** - **13 Modes.** ## Browse by Provider Explore all models from a specific provider. - [OpenRouter (484 models)](https://www.getmaxim.ai/model-library/provider/openrouter.md) - [AWS Bedrock (434 models)](https://www.getmaxim.ai/model-library/provider/bedrock.md) - [Fireworks AI (308 models)](https://www.getmaxim.ai/model-library/provider/fireworks_ai.md) - [Azure (293 models)](https://www.getmaxim.ai/model-library/provider/azure.md) - [Hugging Face (209 models)](https://www.getmaxim.ai/model-library/provider/huggingface.md) - [OpenAI (162 models)](https://www.getmaxim.ai/model-library/provider/openai.md) - [Google Vertex AI (159 models)](https://www.getmaxim.ai/model-library/provider/vertex_ai.md) - [Vercel AI Gateway (101 models)](https://www.getmaxim.ai/model-library/provider/vercel_ai_gateway.md) - [Novita (85 models)](https://www.getmaxim.ai/model-library/provider/novita.md) - [Cloudflare (73 models)](https://www.getmaxim.ai/model-library/provider/cloudflare.md) - [Google Gemini (72 models)](https://www.getmaxim.ai/model-library/provider/gemini.md) - [Replicate (71 models)](https://www.getmaxim.ai/model-library/provider/replicate.md) - [Together AI (70 models)](https://www.getmaxim.ai/model-library/provider/together_ai.md) - [Deepinfra (68 models)](https://www.getmaxim.ai/model-library/provider/deepinfra.md) - [Mistral AI (63 models)](https://www.getmaxim.ai/model-library/provider/mistral.md) - [Databricks (61 models)](https://www.getmaxim.ai/model-library/provider/databricks.md) - [Bedrock Mantle (57 models)](https://www.getmaxim.ai/model-library/provider/bedrock_mantle.md) - [Opencode Zen (57 models)](https://www.getmaxim.ai/model-library/provider/opencode-zen.md) - [Oci (44 models)](https://www.getmaxim.ai/model-library/provider/oci.md) - [xAI (44 models)](https://www.getmaxim.ai/model-library/provider/xai.md) - [Workers AI (43 models)](https://www.getmaxim.ai/model-library/provider/workers-ai.md) - [Dashscope (41 models)](https://www.getmaxim.ai/model-library/provider/dashscope.md) - [Snowflake (37 models)](https://www.getmaxim.ai/model-library/provider/snowflake.md) - [Deepgram (36 models)](https://www.getmaxim.ai/model-library/provider/deepgram.md) - [Github Copilot (32 models)](https://www.getmaxim.ai/model-library/provider/github_copilot.md) - [Nebius (30 models)](https://www.getmaxim.ai/model-library/provider/nebius.md) - [Ollama (29 models)](https://www.getmaxim.ai/model-library/provider/ollama.md) - [Watsonx (29 models)](https://www.getmaxim.ai/model-library/provider/watsonx.md) - [Perplexity (28 models)](https://www.getmaxim.ai/model-library/provider/perplexity.md) - [Anthropic (26 models)](https://www.getmaxim.ai/model-library/provider/anthropic.md) - [Moonshot (26 models)](https://www.getmaxim.ai/model-library/provider/moonshot.md) - [Voyage (25 models)](https://www.getmaxim.ai/model-library/provider/voyage.md) - [Stability (23 models)](https://www.getmaxim.ai/model-library/provider/stability.md) - [Cohere (22 models)](https://www.getmaxim.ai/model-library/provider/cohere.md) - [Lambda AI (20 models)](https://www.getmaxim.ai/model-library/provider/lambda_ai.md) - [Groq (19 models)](https://www.getmaxim.ai/model-library/provider/groq.md) - [Sambanova (19 models)](https://www.getmaxim.ai/model-library/provider/sambanova.md) - [Fal AI (17 models)](https://www.getmaxim.ai/model-library/provider/fal_ai.md) - [Gmi (17 models)](https://www.getmaxim.ai/model-library/provider/gmi.md) - [Scaleway (17 models)](https://www.getmaxim.ai/model-library/provider/scaleway.md) - [Hyperbolic (16 models)](https://www.getmaxim.ai/model-library/provider/hyperbolic.md) - [Llamagate (16 models)](https://www.getmaxim.ai/model-library/provider/llamagate.md) - [Wandb (16 models)](https://www.getmaxim.ai/model-library/provider/wandb.md) - [Ovhcloud (15 models)](https://www.getmaxim.ai/model-library/provider/ovhcloud.md) - [Nscale (14 models)](https://www.getmaxim.ai/model-library/provider/nscale.md) - [Opencode Go (14 models)](https://www.getmaxim.ai/model-library/provider/opencode-go.md) - [Aiml (13 models)](https://www.getmaxim.ai/model-library/provider/aiml.md) - [Gradient AI (13 models)](https://www.getmaxim.ai/model-library/provider/gradient_ai.md) - [Zai (13 models)](https://www.getmaxim.ai/model-library/provider/zai.md) - [Ai21 (12 models)](https://www.getmaxim.ai/model-library/provider/ai21.md) - [Anyscale (12 models)](https://www.getmaxim.ai/model-library/provider/anyscale.md) - [Libertai (12 models)](https://www.getmaxim.ai/model-library/provider/libertai.md) - [Runwayml (12 models)](https://www.getmaxim.ai/model-library/provider/runwayml.md) - [Volcengine (12 models)](https://www.getmaxim.ai/model-library/provider/volcengine.md) - [Baseten (11 models)](https://www.getmaxim.ai/model-library/provider/baseten.md) - [Deepseek (11 models)](https://www.getmaxim.ai/model-library/provider/deepseek.md) - [Chatgpt (10 models)](https://www.getmaxim.ai/model-library/provider/chatgpt.md) - [Tensormesh (10 models)](https://www.getmaxim.ai/model-library/provider/tensormesh.md) - [Publicai (9 models)](https://www.getmaxim.ai/model-library/provider/publicai.md) - [Wafer (9 models)](https://www.getmaxim.ai/model-library/provider/wafer.md) - [Black Forest Labs (8 models)](https://www.getmaxim.ai/model-library/provider/black_forest_labs.md) - [Cerebras (8 models)](https://www.getmaxim.ai/model-library/provider/cerebras.md) - [Crusoe (7 models)](https://www.getmaxim.ai/model-library/provider/crusoe.md) - [Gigachat (6 models)](https://www.getmaxim.ai/model-library/provider/gigachat.md) - [Minimax (6 models)](https://www.getmaxim.ai/model-library/provider/minimax.md) - [Palm (6 models)](https://www.getmaxim.ai/model-library/provider/palm.md) - [Pinstripes (6 models)](https://www.getmaxim.ai/model-library/provider/pinstripes.md) - [Sagemaker (6 models)](https://www.getmaxim.ai/model-library/provider/sagemaker.md) - [Text Completion Openai (6 models)](https://www.getmaxim.ai/model-library/provider/text-completion-openai.md) - [Fireworks AI Embedding Models (5 models)](https://www.getmaxim.ai/model-library/provider/fireworks_ai-embedding-models.md) - [Lemonade (5 models)](https://www.getmaxim.ai/model-library/provider/lemonade.md) - [Amazon Nova (4 models)](https://www.getmaxim.ai/model-library/provider/amazon_nova.md) - [Heroku (4 models)](https://www.getmaxim.ai/model-library/provider/heroku.md) - [Meta Llama (4 models)](https://www.getmaxim.ai/model-library/provider/meta_llama.md) - [Azure Text (3 models)](https://www.getmaxim.ai/model-library/provider/azure_text.md) - [Cognition (3 models)](https://www.getmaxim.ai/model-library/provider/cognition.md) - [Meta (3 models)](https://www.getmaxim.ai/model-library/provider/meta.md) - [Nvidia Nim (3 models)](https://www.getmaxim.ai/model-library/provider/nvidia_nim.md) - [V0 (3 models)](https://www.getmaxim.ai/model-library/provider/v0.md) - [Assemblyai (2 models)](https://www.getmaxim.ai/model-library/provider/assemblyai.md) - [Codestral (2 models)](https://www.getmaxim.ai/model-library/provider/codestral.md) - [Darkbloom (2 models)](https://www.getmaxim.ai/model-library/provider/darkbloom.md) - [ElevenLabs (2 models)](https://www.getmaxim.ai/model-library/provider/elevenlabs.md) - [Featherless AI (2 models)](https://www.getmaxim.ai/model-library/provider/featherless_ai.md) - [Friendliai (2 models)](https://www.getmaxim.ai/model-library/provider/friendliai.md) - [Morph (2 models)](https://www.getmaxim.ai/model-library/provider/morph.md) - [Nlp Cloud (2 models)](https://www.getmaxim.ai/model-library/provider/nlp_cloud.md) - [Recraft (2 models)](https://www.getmaxim.ai/model-library/provider/recraft.md) - [Scx AI (2 models)](https://www.getmaxim.ai/model-library/provider/scx-ai.md) - [Soniox (2 models)](https://www.getmaxim.ai/model-library/provider/soniox.md) - [Tencent (2 models)](https://www.getmaxim.ai/model-library/provider/tencent.md) - [Text Completion Codestral (2 models)](https://www.getmaxim.ai/model-library/provider/text-completion-codestral.md) - [Inception (1 models)](https://www.getmaxim.ai/model-library/provider/inception.md) - [Jina AI (1 models)](https://www.getmaxim.ai/model-library/provider/jina_ai.md) - [Sarvam (1 models)](https://www.getmaxim.ai/model-library/provider/sarvam.md) - [Text Completion Inception (1 models)](https://www.getmaxim.ai/model-library/provider/text-completion-inception.md) ## All Models 3,827 models across 96 providers. Showing the first 100; open a provider page for the full list. | Model | Provider | Mode | Max Input Tokens | Input $/1M | | --- | --- | --- | --- | --- | | ai21.j2-mid-v1 | AWS Bedrock | Chat | 8,191 | $12.50 | | ai21.j2-ultra-v1 | AWS Bedrock | Chat | 8,191 | $18.80 | | ai21.jamba-1-5-large-v1:0 | AWS Bedrock | Chat | 256,000 | $2.00 | | ai21.jamba-1-5-mini-v1:0 | AWS Bedrock | Chat | 256,000 | $0.20 | | ai21.jamba-instruct-v1:0 | AWS Bedrock | Chat | 70,000 | $0.50 | | dall-e-2 | Aiml | Image Generation | - | - | | dall-e-3 | Aiml | Image Generation | - | - | | flux-pro | Aiml | Image Generation | - | - | | v1.1 | Aiml | Image Generation | - | - | | v1.1-ultra | Aiml | Image Generation | - | - | | flux-realism | Aiml | Image Generation | - | - | | dev | Aiml | Image Generation | - | - | | text-to-image | Aiml | Image Generation | - | - | | text-to-image | Aiml | Image Generation | - | - | | schnell | Aiml | Image Generation | - | - | | imagen-4.0-ultra-generate-001 | Aiml | Image Generation | - | - | | nano-banana-pro | Aiml | Image Generation | - | - | | gpt-image-2 | Aiml | Image Generation | - | - | | us.amazon.nova-canvas-v1:0 | AWS Bedrock | Image Generation | 2,600 | - | | us.writer.palmyra-x4-v1:0 | AWS Bedrock | Chat | 128,000 | $2.50 | | us.writer.palmyra-x5-v1:0 | AWS Bedrock | Chat | 1,000,000 | $0.60 | | writer.palmyra-x4-v1:0 | AWS Bedrock | Chat | 128,000 | $2.50 | | writer.palmyra-x5-v1:0 | AWS Bedrock | Chat | 1,000,000 | $0.60 | | amazon.nova-lite-v1:0 | AWS Bedrock | Chat | 300,000 | $0.06 | | amazon.nova-2-lite-v1:0 | AWS Bedrock | Chat | 1,000,000 | $0.30 | | amazon.nova-2-pro-preview-20251202-v1:0 | AWS Bedrock | Chat | 1,000,000 | $2.19 | | apac.amazon.nova-2-lite-v1:0 | AWS Bedrock | Chat | 1,000,000 | $0.33 | | apac.amazon.nova-2-pro-preview-20251202-v1:0 | AWS Bedrock | Chat | 1,000,000 | $2.19 | | eu.amazon.nova-2-lite-v1:0 | AWS Bedrock | Chat | 1,000,000 | $0.33 | | eu.amazon.nova-2-pro-preview-20251202-v1:0 | AWS Bedrock | Chat | 1,000,000 | $2.19 | | us.amazon.nova-2-lite-v1:0 | AWS Bedrock | Chat | 1,000,000 | $0.33 | | us.amazon.nova-2-pro-preview-20251202-v1:0 | AWS Bedrock | Chat | 1,000,000 | $2.19 | | amazon.nova-2-multimodal-embeddings-v1:0 | AWS Bedrock | Embedding | 8,172 | $0.14 | | amazon.nova-micro-v1:0 | AWS Bedrock | Chat | 128,000 | $0.04 | | amazon.nova-pro-v1:0 | AWS Bedrock | Chat | 300,000 | $0.80 | | amazon.rerank-v1:0 | AWS Bedrock | Rerank | 32,000 | $0.0000 | | amazon.titan-embed-image-v1 | AWS Bedrock | Embedding | 128 | $0.80 | | amazon.titan-embed-text-v1 | AWS Bedrock | Embedding | 8,192 | $0.10 | | amazon.titan-embed-g1-text-02 | AWS Bedrock | Embedding | 8,192 | $0.10 | | amazon.titan-embed-text-v2:0 | AWS Bedrock | Embedding | 8,192 | $0.02 | | amazon.titan-image-generator-v1 | AWS Bedrock | Image Generation | - | - | | amazon.titan-image-generator-v2 | AWS Bedrock | Image Generation | - | - | | amazon.titan-image-generator-v2:0 | AWS Bedrock | Image Generation | - | - | | twelvelabs.marengo-embed-2-7-v1:0 | AWS Bedrock | Embedding | 77 | $70.00 | | us.twelvelabs.marengo-embed-2-7-v1:0 | AWS Bedrock | Embedding | 77 | $70.00 | | eu.twelvelabs.marengo-embed-2-7-v1:0 | AWS Bedrock | Embedding | 77 | $70.00 | | twelvelabs.pegasus-1-2-v1:0 | AWS Bedrock | Chat | - | - | | us.twelvelabs.pegasus-1-2-v1:0 | AWS Bedrock | Chat | - | - | | eu.twelvelabs.pegasus-1-2-v1:0 | AWS Bedrock | Chat | - | - | | amazon.titan-text-express-v1 | AWS Bedrock | Chat | 42,000 | $1.30 | | amazon.titan-text-lite-v1 | AWS Bedrock | Chat | 42,000 | $0.30 | | amazon.titan-text-premier-v1:0 | AWS Bedrock | Chat | 42,000 | $0.50 | | anthropic.claude-3-5-haiku-20241022-v1:0 | AWS Bedrock | Chat | 200,000 | $0.80 | | anthropic.claude-haiku-4-5-20251001-v1:0 | AWS Bedrock | Chat | 200,000 | $1.00 | | anthropic.claude-haiku-4-5@20251001 | AWS Bedrock | Chat | 200,000 | $1.00 | | anthropic.claude-3-5-sonnet-20240620-v1:0 | AWS Bedrock | Chat | 1,000,000 | $3.00 | | anthropic.claude-3-5-sonnet-20241022-v2:0 | AWS Bedrock | Chat | 1,000,000 | $3.00 | | anthropic.claude-3-7-sonnet-20240620-v1:0 | AWS Bedrock | Chat | 200,000 | $3.60 | | anthropic.claude-3-7-sonnet-20250219-v1:0 | AWS Bedrock | Chat | 200,000 | $3.00 | | anthropic.claude-3-haiku-20240307-v1:0 | AWS Bedrock | Chat | 200,000 | $0.25 | | anthropic.claude-3-opus-20240229-v1:0 | AWS Bedrock | Chat | 200,000 | $15.00 | | anthropic.claude-3-sonnet-20240229-v1:0 | AWS Bedrock | Chat | 200,000 | $3.00 | | anthropic.claude-instant-v1 | AWS Bedrock | Chat | 100,000 | $0.80 | | anthropic.claude-opus-4-1-20250805-v1:0 | AWS Bedrock | Chat | 200,000 | $15.00 | | anthropic.claude-opus-4-20250514-v1:0 | AWS Bedrock | Chat | 200,000 | $15.00 | | anthropic.claude-opus-4-5-20251101-v1:0 | AWS Bedrock | Chat | 200,000 | $5.00 | | anthropic.claude-opus-4-6-v1 | AWS Bedrock | Chat | 1,000,000 | $5.00 | | global.anthropic.claude-opus-4-6-v1 | AWS Bedrock | Chat | 1,000,000 | $5.00 | | us.anthropic.claude-opus-4-6-v1 | AWS Bedrock | Chat | 1,000,000 | $5.50 | | eu.anthropic.claude-opus-4-6-v1 | AWS Bedrock | Chat | 1,000,000 | $5.50 | | au.anthropic.claude-opus-4-6-v1 | AWS Bedrock | Chat | 1,000,000 | $5.50 | | anthropic.claude-opus-4-7 | AWS Bedrock | Chat | 1,000,000 | $5.00 | | anthropic.claude-mythos-preview | AWS Bedrock | Chat | 1,000,000 | $0.0000 | | global.anthropic.claude-opus-4-7 | AWS Bedrock | Chat | 1,000,000 | $5.00 | | us.anthropic.claude-opus-4-7 | AWS Bedrock | Chat | 1,000,000 | $5.50 | | eu.anthropic.claude-opus-4-7 | AWS Bedrock | Chat | 1,000,000 | $5.50 | | au.anthropic.claude-opus-4-7 | AWS Bedrock | Chat | 1,000,000 | $5.50 | | anthropic.claude-fable-5 | AWS Bedrock | Chat | 1,000,000 | $10.00 | | global.anthropic.claude-fable-5 | AWS Bedrock | Chat | 1,000,000 | $10.00 | | us.anthropic.claude-fable-5 | AWS Bedrock | Chat | 1,000,000 | $11.00 | | eu.anthropic.claude-fable-5 | AWS Bedrock | Chat | 1,000,000 | $11.00 | | anthropic.claude-opus-5 | AWS Bedrock | Chat | 1,000,000 | $5.00 | | global.anthropic.claude-opus-5 | AWS Bedrock | Chat | 1,000,000 | $5.00 | | us.anthropic.claude-opus-5 | AWS Bedrock | Chat | 1,000,000 | $5.50 | | eu.anthropic.claude-opus-5 | AWS Bedrock | Chat | 1,000,000 | $5.50 | | au.anthropic.claude-opus-5 | AWS Bedrock | Chat | 1,000,000 | $5.50 | | jp.anthropic.claude-opus-5 | AWS Bedrock | Chat | 1,000,000 | $5.50 | | anthropic.claude-opus-4-8 | AWS Bedrock | Chat | 1,000,000 | $5.00 | | global.anthropic.claude-opus-4-8 | AWS Bedrock | Chat | 1,000,000 | $5.00 | | us.anthropic.claude-opus-4-8 | AWS Bedrock | Chat | 1,000,000 | $5.50 | | eu.anthropic.claude-opus-4-8 | AWS Bedrock | Chat | 1,000,000 | $5.50 | | au.anthropic.claude-opus-4-8 | AWS Bedrock | Chat | 1,000,000 | $5.50 | | jp.anthropic.claude-opus-4-8 | AWS Bedrock | Chat | 1,000,000 | $5.50 | | jp.anthropic.claude-opus-4-7 | AWS Bedrock | Chat | 1,000,000 | $5.50 | | anthropic.claude-sonnet-5 | AWS Bedrock | Chat | 1,000,000 | $2.00 | | global.anthropic.claude-sonnet-5 | AWS Bedrock | Chat | 1,000,000 | $2.00 | | us.anthropic.claude-sonnet-5 | AWS Bedrock | Chat | 1,000,000 | $2.20 | | eu.anthropic.claude-sonnet-5 | AWS Bedrock | Chat | 1,000,000 | $2.20 | | au.anthropic.claude-sonnet-5 | AWS Bedrock | Chat | 1,000,000 | $2.20 | | jp.anthropic.claude-sonnet-5 | AWS Bedrock | Chat | 1,000,000 | $2.20 | ## FAQ ### What is the AI model library? The model library is a browsable catalog of AI models across every major provider, showing capabilities, context limits, supported modes, and pricing in one place. Use it to discover and compare models for chat, image generation, audio, embeddings, and more. ### How do I compare two models? Open any model to see its full specification, then add a second model to view a side-by-side comparison of context windows, capabilities, and pricing. This makes it easy to pick the right model for a given workload. ### What details does each model page show? Each model page lists the mode, maximum input and output tokens, supported capabilities (function calling, vision, reasoning, audio, and more), and input/output pricing per 1M tokens, alongside comparable alternatives from other providers. ### How current is the model data? Model specifications and pricing are regenerated from the upstream datasheet on every build, so they reflect the latest published capabilities and rates for hundreds of models. ### How does Bifrost help me use multiple models? Bifrost AI Gateway gives you a single, unified API across every provider in this library, with automatic failover, load balancing, and cost-based routing so you can switch or combine models without rewriting your application. ## Related Resources - [Model library](https://www.getmaxim.ai/model-library.md) - [LLM cost calculator](https://www.getmaxim.ai/llm-cost-calculator.md) - [Docs: Bifrost docs](https://docs.getbifrost.ai) - [GitHub: maximhq/bifrost](https://github.com/maximhq/bifrost) - [Pricing: Bifrost pricing](https://www.getmaxim.ai/pricing) - [Enterprise: Bifrost enterprise](https://www.getmaxim.ai/enterprise) - [Book a Demo: Bifrost demo](https://www.getmaxim.ai/book-a-demo) - [Resources: Bifrost resources](https://www.getmaxim.ai/resources.md) --- *This is a markdown version of [https://www.getmaxim.ai/bifrost/model-library](https://www.getmaxim.ai/bifrost/model-library) for AI/LLM consumption.* --- title: "LLM Cost Calculator" description: "Calculate and compare LLM API costs across hundreds of AI models and dozens of providers. See input and output pricing per 1M tokens for OpenAI, Anthropic, Google, and more." url: "https://www.getmaxim.ai/bifrost/llm-cost-calculator" markdown: "https://www.getmaxim.ai/bifrost/llm-cost-calculator.md" --- # LLM Cost Calculator > Calculate and compare LLM API costs across hundreds of AI models and dozens of providers. See input and output pricing per 1M tokens for OpenAI, Anthropic, Google, and more. ## Important Links - [View MCP Gateway](https://www.getmaxim.ai/resources/mcp-gateway.md) - [Features](https://www.getmaxim.ai/#features) - [Enterprise](https://www.getmaxim.ai/enterprise) - [Pricing](https://www.getmaxim.ai/pricing) - [Docs](https://docs.getbifrost.ai) - [GitHub](https://github.com/maximhq/bifrost) - [Book a Demo](https://www.getmaxim.ai/book-a-demo) ## Calculator Overview - **3,302 Models.** - **73 Providers.** - **11 Modes.** ## Browse by Provider Jump to every model and price for a specific provider. - [OpenRouter (464 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/openrouter.md) - [AWS Bedrock (377 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/bedrock.md) - [Fireworks AI (293 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/fireworks_ai.md) - [Azure (281 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/azure.md) - [Hugging Face (209 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/huggingface.md) - [OpenAI (150 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/openai.md) - [Vertex AI (133 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/vertex_ai.md) - [Vercel Ai Gateway (101 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/vercel_ai_gateway.md) - [Novita (85 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/novita.md) - [DeepInfra (68 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/deepinfra.md) - [Cloudflare (62 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/cloudflare.md) - [Databricks (61 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/databricks.md) - [Together AI (61 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/together_ai.md) - [Bedrock Mantle (57 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/bedrock_mantle.md) - [Mistral (57 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/mistral.md) - [Google Gemini (54 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/gemini.md) - [Opencode-zen (51 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/opencode-zen.md) - [Oci (44 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/oci.md) - [xAI (44 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/xai.md) - [Replicate (40 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/replicate.md) - [Workers-ai (36 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/workers-ai.md) - [Nebius (30 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/nebius.md) - [IBM watsonx (28 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/watsonx.md) - [Anthropic (26 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/anthropic.md) - [Moonshot (26 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/moonshot.md) - [Perplexity (26 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/perplexity.md) - [Dashscope (25 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/dashscope.md) - [Voyage (25 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/voyage.md) - [Snowflake (21 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/snowflake.md) - [Lambda Ai (20 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/lambda_ai.md) - [SambaNova (19 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/sambanova.md) - [Cohere (17 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/cohere.md) - [Gmi (17 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/gmi.md) - [Hyperbolic (16 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/hyperbolic.md) - [Llamagate (16 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/llamagate.md) - [Scaleway (16 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/scaleway.md) - [Wandb (16 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/wandb.md) - [Ovhcloud (15 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/ovhcloud.md) - [Groq (14 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/groq.md) - [Nscale (14 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/nscale.md) - [Opencode-go (14 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/opencode-go.md) - [AI21 (12 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/ai21.md) - [Anyscale (12 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/anyscale.md) - [Libertai (12 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/libertai.md) - [Baseten (11 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/baseten.md) - [DeepSeek (11 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/deepseek.md) - [Zai (11 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/zai.md) - [Gradient Ai (10 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/gradient_ai.md) - [Tensormesh (10 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/tensormesh.md) - [Cerebras (8 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/cerebras.md) - [Wafer (8 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/wafer.md) - [Crusoe (7 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/crusoe.md) - [Minimax (6 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/minimax.md) - [PaLM (6 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/palm.md) - [Pinstripes (6 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/pinstripes.md) - [Text-completion-openai (6 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/text-completion-openai.md) - [Amazon Nova (4 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/amazon_nova.md) - [Azure Text (3 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/azure_text.md) - [Cognition (3 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/cognition.md) - [Meta (3 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/meta.md) - [V0 (3 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/v0.md) - [Darkbloom (2 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/darkbloom.md) - [Fireworks Ai-embedding-models (2 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/fireworks_ai-embedding-models.md) - [FriendliAI (2 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/friendliai.md) - [Github Copilot (2 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/github_copilot.md) - [Morph (2 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/morph.md) - [NLP Cloud (2 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/nlp_cloud.md) - [Scx-ai (2 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/scx-ai.md) - [Tencent (2 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/tencent.md) - [Volcengine (2 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/volcengine.md) - [Inception (1 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/inception.md) - [Jina AI (1 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/jina_ai.md) - [Text-completion-inception (1 models)](https://www.getmaxim.ai/llm-cost-calculator/provider/text-completion-inception.md) ## All Models Pricing for 3,302 models across 73 providers. Showing the first 100; open a provider page for the full list. | Model | Provider | Mode | Input $/1M | Output $/1M | | --- | --- | --- | --- | --- | | ai21.j2-mid-v1 | AWS Bedrock | Chat | $12.50 | $12.50 | | ai21.j2-ultra-v1 | AWS Bedrock | Chat | $18.80 | $18.80 | | ai21.jamba-1-5-large-v1:0 | AWS Bedrock | Chat | $2.00 | $8.00 | | ai21.jamba-1-5-mini-v1:0 | AWS Bedrock | Chat | $0.20 | $0.40 | | ai21.jamba-instruct-v1:0 | AWS Bedrock | Chat | $0.50 | $0.70 | | us.writer.palmyra-x4-v1:0 | AWS Bedrock | Chat | $2.50 | $10.00 | | us.writer.palmyra-x5-v1:0 | AWS Bedrock | Chat | $0.60 | $6.00 | | writer.palmyra-x4-v1:0 | AWS Bedrock | Chat | $2.50 | $10.00 | | writer.palmyra-x5-v1:0 | AWS Bedrock | Chat | $0.60 | $6.00 | | amazon.nova-lite-v1:0 | AWS Bedrock | Chat | $0.06 | $0.24 | | amazon.nova-2-lite-v1:0 | AWS Bedrock | Chat | $0.30 | $2.50 | | amazon.nova-2-pro-preview-20251202-v1:0 | AWS Bedrock | Chat | $2.19 | $17.50 | | apac.amazon.nova-2-lite-v1:0 | AWS Bedrock | Chat | $0.33 | $2.75 | | apac.amazon.nova-2-pro-preview-20251202-v1:0 | AWS Bedrock | Chat | $2.19 | $17.50 | | eu.amazon.nova-2-lite-v1:0 | AWS Bedrock | Chat | $0.33 | $2.75 | | eu.amazon.nova-2-pro-preview-20251202-v1:0 | AWS Bedrock | Chat | $2.19 | $17.50 | | us.amazon.nova-2-lite-v1:0 | AWS Bedrock | Chat | $0.33 | $2.75 | | us.amazon.nova-2-pro-preview-20251202-v1:0 | AWS Bedrock | Chat | $2.19 | $17.50 | | amazon.nova-2-multimodal-embeddings-v1:0 | AWS Bedrock | Embedding | $0.14 | $0.0000 | | amazon.nova-micro-v1:0 | AWS Bedrock | Chat | $0.04 | $0.14 | | amazon.nova-pro-v1:0 | AWS Bedrock | Chat | $0.80 | $3.20 | | amazon.titan-embed-image-v1 | AWS Bedrock | Embedding | $0.80 | $0.0000 | | amazon.titan-embed-text-v1 | AWS Bedrock | Embedding | $0.10 | $0.0000 | | amazon.titan-embed-g1-text-02 | AWS Bedrock | Embedding | $0.10 | $0.0000 | | amazon.titan-embed-text-v2:0 | AWS Bedrock | Embedding | $0.02 | $0.0000 | | twelvelabs.marengo-embed-2-7-v1:0 | AWS Bedrock | Embedding | $70.00 | $0.0000 | | us.twelvelabs.marengo-embed-2-7-v1:0 | AWS Bedrock | Embedding | $70.00 | $0.0000 | | eu.twelvelabs.marengo-embed-2-7-v1:0 | AWS Bedrock | Embedding | $70.00 | $0.0000 | | twelvelabs.pegasus-1-2-v1:0 | AWS Bedrock | Chat | - | $7.50 | | us.twelvelabs.pegasus-1-2-v1:0 | AWS Bedrock | Chat | - | $7.50 | | eu.twelvelabs.pegasus-1-2-v1:0 | AWS Bedrock | Chat | - | $7.50 | | amazon.titan-text-express-v1 | AWS Bedrock | Chat | $1.30 | $1.70 | | amazon.titan-text-lite-v1 | AWS Bedrock | Chat | $0.30 | $0.40 | | amazon.titan-text-premier-v1:0 | AWS Bedrock | Chat | $0.50 | $1.50 | | anthropic.claude-3-5-haiku-20241022-v1:0 | AWS Bedrock | Chat | $0.80 | $4.00 | | anthropic.claude-haiku-4-5-20251001-v1:0 | AWS Bedrock | Chat | $1.00 | $5.00 | | anthropic.claude-haiku-4-5@20251001 | AWS Bedrock | Chat | $1.00 | $5.00 | | anthropic.claude-3-5-sonnet-20240620-v1:0 | AWS Bedrock | Chat | $3.00 | $15.00 | | anthropic.claude-3-5-sonnet-20241022-v2:0 | AWS Bedrock | Chat | $3.00 | $15.00 | | anthropic.claude-3-7-sonnet-20240620-v1:0 | AWS Bedrock | Chat | $3.60 | $18.00 | | anthropic.claude-3-7-sonnet-20250219-v1:0 | AWS Bedrock | Chat | $3.00 | $15.00 | | anthropic.claude-3-haiku-20240307-v1:0 | AWS Bedrock | Chat | $0.25 | $1.25 | | anthropic.claude-3-opus-20240229-v1:0 | AWS Bedrock | Chat | $15.00 | $75.00 | | anthropic.claude-3-sonnet-20240229-v1:0 | AWS Bedrock | Chat | $3.00 | $15.00 | | anthropic.claude-instant-v1 | AWS Bedrock | Chat | $0.80 | $2.40 | | anthropic.claude-opus-4-1-20250805-v1:0 | AWS Bedrock | Chat | $15.00 | $75.00 | | anthropic.claude-opus-4-20250514-v1:0 | AWS Bedrock | Chat | $15.00 | $75.00 | | anthropic.claude-opus-4-5-20251101-v1:0 | AWS Bedrock | Chat | $5.00 | $25.00 | | anthropic.claude-opus-4-6-v1 | AWS Bedrock | Chat | $5.00 | $25.00 | | global.anthropic.claude-opus-4-6-v1 | AWS Bedrock | Chat | $5.00 | $25.00 | | us.anthropic.claude-opus-4-6-v1 | AWS Bedrock | Chat | $5.50 | $27.50 | | eu.anthropic.claude-opus-4-6-v1 | AWS Bedrock | Chat | $5.50 | $27.50 | | au.anthropic.claude-opus-4-6-v1 | AWS Bedrock | Chat | $5.50 | $27.50 | | anthropic.claude-opus-4-7 | AWS Bedrock | Chat | $5.00 | $25.00 | | global.anthropic.claude-opus-4-7 | AWS Bedrock | Chat | $5.00 | $25.00 | | us.anthropic.claude-opus-4-7 | AWS Bedrock | Chat | $5.50 | $27.50 | | eu.anthropic.claude-opus-4-7 | AWS Bedrock | Chat | $5.50 | $27.50 | | au.anthropic.claude-opus-4-7 | AWS Bedrock | Chat | $5.50 | $27.50 | | anthropic.claude-fable-5 | AWS Bedrock | Chat | $10.00 | $50.00 | | global.anthropic.claude-fable-5 | AWS Bedrock | Chat | $10.00 | $50.00 | | us.anthropic.claude-fable-5 | AWS Bedrock | Chat | $11.00 | $55.00 | | eu.anthropic.claude-fable-5 | AWS Bedrock | Chat | $11.00 | $55.00 | | anthropic.claude-opus-5 | AWS Bedrock | Chat | $5.00 | $25.00 | | global.anthropic.claude-opus-5 | AWS Bedrock | Chat | $5.00 | $25.00 | | us.anthropic.claude-opus-5 | AWS Bedrock | Chat | $5.50 | $27.50 | | eu.anthropic.claude-opus-5 | AWS Bedrock | Chat | $5.50 | $27.50 | | au.anthropic.claude-opus-5 | AWS Bedrock | Chat | $5.50 | $27.50 | | jp.anthropic.claude-opus-5 | AWS Bedrock | Chat | $5.50 | $27.50 | | anthropic.claude-opus-4-8 | AWS Bedrock | Chat | $5.00 | $25.00 | | global.anthropic.claude-opus-4-8 | AWS Bedrock | Chat | $5.00 | $25.00 | | us.anthropic.claude-opus-4-8 | AWS Bedrock | Chat | $5.50 | $27.50 | | eu.anthropic.claude-opus-4-8 | AWS Bedrock | Chat | $5.50 | $27.50 | | au.anthropic.claude-opus-4-8 | AWS Bedrock | Chat | $5.50 | $27.50 | | jp.anthropic.claude-opus-4-8 | AWS Bedrock | Chat | $5.50 | $27.50 | | jp.anthropic.claude-opus-4-7 | AWS Bedrock | Chat | $5.50 | $27.50 | | anthropic.claude-sonnet-5 | AWS Bedrock | Chat | $2.00 | $10.00 | | global.anthropic.claude-sonnet-5 | AWS Bedrock | Chat | $2.00 | $10.00 | | us.anthropic.claude-sonnet-5 | AWS Bedrock | Chat | $2.20 | $11.00 | | eu.anthropic.claude-sonnet-5 | AWS Bedrock | Chat | $2.20 | $11.00 | | au.anthropic.claude-sonnet-5 | AWS Bedrock | Chat | $2.20 | $11.00 | | jp.anthropic.claude-sonnet-5 | AWS Bedrock | Chat | $2.20 | $11.00 | | anthropic.claude-sonnet-4-6 | AWS Bedrock | Chat | $3.00 | $15.00 | | global.anthropic.claude-sonnet-4-6 | AWS Bedrock | Chat | $3.00 | $15.00 | | us.anthropic.claude-sonnet-4-6 | AWS Bedrock | Chat | $3.30 | $16.50 | | eu.anthropic.claude-sonnet-4-6 | AWS Bedrock | Chat | $3.30 | $16.50 | | au.anthropic.claude-sonnet-4-6 | AWS Bedrock | Chat | $3.30 | $16.50 | | jp.anthropic.claude-sonnet-4-6 | AWS Bedrock | Chat | $3.30 | $16.50 | | anthropic.claude-sonnet-4-20250514-v1:0 | AWS Bedrock | Chat | $3.00 | $15.00 | | anthropic.claude-sonnet-4-5-20250929-v1:0 | AWS Bedrock | Chat | $3.00 | $15.00 | | anthropic.claude-v1 | AWS Bedrock | Chat | $8.00 | $24.00 | | anthropic.claude-v2:1 | AWS Bedrock | Chat | $8.00 | $24.00 | | zephyr-7b-beta | Anyscale | Chat | $0.15 | $0.15 | | CodeLlama-34b-Instruct-hf | Anyscale | Chat | $1.00 | $1.00 | | CodeLlama-70b-Instruct-hf | Anyscale | Chat | $1.00 | $1.00 | | gemma-7b-it | Anyscale | Chat | $0.15 | $0.15 | | Llama-2-13b-chat-hf | Anyscale | Chat | $0.25 | $0.25 | | Llama-2-70b-chat-hf | Anyscale | Chat | $1.00 | $1.00 | | Llama-2-7b-chat-hf | Anyscale | Chat | $0.15 | $0.15 | | Meta-Llama-3-70B-Instruct | Anyscale | Chat | $1.00 | $1.00 | | Meta-Llama-3-8B-Instruct | Anyscale | Chat | $0.15 | $0.15 | ## FAQ ### How does the LLM cost calculator work? Enter your expected input and output token volumes (or images, audio seconds, or pages depending on the model mode) and the calculator multiplies them by each model’s published per-token pricing to estimate your total cost. Pricing is sourced from each provider’s official rate card and normalized to a per-1M-token basis for easy comparison. ### Is the pricing data up to date? Model pricing is regenerated from the upstream datasheet on every build, so the figures reflect the latest published rates for hundreds of models across dozens of providers. Always confirm against the provider’s own pricing page before committing to production spend. ### How can I compare costs across providers? Use the provider pages to see every model from a given provider with input and output pricing side by side, or open an individual model page to view alternatives ranked by price. Bifrost AI Gateway can then route traffic to the most cost-effective provider automatically. ### Why do input and output tokens cost different amounts? Most providers charge more for output (generated) tokens than input (prompt) tokens because generation is more compute-intensive. For generation-heavy workloads, output pricing dominates your bill; for retrieval or long-context workloads, input pricing matters more. ### How can I reduce my LLM API costs? Route each request to the cheapest model that meets your quality bar, cache repeated prompts, and fail over to alternative providers during outages or price spikes. Bifrost AI Gateway provides this multi-provider routing, load balancing, and caching out of the box. ## Related Resources - [LLM cost calculator](https://www.getmaxim.ai/llm-cost-calculator.md) - [Model library](https://www.getmaxim.ai/model-library.md) - [Docs: Bifrost docs](https://docs.getbifrost.ai) - [GitHub: maximhq/bifrost](https://github.com/maximhq/bifrost) - [Pricing: Bifrost pricing](https://www.getmaxim.ai/pricing) - [Enterprise: Bifrost enterprise](https://www.getmaxim.ai/enterprise) - [Book a Demo: Bifrost demo](https://www.getmaxim.ai/book-a-demo) - [Resources: Bifrost resources](https://www.getmaxim.ai/resources.md) --- *This is a markdown version of [https://www.getmaxim.ai/bifrost/llm-cost-calculator](https://www.getmaxim.ai/bifrost/llm-cost-calculator) for AI/LLM consumption.* --- title: "LLM Provider Status" description: "Real-time status monitoring for AI model providers. Check if ChatGPT, Claude, Groq, Azure OpenAI, and other LLM APIs are operational. Data comes straight from each provider’s official status source." url: "https://www.getmaxim.ai/bifrost/provider-status" markdown: "https://www.getmaxim.ai/bifrost/provider-status.md" --- # LLM Provider Status > Real-time status monitoring for AI model providers. Check if ChatGPT, Claude, Groq, Azure OpenAI, and other LLM APIs are operational. Data comes straight from each provider’s official status source. ## Important Links - [View MCP Gateway](https://www.getmaxim.ai/resources/mcp-gateway.md) - [Features](https://www.getmaxim.ai/#features) - [Enterprise](https://www.getmaxim.ai/enterprise) - [Pricing](https://www.getmaxim.ai/pricing) - [Docs](https://docs.getbifrost.ai) - [GitHub](https://github.com/maximhq/bifrost) - [Book a Demo](https://www.getmaxim.ai/book-a-demo) ## Status Overview - **15 Providers Monitored.** - **10/15 Operational.** - **4 Active Incidents.** ## Is the LLM or API you rely on down? Quick links to check live component status and active incidents for each provider. - [Is ChatGPT down?](https://www.getmaxim.ai/provider-status/openai.md) - [Is Claude down?](https://www.getmaxim.ai/provider-status/anthropic.md) - [Is Groq down?](https://www.getmaxim.ai/provider-status/groq.md) - [Is Azure OpenAI down?](https://www.getmaxim.ai/provider-status/azure.md) - [Is DeepSeek down?](https://www.getmaxim.ai/provider-status/deepseek.md) - [Is Cohere down?](https://www.getmaxim.ai/provider-status/cohere.md) ## All Providers Click on any provider to see detailed status, components, and incident history. | Provider | Current Status | Active Incidents | | --- | --- | --- | | OpenAI | Operational | 0 | | Anthropic | Operational | 0 | | Cohere | Operational | 0 | | Groq | Operational | 0 | | Azure AI | Unknown | 0 | | ElevenLabs | Operational | 0 | | Cerebras | Operational | 0 | | DeepSeek | Unknown | 0 | | Stability AI | Operational | 0 | | Replicate | Operational | 0 | | Together AI | Major Outage | 0 | | Fireworks AI | Unknown | 0 | | Hugging Face | Operational | 4 | | Perplexity | Operational | 0 | | OpenRouter | Degraded | 0 | ## Quick Links Jump to a specific provider's status page. - [OpenAI](https://www.getmaxim.ai/provider-status/openai.md) - [Anthropic](https://www.getmaxim.ai/provider-status/anthropic.md) - [Cohere](https://www.getmaxim.ai/provider-status/cohere.md) - [Groq](https://www.getmaxim.ai/provider-status/groq.md) - [Azure AI](https://www.getmaxim.ai/provider-status/azure.md) - [ElevenLabs](https://www.getmaxim.ai/provider-status/elevenlabs.md) - [Cerebras](https://www.getmaxim.ai/provider-status/cerebras.md) - [DeepSeek](https://www.getmaxim.ai/provider-status/deepseek.md) - [Stability AI](https://www.getmaxim.ai/provider-status/stability-ai.md) - [Replicate](https://www.getmaxim.ai/provider-status/replicate.md) - [Together AI](https://www.getmaxim.ai/provider-status/together-ai.md) - [Fireworks AI](https://www.getmaxim.ai/provider-status/fireworks-ai.md) - [Hugging Face](https://www.getmaxim.ai/provider-status/huggingface.md) - [Perplexity](https://www.getmaxim.ai/provider-status/perplexity.md) - [OpenRouter](https://www.getmaxim.ai/provider-status/openrouter.md) ## Build Resilient AI Apps With Automatic Failover Bifrost AI Gateway can automatically detect provider outages and route requests to healthy alternatives, so your application maintains uptime even when individual providers experience issues. - [Bifrost AI Gateway](https://www.getmaxim.ai) - [Book a Demo](https://www.getmaxim.ai/book-a-demo) ## FAQ ### Is ChatGPT down? Open the live ChatGPT / OpenAI status page for current uptime, component health, and incidents (from OpenAI's official status feed). [ChatGPT status](https://www.getmaxim.ai/provider-status/openai.md). ### Is Claude down? Open the live Claude / Anthropic status page for current uptime, component health, and incidents (from Anthropic's official status feed). [Claude status](https://www.getmaxim.ai/provider-status/anthropic.md). ### Is Groq down? Open the live Groq / Groq status page for current uptime, component health, and incidents (from Groq's official status feed). [Groq status](https://www.getmaxim.ai/provider-status/groq.md). ### Is Azure OpenAI down? Open the live Azure OpenAI / Azure AI status page for current uptime, component health, and incidents (from Azure AI's official status feed). [Azure OpenAI status](https://www.getmaxim.ai/provider-status/azure.md). ### Is DeepSeek down? Open the live DeepSeek / DeepSeek status page for current uptime, component health, and incidents (from DeepSeek's official status feed). [DeepSeek status](https://www.getmaxim.ai/provider-status/deepseek.md). ### Is Cohere down? Open the live Cohere / Cohere status page for current uptime, component health, and incidents (from Cohere's official status feed). [Cohere status](https://www.getmaxim.ai/provider-status/cohere.md). ### What is the AI Provider Status Monitor? The AI Provider Status Monitor is a real-time tracking dashboard that aggregates uptime, incident, and maintenance data from official status pages of major AI model providers including OpenAI, Anthropic, Groq, Azure AI, and 15+ others. It provides a single view of operational health across the AI infrastructure ecosystem, helping teams make informed decisions about provider routing and failover strategies. ### How often is provider status data updated? Status data is fetched directly from each provider's official status API and revalidated every 60 seconds to ensure you always see the latest operational status, active incidents, and component health across all monitored providers. This near real-time visibility helps teams quickly identify and route around provider issues. ### Which AI providers are monitored? We currently monitor 15+ major AI providers including OpenAI (ChatGPT, GPT-4, GPT-4o), Anthropic (Claude 4, Claude 3.5), Azure AI, Groq, Cohere, ElevenLabs, Together AI, Fireworks AI, Perplexity, OpenRouter, Hugging Face, Cerebras, DeepSeek, Stability AI, and Replicate. Each provider page shows detailed component status and incident history to help you build resilient multi-provider architectures. ### How can I check if a specific AI service is down? Click on any provider from the main status table or quick links section to view detailed component-level status. Each provider page breaks down services by component (API, Chat Interface, Embeddings, etc.) and shows real-time operational status, active incidents, and recent incident history. This visibility is essential for teams implementing automatic failover with Bifrost AI Gateway. ### What does "operational", "degraded", and "major outage" mean? Operational means all services are functioning normally. Degraded performance indicates partial service disruptions or slower response times. Bifrost can automatically route traffic away from degraded providers. Major outage means critical services are unavailable. Maintenance indicates scheduled downtime. Unknown means status data is temporarily unavailable from the provider's API. ### Can I see historical incident data for AI providers? Yes, each provider page includes a complete incident history section showing past outages, degraded performance events, maintenance windows, and resolutions. This helps teams understand reliability patterns, plan redundancy strategies, and make informed decisions about which providers to include in their Bifrost routing configuration for maximum uptime. ### How can I prevent AI provider downtime from affecting my application? The best approach is automatic multi-provider failover. Bifrost AI Gateway monitors provider health and automatically routes LLM traffic to operational alternatives when a provider degrades or goes down. This ensures your application maintains uptime even during provider outages, with zero manual intervention required. ### What data sources are used for status monitoring? All status data comes directly from each provider's official status page APIs including Statuspage.io (Atlassian), BetterStack, Instatus, and OnlineOrNot platforms. We do not perform independent uptime checks. All data reflects what providers report publicly about their service health. ## Related Resources - [All provider status](https://www.getmaxim.ai/provider-status.md) - [Docs: Bifrost docs](https://docs.getbifrost.ai) - [GitHub: maximhq/bifrost](https://github.com/maximhq/bifrost) - [Pricing: Bifrost pricing](https://www.getmaxim.ai/pricing) - [Enterprise: Bifrost enterprise](https://www.getmaxim.ai/enterprise) - [Book a Demo: Bifrost demo](https://www.getmaxim.ai/book-a-demo) - [Resources: Bifrost resources](https://www.getmaxim.ai/resources.md) --- *This is a markdown version of [https://www.getmaxim.ai/bifrost/provider-status](https://www.getmaxim.ai/bifrost/provider-status) for AI/LLM consumption.*