Bifrost v2.0 is here

Today we are releasing Bifrost v2.0, the largest release we have shipped.

A year ago we open sourced Bifrost with one belief: a gateway only earns the right to sit in front of every AI request if it is performant and resilient at scale. Bifrost now runs in production at Fortune 500 companies across financial services, healthcare, technology, pharma and defense, moving trillions of tokens a day.

We also realized that most of an organization's AI traffic never reaches the gateway at all, because it comes from laptops, browser tabs, extensions and desktop apps that nobody configures.

Bifrost v2.0 does three things. It goes deeper on the request path: how large requests move, how clusters hold up on restrictive networks, and how routing decides which provider and which model a request deserves. It fortifies governance so it mirrors how your organization is shaped, who people are and what work they are doing. And it extends the same governance and security policies to every endpoint in the enterprise.

Speed and resilience are not features

Speed and resilience are table stakes for critical infrastructure and if your gateway fails to perform at scale, enterprises pay the price.

A public benchmark LiteLLM published in July, run against that month's Bifrost build, records the same throughput for Bifrost and their Rust gateway at every concurrency level they charted, and at the highest concurrency, Bifrost adds lower latency to the slowest requests.

We are re-running that benchmark against v2.0 that contains the streaming and memory improvements we've highlighted below, and a full write-up is coming as its own engineering blog. 

Every design decision in v2.0.0 continues our focus on performance and resilience at scale. The clearest example is how large requests move.

Payloads got big. People now send video, long documents and entire repositories through these APIs. In v2.0, the Gateway streams a large request straight through instead of buffering it. Memory stays flat at roughly 200KB per request whether the payload is 1MB or 20MB. Buffering allocated about 3MB at 1MB and about 60MB at 20MB, growing with every byte the caller sent. Ten concurrent 10MB requests now run in bounded memory.

The same discipline shows up in smaller places. Computing a governance state hash, which happens constantly on every node, has a hard budget of 512 bytes per call, enforced by a robust test suite that fails a release if we go over. 

Governance fortified to mirror your organization

Governance in Bifrost attaches to an identity (a user). v2.0 extends that model to organizations with thousands of people, contractors, several business units, and an auditor asking who could see what on a given day. Three layers now compose.

Access Profiles are the unit of what someone can do: which models, which providers, which budgets and rate limits, which guardrails apply. A profile is defined once and can be attached to many people. And because real people wear more than one hat, a user can now hold several profiles at once. Access is the union of everything their profiles allow, one profile pays for each request, and when that profile's budget runs out another can cover it.

Data access control per entity decides what someone can see, separately from what they can do. Logs, keys, profiles and configuration each carry their own visibility rules.

Business units are now a real many to many relationship between users and units. Someone can belong to two units, and each membership records where it came from: set by hand, or written by SCIM or an OIDC claim.

Underneath all of it, identity provisioning got stricter. You can now declare SCIM the single owner of user identity. A login token that tries to change a role, a team or a business unit is refused, so the attributes your IdP provisioned stay exactly as provisioned.

Projects: governance and cost attribution for enterprise work

Access Profiles answer who is making a request. Projects answer what the request is for, which is the question finance and platform teams ask most. A single engineer can run a nightly batch job, evaluate a new provider, and ship a customer-facing feature in the same week. Each deserves its own budget, its own model allowlist and its own line in the report.

v2.0 adds Projects for exactly this kind of work. A project is a named, budgeted scope that a request opts into. It carries its own members, budgets and rate limits, provider and model rules, and MCP tool allowlists. It can restrict or extend a caller's own access, and it decides who pays: the project's budget, the caller's, or both.

Clustering for every network type you deploy into

Bifrost clusters run as a mesh: nodes gossip membership and hold a stream to every peer. That is the fastest topology, and it needs inbound peer connectivity, which Cloud Run, some locked-down VPCs and many enterprise networks do not allow. v2.0 adds a second mode for those networks.

Mesh mode stays the default. Fast, no extra moving part, needs inbound connectivity.

Broker mode routes everything through a central broker, so nodes only ever dial out. It works on platforms where mesh cannot, with optional TLS and a shared secret, so Bifrost runs in serverless environments like Cloud Run and ECS without giving up speed. 

Inside both, we split what used to be one channel into separate typed lanes: key value state, circuit breaker state, load balancing telemetry, governance updates and diagnostics each move independently. One lane backing up no longer delays the others. All of them still multiplex over a single TCP connection per peer, which is robustly tested. A second connection appearing is a straight up test case failure.

We also built a cluster harness that runs a multi node cluster locally, drives budget cycles across 45 classified entity types, and exposes profiling while it does. Distributed state bugs are found by stress-testing the system, not left to discover as edge-cases post a release.

Balancing load by learning what normal looks like

A slow response is not always a degraded provider. A request carrying 40,000 tokens of context should take longer than one carrying 400. Most load balancers cannot tell those apart, so they either react to normal variation and pull healthy providers out of rotation, or they set thresholds loose enough that a genuinely degraded provider keeps taking traffic for minutes.

v2.0 learns the relationship between token count and latency for each route, continuously, and judges every response against what that route should have taken for a request of that size. When a provider drifts, it loses traffic gradually instead of being cut off. When it recovers, it earns traffic back gradually instead of being flooded the moment it looks healthy. A small slice of traffic keeps probing throughout, so a recovered provider gets noticed without manual involvement.

Routing that understands the request

Health decides which provider gets a request. It cannot decide which model the request deserves, and most requests do not need a frontier model. Sending all requests to the most capable (read: expensive) model is the largest line item most teams want to cut without giving up response quality. Balancing this trade off is critical at the gateway. 

We also believe that defining “simple”, “medium” or “complex” request types should be your call, not ours. Much of the category is building in a different manner - ship the gateway with a classifier calibrated on someone else's graded conversations. It demos well but it also puts a decision in your hot path that you did not make, cannot reproduce, and is difficult to audit. A gateway is a control point and an Auto Router you cannot reliably audit or explain is simply a bet on the cost-quality tradeoff.

In v2.0, Complexity Router moves from keyword scoring to a semantic classifier. Each request is embedded and matched against reference phrases in three tiers, Simple, Medium and Complex. We ship a few example phrases as a starting set, balanced across coding, math, writing, extraction, translation and agentic work, and every one is editable, so a team whose "simple" looks different from ours can teach the router in a few minutes instead of curating a vocabulary.

But what happens when the semantic match is under your set threshold? v2.0 also offers an optional LLM classifier as a fallback, ensuring users continue to get the most optimal model for their requests.

In the real world, the most critical part is what happens to user requests across a conversation. An agent session rarely holds one complexity level. It can open with a hard architectural question, then spend twenty turns on simple to medium follow up tasks. Bouncing between complexity tiers would mean switching models, thus breaking cache, increasing cost and having questionable conversation quality.

We tackled this with session-aware routing. The first classifiable turn sets the tier. A later turn can raise it, but never lower it. A conversation that starts at Medium stays on the Medium model even when the next request is trivially Simple. Once a session reaches Complex, Bifrost stops calling the classifier at all and keeps it there.

That is the trade-off we chose: cost savings come from the sessions that never needed the frontier model, quality is protected in the ones that did, and models never get constantly swapped as a conversation progresses.

Bringing all traffic to the Gateway

Everything above applies to traffic that reaches the Gateway. The question is what happens to traffic that never reaches it?

Your engineers run Claude Code, Codex, Cursor and Copilot on their laptops. Your finance team is pasting into a chat window in a browser tab. Your marketing team uses Claude Cowork. Some of these tools cannot be pointed at a gateway at all. Others can, but only if someone configures each one, on each machine, and keeps it configured. Either way, most of that traffic never touches your Gateway, but most of it touches your code and your data.

You cannot fix that with a policy document, and you should not fix it by banning the tools. The tools are how the work gets done now.

Bifrost Edge is now available to all

Every AI tool on every laptop, governed from the place you already govern everything else. No other AI gateway on the market offers endpoint coverage of this kind.

Edge governs the endpoint

Edge runs on macOS, Windows and Linux, rolls out through Jamf, Intune, Kandji or any other MDM solution, and sends AI traffic from locally installed tools through your Bifrost Gateway. That covers desktop apps, IDEs, coding agents in the terminal, web sites and the MCP servers they call (including STDIO tool calls). Nobody changes a setting, a key or a habit.

The Gateway is the control panel

There is no second console. The Gateway that serves your AI traffic also manages the fleet, and every change reaches devices on their next policy sync.

• Devices gives you visibility into every machine running Edge: hostname, platform, OS version, architecture, agent version, first seen, last active, and the AI apps and MCP servers found on it
• Approvals let you approve or deny each AI app and each MCP server for the whole fleet, then override for a team or a single user. Users and teams sync from your identity provider (Okta, Entra ID, Google, Keycloak, Auth0), so policy follows your directory.
• Surfacing tools/agents in use - Allowed and logged by default so you see the fleet first. You can choose to block anything not yet approved with a simple setting
• Interception - On or off for the fleet, or scoped to specific teams and users. The fleet setting is your kill switch.
• Routing per app lets you switch between Gateway mode or passthrough, set as a fleet default with per-app overrides.
• Custom domains - Add an internal model endpoint or any AI API that uses the OpenAI, Anthropic, Gemini, Cohere or Bedrock format, and Edge sends it through the matching Gateway route. Or block a domain outright.
• Certificates - Generate a CA or bring your own via your key management system, and rotate it using APIs. With remote signing enforced, the private key never leaves the Gateway. The Gateway only signs certificates for the AI domains you govern, so a compromised laptop cannot get a certificate for your bank or your intranet.
• Sign-in with SSO for your employees. Virtual-key sign-in covers deployments without an identity provider, and you can turn it off.
• Everything you already set - Virtual keys, budgets, rate limits, guardrails, routing, fallbacks and audit logs apply to Edge traffic the same way they apply to all Gateway traffic. People can see their own budgets and rate limits from the Edge menu.

Works cleanly with the security stack you already run

Edge is a good neighbour to any secure web gateway, SASE client or EDR. This is because:

• Edge only takes AI destinations. Your security agents' own cloud traffic never passes through it.
• Edge weakens nothing. It pins no certificates and trusts the operating system's certificate store, including the inspection root your security team deployed. No SSL inspection exemptions, no bypass for Edge.
• The forwarded request is a new connection. Edge's request to your Gateway leaves the device like any other traffic, so your security stack still sees and inspects it.

Edge has been battle-tested and deployed alongside Zscaler, Palo Alto Networks, Rapid7, Cloudflare WARP and CrowdStrike deployments without an exemption on either side.

Edge has two capture modes, so it fits whatever is already on the device:

• Routing mode adds narrow routes for AI destinations only. Use it with security clients that choose traffic by route.
• Socket-layer mode claims AI connections before routing happens, through a network extension on macOS and a connect-redirect filter on Windows. Use it with clients that grab traffic at the connection level.

Edge governs MCP too

• Discovery. Edge reads the MCP config of every AI client on the device and reports each server. The Gateway deduplicates them into one fleet catalog.
• Local servers launch behind a small Edge layer that checks every tool call against fleet policy and logs it next to your Gateway traffic. Remote servers are covered by Edge's traffic capture.
• Approval works per server and per tool, for everyone, a team or one user. A denied server is removed from the client's config and restored exactly if you approve it later. Tool arguments can be kept out of logs.

Bifrost Egress is releasing in private alpha

Edge covers people. Egress covers the machines that do the work.

Your production services, CI pipelines, batch jobs and autonomous agents make AI calls too, far more of them than any person does, and they run where no desktop agent can be installed. Egress brings the Edge model to those hosts. It runs alongside the workload and sends its AI traffic through the Gateway, without someone having to configure a base URL explicitly.

Egress is in private alpha with a small number of design partners. Register to help shape it: https://tally.so/r/J9Pa14 

Air-gapped, inside your infrastructure

Bifrost has no SaaS offering, and that is deliberate.

If a gateway is the point every AI request passes through, it sees every prompt, every response, every piece of code and every document your organization sends to a model. That is not something a security team should have to hand to a vendor.

Bifrost runs in your VPC, on your cloud, in your account, or in a fully disconnected environment. Deployments today run on EKS, GKE and AKS, including inside VPC Service Controls perimeters and across cloud identity federation.

No data leaves your environment, not even telemetry.

What else landed

• Virtual MCPs succeed MCP tool groups: a curated set of servers and tools served at its own /mcp/<slug> endpoint, attachable to access profiles and projects, replicated across the cluster
• Scheduled virtual key rotation on access profiles, from hourly to yearly, with a cooldown window so the previous key keeps working while clients roll over
• On-behalf-of auth support for MCP authentication
• Guardrail redaction now rewrites Anthropic and Gemini native passthrough traffic, including streams, and rules can choose whether to evaluate the whole conversation or only the latest turn
• Every request now settles onto exactly one resolved grant before governance and billing run, so "which profile paid for this" has one answer
• Prompt cache breakpoints injected automatically for clients that send none, off by default
• Databricks and GitHub Copilot as first class providers
• Skills repository for Agent Skills
• Splunk, Datadog, BigQuery, Kafka, Pub/Sub and Vault connectors, so gateway telemetry lands in whatever you already run
• Signed audit logs with archiving
• Guardrails covering PII detection, secret detection, prompt injection and content filtering, applied to model traffic and MCP tool calls alike
• Routing extracted into its own API surface
• Circuit breakers and Alerting as first class entities
• White label branding for teams that deploy Bifrost as internal platform
• Go 1.27 across the stack

Looking forward

The Gateway is the control plane. v2.0 extends it: deeper on the request path, governance that mirrors your organization, and Edge so the same policies and the same logs cover the laptop and the browser tab as well as the application. Egress is where that reach goes next: the CI job, the batch run, the agent on a server nobody installs anything on.

Bifrost is open source. Start with the Gateway, and talk to us when you need the rest.