AI Governance and Observability in Bifrost

Most teams that run AI in production eventually get the same question from finance. Spending went up last month, and someone wants to know which team, feature, or model caused it. Often there is no clear answer. The numbers exist, but they sit in two systems that were never built to share data.

This is a design problem, not a sign that anyone was careless. Often a team builds its metrics pipeline and its budgeting system at different times, owned by different people. One tracks requests and tokens. The other tracks teams and keys. Neither knows about the other. So when finance asks which team caused the increase, the cost data is in one table and the team data in another, with no shared key to join them on. Someone then spends hours reconstructing an answer that a single filter should have produced.

That gap between what you can see and who you can hold responsible is the main problem. Bifrost closes it.

What a gateway does

An LLM gateway sits between your applications and your model providers. Every request goes through it on the way out, and every response comes back through it. It exposes an OpenAI-compatible endpoint, so for most applications, adopting it is a base-URL change and little else.

Because every request passes through this one place, the gateway is where you can both observe usage and enforce policy. If you only use it to forward traffic, you waste that position. Bifrost uses it for both at once, because observability and governance run on one identity model. The team you set a budget for is the same team every metric is tagged with. The customer you rate limit is the same customer your cost dashboard breaks down by. There is no separate mapping to reconstruct later, because the two sides were never split.

Measuring usage without adding latency

A common assumption is that observability is expensive, and that every span you capture adds latency to a user's request. With Bifrost it does not.

Bifrost captures a full OpenTelemetry trace of every call and sends it wherever you already look. This includes Prometheus, any OTLP-compatible backend, Datadog, and streaming exporters for Kafka, Google Cloud Pub/Sub, and BigQuery. There is no proprietary format, and there is no dashboard you are forced to adopt.

Telemetry is emitted off the critical path, after the response is already on its way back to the user, so it adds no latency you would notice. The latency cost comes from capturing telemetry inline, before the response returns, which Bifrost does not do.

The data you get is what starts to matter once you move past a demo:

• Latency users feel. You get time-to-first-token and inter-token latency, the numbers a user notices while a response streams in.
• Token detail. You get cached versus fresh tokens, plus reasoning tokens and per-modality splits through the Datadog and BigQuery exporters.
• Cost per request. You get the dollar cost of each call from a live pricing catalog, summable across any dimension you tag.

Cost per request is the one that does the most work. It turns "spend tripled" into "the summarization service moved to a pricier model, and one team is 60% of the increase." The first is guesswork. The second is a decision you can make.

Once you can see cost, the next step is to stop paying for it twice. Bifrost can serve semantically similar requests from cache, so you do not pay tokens on a cache hit. Cache hit rate is itself a metric, so you can see the savings rather than estimate them.

Controlling spend and access

Seeing your spend only helps if you can act on it, so the other half of Bifrost is control.

Governance is organized around the entities a company already has, which are virtual keys, users, teams, business units, customers, and access profiles. These form a graph, not a strict ladder. A key attaches to a user or a team, teams roll up to business units and customers, and you attach controls at whichever of these dimensions fits. On any of them you can set two things. The first is a budget, which caps spend over a rolling or calendar-aligned window. The second is rate limits, which cap tokens and requests separately. Calendar-aligned windows line up with how finance reconciles, so "$5,000 a month" can mean the actual first of the month.

Budgets are enforced cumulatively. A request has to clear every budget above it, its own, its team's, and its business unit's, and if any one is tapped out the request is refused even if the others have room. When a request succeeds, the cost is debited from every level at once. So a runaway retry loop cannot exhaust the org's budget through a key that still has headroom, because the team budget stops it first.

When a request is blocked, the response uses a distinct status code, not a catch-all. Over budget returns a 402. A model the key is not allowed to use returns a 403. A rate limit returns its own code. Because these differ, a client can tell them apart and back off on a rate limit but fail fast on a permission error.

Virtual keys and access profiles keep this from becoming a full-time job. A virtual key maps to a policy instead of a raw provider key, so you never ship provider keys to your applications. You define an access profile once, listing allowed providers and models, budgets, and limits. Each key you mint inherits that profile's provider, model, budget, and MCP tool scope, and updating the profile in one place updates every key built from it. Anyone who has run a key-rotation drill across a fleet of services knows why this helps.

Access control runs on two axes people often conflate:

• What you can do. Role-based access control (RBAC), granular enough that a custom role can allow chat completions but not embeddings.
• What you can see. Data scoping, so the dashboards a developer opens show their team's traffic and not another business unit's numbers. This is the same telemetry from the previous section, filtered by who is asking.

None of this needs a shadow user directory. You connect it to your existing IdP, such as Okta, Entra, Google, Auth0, Keycloak, or any OIDC-compliant provider, mapping the groups you already maintain onto roles and teams, and deprovisioning people when they leave.

Every check runs in front of the provider call without adding noticeable latency. Bifrost's overhead stays in the microseconds even at thousands of requests per second. This is the point, not a benchmark to brag about. If controls slow down every call, teams tend to route around them. This is the same deal as the observability side. You can observe and govern every request without a latency cost you would notice.

Guardrails and risk

Budgets and roles govern usage. The harder questions are about risk, and the gateway is again the natural place to handle them, because every request already passes through it.

In Bifrost, guardrails are a policy layer. Rules route requests, scoped by team or customer, through whatever checks you stack, such as in-process regex and secret-scanning, or PII and jailbreak detection through external providers like AWS Bedrock, Azure Content Safety, Google Model Armor, and CrowdStrike. Two things make this more than a checkbox. Guardrails run on streaming responses, so unsafe output is inspected and blocked before it reaches the user. And they can redact PII in place and let the request through, instead of blocking the whole call.

A newer version of this problem is agents that call tools. Once a model can reach a filesystem, a database, or a shell, tool access becomes as important to govern as model access. A key that can run a delete command is a different level of risk than one that can only return text.

Bifrost governs MCP tool access per key. Each key carries an explicit allow-list of tools, enforced before and during execution. So a research assistant's key might expose read-only filesystem access, while a senior engineer's key allows database queries and Git operations. This is the same identity model and the same policy layer as everything above, extended to what an agent can do, not only which model it can call.

Audit logs belong to both halves of this story. An audit log is the request stream from the observability section, kept as immutable, append-only evidence. Bifrost can record every request, and each entry captures:

• Identity. The authenticated principal, the virtual key, and the SSO user, team, and customer it maps to.
• Request. The provider, model, parameters, tool calls, and timestamps.
• Response. Token counts, latency, and cost.
• Policy outcome. Whether the request was allowed, denied, redacted, or rerouted, plus the budget and rate-limit state at the time.

Because every line uses the same identity model as your budgets and dashboards, you can query the log per key, per team, or per customer, and every entry is attributable to a specific consumer. For regulated workloads you can disable content logging per environment, so Bifrost still records the model, key, latency, and status but not the prompt or response body. The logs export to your SIEM or data lake for SOC 2, ISO 27001, HIPAA, and GDPR retention.

The rest is the ordinary compliance work auditors ask about. Provider secrets resolve from your secret manager at runtime, such as HashiCorp Vault, AWS Secrets Manager, Google Secret Manager, or Azure Key Vault, instead of sitting in plaintext config.

None of this is impressive in a demo. But it is what you need when auditors or a security review show up months later.

One system, not two

Observability and governance are usually sold as separate products, and that split is the problem. You buy a dashboard from one vendor and a policy engine from another, then write and maintain glue to make them agree on what a team is.

In Bifrost they are one system. You cannot govern what you cannot see, and what you see is only useful when it is attributed to a team, customer, or key you can hold responsible. Put both on one identity model and the finance question stops being an investigation. You filter usage by team, see who spent what, and check whether they were allowed to.