Gateway-Level LLM Controls for Banks: Audit Logs, RBAC, Budgets

The revised interagency model risk management guidance that U.S. banking regulators issued on April 17, 2026, SR 26-2, supersedes SR 11-7 and states that generative AI and agentic AI are novel and rapidly evolving and therefore not within its scope, while still expecting institutions to apply sound risk management principles to them. That leaves banks building LLM governance without a prescriptive rulebook, and the controls an examiner will ask about (who called which model, on whose authority, at what cost, with what record) are infrastructure questions. Bifrost, the open-source AI gateway built in Go by Maxim AI, puts those gateway-level controls in the request path for banks running mission-critical AI workloads. This post covers the three that carry the most audit weight: audit logs, RBAC, and budget enforcement on every LLM call.

Why Banks Need Gateway-Level Controls on Every LLM Call

Gateway-level controls are policies enforced in the request path between an application and an LLM provider, so identity, authorization, spend limits, and logging apply to every call regardless of which team wrote the code. Controls implemented inside each application are per-application promises; controls at the gateway are enforced facts.

The distinction matters in a bank for four reasons:

• Coverage is provable. An examiner asks whether a control applies to all LLM usage. A gateway answers that with a network path, not a survey of repositories.
• Provider keys stop circulating. Application teams hold a gateway credential scoped to their permissions rather than a raw provider key that works from anywhere.
• Evidence is centralized. One system holds the record of calls and the record of policy changes, which is what an audit trail has to reconstruct. The measurement and accountability functions described in the NIST AI Risk Management Framework depend on that record existing.
• Controls survive turnover. Policy configured centrally outlives the engineer who wrote the client wrapper.

Bifrost is deployed as that control point. Banks route model traffic through it, and the governance layer applies before the request leaves the perimeter. Deployment patterns for regulated institutions are covered on the financial services and banking page.

Audit Logs: Two Layers of Evidence Examiners Ask For

Audit evidence for LLM usage has two distinct layers, and conflating them is the most common gap in bank AI governance reviews. One layer records what the models did. The other records what humans changed about the controls. Examiners ask for both, and they answer different questions.

The data plane: a record of every LLM call

Bifrost writes per-request metadata for every call: timestamp, provider, model, latency, token counts, cost, status, and identifiers. Because prompt and response payloads are the sensitive part, log exports stream large payload fields to your own S3 or GCS bucket while the database keeps searchable metadata and a pointer. Two properties matter for banks:

• Field-level exclusion: specific fields can be excluded from offload entirely, so a bank that has decided not to retain raw customer prompts can enforce that at the gateway rather than by policy memo.
• Separate retention: payload retention in object storage is configured independently from database retention, which lets record-retention schedules and operational log hygiene diverge without a custom pipeline.

The control plane: a record of every policy change

Audit logs in Bifrost Enterprise record administrative activity: who changed what, when, and which resource was affected. Entries can be signed with an HMAC key so their integrity is verifiable, filtered by action, outcome, and date range, exported as JSON, JSON Lines, or Syslog for a SIEM, and archived continuously to object storage for long-term retention. This is the layer that answers the question an examiner actually asks after an incident: not what the model returned, but who raised that team's budget ceiling in March, and whether anyone approved it.

RBAC: Deciding Who Can Call Which Model

Role-based access control at the gateway determines which people can view and change AI infrastructure, and which applications can call which models. Both halves are needed: separation of duties is a control-plane concern, and least privilege is a data-plane concern.

Bifrost Enterprise ships three system roles, Admin, Developer, and Viewer, carrying decreasing permission sets, and supports custom roles built from resource and operation pairs. Banks typically create roles that do not map to a generic product: an Auditor role with read access to logs and no configuration rights, or a temporary contractor role scoped to one project. Roles assign automatically from identity provider groups and claims through OIDC user provisioning, so access derives from the same directory that governs everything else in the bank, with Okta and Microsoft Entra supported.

RBAC governs operations. Data access control scopes results. Each role carries a data access scope, so a developer on the retail banking team sees the virtual keys, prompts, and logs belonging to their team rather than the whole institution's, and inference requests authenticated with a virtual key resolve to that key's owner. For a bank, that is the difference between a shared platform and a platform that a control owner can attest to.

Budget Enforcement: Limits That Bind at Request Time

Budget enforcement at the gateway rejects a request that would exceed an allocated limit, rather than reporting the overage after it is billed. That is the difference between a cost control and a cost dashboard, and it is the property that lets a bank delegate model access to a business unit without underwriting an open-ended spend commitment.

Bifrost implements budgets and rate limits hierarchically. Each level holds an independent budget, and checking is cumulative down the chain:

• Customer: the top-level allocation, useful for a line of business or a legal entity.
• Team: an allocation inside a customer, matching a cost center.
• Virtual key: the credential an application uses, with its own budget and its own request and token rate limits.
• Provider config: a per-provider allocation inside a virtual key, so a team's spend on one provider can be capped separately.

Virtual keys are the entity that ties this together. A key carries budgets, rate limits, and model and provider filtering, so the fraud analytics team's key can be limited to approved models on an approved provider with a monthly ceiling and a per-minute token cap. Costs are computed from live provider pricing against actual token usage per request type, including reduced cost for cached responses, so the number the limit binds against is the number that appears on the invoice.

Deployment Constraints in a Regulated Environment

Three constraints usually decide whether a gateway is viable inside a bank.

Data path. In-VPC deployment keeps the gateway, its logs, and its credentials inside your own cloud account, and Bifrost Enterprise supports air-gapped and on-premises installations for environments with no public egress. No third-party service sits between the bank and the provider, and the enterprise deployment options cover the topologies in more detail.

Leakage controls. Secrets detection inspects prompts and completions for credentials and tokens, and custom regex patterns let a bank redact or reject organization-specific identifiers, such as account number formats, before a request reaches an external provider.

High availability. The open-source gateway handles roughly 3,000 to 5,000 requests per second on a single instance, which is sufficient for many workloads, but running multiple open-source nodes against a shared Postgres is not supported because governance state is held in memory. Enterprise clustering synchronizes that state across nodes in real time, which is what a bank needs for an active-active deployment where budget counters must be consistent across every node.

Questions Banks Ask About Gateway-Level LLM Controls

Does SR 26-2 require this?

No. SR 26-2 places generative and agentic AI outside its formal scope and is explicitly non-binding in character, so no line item mandates a gateway. The agencies issued it jointly, with the OCC publishing it as Bulletin 2026-13. The expectation that survives the carve-out is that institutions apply existing risk management principles, and the controls examiners have always looked for (documented access, traceable decisions, enforced limits) still need somewhere to live.

How is this different from an internal proxy?

Function, mostly. Many banks build a thin proxy first and then discover the requirements arriving one at a time: per-team budgets, then SSO, then row-level log visibility, then immutable admin trails. The governance capabilities described here are the accumulated version of that list, and Bifrost is open source under Apache 2.0, so the proxy layer is inspectable rather than a black box in the request path.

Where do prompts and responses live?

Wherever the bank puts them. Payloads offload to your S3 or GCS bucket, specific fields can be excluded from retention entirely, and in an in-VPC or air-gapped deployment none of it leaves your network.

What does this add to request latency?

Bifrost adds 11 microseconds of overhead per request at 5,000 requests per second in sustained benchmarks. Governance checks resolve against in-memory state rather than a database round trip per call.

Getting Started with Gateway-Level Controls for Banks

Gateway-level controls give a bank one place to answer the questions that follow every LLM deployment: who called the model, under which role, against which budget, and where the record is. Bifrost provides audit logs across both the data and control planes, RBAC bound to your identity provider, hierarchical budget enforcement at request time, and in-VPC or air-gapped deployment, with an open-source core and 11 microseconds of overhead.

To map these controls to your institution's model risk framework and deployment constraints, book a Bifrost demo with the team.