AI Gateway

Implementing LLM Guardrails with the Bifrost Enterprise AI Gateway

Implementing LLM Guardrails with the Bifrost Enterprise AI Gateway

Implement LLM guardrails at the gateway layer with Bifrost: content safety, PII detection, prompt injection blocking, and MCP tool governance in one platform.

LLM guardrails are the runtime controls that validate every prompt and response flowing through an AI application, blocking harmful content, redacting sensitive data, and enforcing policy before a request reaches a model or returns to a user. Implementing LLM guardrails inside each application is brittle: every team rebuilds the same checks, every new model integration drifts from the standard, and audit evidence is fragmented across services. Bifrost consolidates these controls at the gateway layer, validating inputs and outputs inline across every LLM provider and every connected MCP tool. The gateway is open source on GitHub, and the Bifrost documentation walks through guardrail configuration end to end.

Understanding LLM Guardrails at the Gateway Layer

LLM guardrails are policy-enforcement components that validate input prompts before they reach a model and inspect model outputs before they return to a user. They run synchronously in the request path, sit outside the model itself, and apply identical rules across every provider, model, and team that touches the gateway.

The OWASP Top 10 for LLM Applications 2025 places prompt injection at #1 and sensitive information disclosure at #2, with both classes of failure mitigated primarily through input and output validation rather than through prompt engineering alone (see the OWASP LLM01 reference for the full mitigation list). Implementing LLM guardrails at the gateway, rather than in each application, is the architectural decision that lets a single policy update propagate to every model call across the organization.

Bifrost's guardrail layer covers six provider integrations behind one configuration interface: Bifrost-native Secrets Detection (Gitleaks-backed), Bifrost-native Custom Regex (including a PII Detection template), AWS Bedrock Guardrails, Azure AI Content Safety, GraySwan Cygnal, and Patronus AI. A single rule can route content through multiple providers in sequence, which is the basis of defense-in-depth at the gateway.

Why Application-Layer Guardrails Break Down

Teams that start with library-based guardrails inside each service typically run into the same set of problems within months:

  • Fragmented enforcement. A new microservice ships with a different filter version, and policy coverage develops gaps.
  • Per-service credential sprawl. Every service holds its own Bedrock keys, Azure endpoint, or Patronus AI token, and rotation becomes a coordination problem.
  • Inconsistent audit evidence. Compliance reviews require pulling traces from each service rather than a single source of truth.
  • Engineering tax. Each team rebuilds the same checks, often with different timeouts, sampling rates, and failure modes.
  • Uncontrolled MCP tool exposure. Without a central control plane, every consumer of an MCP server can invoke every tool from that server, and tool outputs return unfiltered into the model context.

These failure modes are exactly what regulated industries cannot afford under the 2 August 2026 application date for most provisions of the EU AI Act, which require demonstrable policy enforcement and tamper-evident audit trails for high-risk AI systems. Centralizing guardrails in an enterprise AI gateway addresses each of these failure modes in one place.

How the Bifrost AI Gateway Implements Guardrails

Bifrost builds enterprise guardrails on two primitives: Rules and Profiles.

  • Profiles are provider configurations: an AWS Bedrock guardrail ARN, an Azure Content Safety endpoint, a Patronus AI key, a set of regex patterns, or a Secrets Detection configuration. Profiles encapsulate how content is evaluated.
  • Rules are CEL (Common Expression Language) expressions that determine when a check fires. A rule can match by message role, model name, content size, keyword presence, or a sampling rate, and it can apply to inputs, outputs, or both.

A single rule can link to multiple profiles. Profiles are reusable across rules. This separation lets a platform team configure credentials once and reference them from any number of downstream policies.

Bifrost adds 11 microseconds of overhead at 5,000 requests per second in sustained performance benchmarks, so even with multiple guardrail providers attached, enforcement does not become a latency bottleneck on high-throughput endpoints.

Dual-stage validation

Every rule can be scoped to input, output, or both. This produces a dual-stage pipeline:

  • Input validation catches prompt injection, PII entering the provider, credential leakage in prompts, and prompt-level policy violations.
  • Output validation catches hallucinations, PII leakage in responses, toxic generations, and indirect injection fallout from tool results.

Independent profile assignments at each stage let teams use, for example, AWS Bedrock for input PII detection and Patronus AI for output hallucination scoring on the same request.

Configuring Guardrails Step by Step

The configuration flow is the same whether teams use the Bifrost dashboard, the REST API, config.json, or Helm values.

Step 1: Create a profile

A profile defines a provider configuration. The example below registers an AWS Bedrock guardrail using a pre-existing Bedrock guardrail ARN:

curl -X POST <http://localhost:8080/api/enterprise/guardrails/providers> \\
  -H "Content-Type: application/json" \\
  -d '{
    "id": 1,
    "provider_name": "bedrock",
    "policy_name": "PII Detection Profile",
    "enabled": true,
    "config": {
      "access_key": "env.AWS_ACCESS_KEY_ID",
      "secret_key": "env.AWS_SECRET_ACCESS_KEY",
      "guardrail_arn": "arn:aws:bedrock:us-east-1:123456789:guardrail/abc123",
      "guardrail_version": "1",
      "region": "us-east-1"
    }
  }'

The same endpoint accepts azure, grayswan, patronus-ai, regex, and secrets as provider_name values. Credentials reference environment variables, which keeps secrets out of the configuration store and integrates with HashiCorp Vault, AWS Secrets Manager, Google Secret Manager, or Azure Key Vault when those are present.

Step 2: Define a rule

A rule binds a CEL expression to one or more profiles:

curl -X POST <http://localhost:8080/api/enterprise/guardrails/rules> \\
  -H "Content-Type: application/json" \\
  -d '{
    "id": 1,
    "name": "Block PII in Prompts",
    "description": "Prevent PII from being sent to LLM providers",
    "enabled": true,
    "cel_expression": "request.messages.exists(m, m.role == \\"user\\")",
    "apply_to": "input",
    "sampling_rate": 100,
    "timeout": 5000,
    "provider_config_ids": [1, 2]
  }'

CEL expressions let teams scope a rule narrowly:

  • request.model.startsWith("gpt-4") to apply only to a specific model family
  • request.messages.exists(m, m.content.contains("confidential")) to gate on keyword presence
  • request.messages.filter(m, m.role == "user").map(m, m.content.size()).sum() > 1000 to fire only on long prompts
  • Combined expressions for fine-grained policy boundaries

Step 3: Attach guardrails to requests

Once profiles and rules exist, applications attach guardrails by header or by request body. The header form is the lightest touch:

curl -X POST <http://localhost:8080/v1/chat/completions> \\
  -H "Content-Type: application/json" \\
  -H "x-bf-guardrail-ids: bedrock-prod-guardrail,azure-content-safety-001" \\
  -d '{
    "model": "gpt-4o-mini",
    "messages": [{"role": "user", "content": "Help me with this task"}]
  }'

Blocked requests return HTTP 446 with a structured violations array that specifies the offending policy, severity, and the action taken. Warning-only validations return HTTP 246 with the same diagnostic envelope. This shape lets applications surface meaningful errors without parsing free-text responses.

Extending Guardrails to the MCP Gateway

Bifrost's MCP gateway brings the same governance model to tool execution. A single Bifrost instance acts as both an LLM gateway and an MCP gateway, so content guardrails, tool-access controls, audit logs, and identity share one control plane.

Two layers of control apply to MCP traffic:

  • **Tool filtering per virtual key.** Each virtual key carries an explicit list of MCP clients and tools it can invoke. The default is deny: a virtual key with no MCP configuration sees no tools. Platform teams attach only the clients and tools each consumer needs, enforcing least privilege across the agent fleet.
  • Content guardrails on tool inputs and outputs. The same rules that validate LLM prompts also validate the arguments passed to MCP tools and the results returned from them. A guardrail rule attached to output validation catches PII or credentials returned by a tool before they propagate back into the model's context window.

By default, Bifrost does not auto-execute tool calls from a model. Tool calls returned in a response are suggestions until the application explicitly calls /v1/mcp/tool/execute, and Agent Mode auto-approval is opt-in per tool. This separates the "model proposes" and "system executes" steps, which is the architectural condition for meaningful audit and human approval.

Deployment Patterns for Regulated Industries

Guardrails are only credible to auditors if they cannot be bypassed by deploying in a different region, routing around the gateway, or losing evidence. Bifrost Enterprise addresses these through deployment patterns common in regulated environments:

  • In-VPC and on-premises deployments. The gateway, guardrail profiles, and audit logs run entirely inside a customer VPC or private Kubernetes cluster through in-VPC deployments, so request bodies and detection events never leave the customer's network perimeter.
  • Tamper-evident audit logs. Every guardrail evaluation, blocked request, redaction, and tool execution writes to immutable audit logs suitable for SOC 2, GDPR, HIPAA, and ISO 27001 evidence.
  • Defense-in-depth composition. A single rule can layer AWS Bedrock for PII, Azure Content Safety for moderation, and Patronus AI for hallucination scoring on the same high-risk endpoint. No single provider covers every failure mode, so Bifrost's open-source gateway is designed to compose them.

Teams running healthcare, financial services, insurance, or government workloads can review the Bifrost governance resource page and the Bifrost guardrails overview for industry-specific deployment patterns and policy templates.

Start Implementing LLM Guardrails with Bifrost

Bifrost gives enterprise platform teams a single control plane for implementing LLM guardrails across every model and every MCP tool the organization touches. Rules and profiles separate policy from configuration, dual-stage validation covers inputs and outputs, virtual keys and tool filtering extend the same governance model to MCP traffic, and in-VPC deployment keeps regulated data inside the customer perimeter. To see how Bifrost can centralize guardrails across an enterprise AI stack, book a Bifrost demo with the team.

Read next