---
title: "Observability for LLM Traffic: Collect, Filter, Enrich, Route"
description: "TL;DR\n\n * An observability pipeline for LLM traffic moves telemetry through four stages: collect at the AI gateway, filter payloads and low-value spans, enrich with cost and routing context, and route each signal to the backend that can use it.\n * LLM telemetry breaks conventional pipelines on four axes: multi-"
image: https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/size/w1200/2026/09/observability-for-llm-traffic-collect-filter-enrich-route-bifrost-isometric.png
---

Try Bifrost Enterprise free for 14 days. [Request access](https://www.getmaxim.ai/#enterprise-trial)

**TL;DR**

- An observability pipeline for LLM traffic moves telemetry through four stages: collect at the AI gateway, filter payloads and low-value spans, enrich with cost and routing context, and route each signal to the backend that can use it.
- LLM telemetry breaks conventional pipelines on four axes: multi-kilobyte prompt payloads, unbounded label cardinality, personal data inside spans, and telemetry cost that scales with token volume.
- Only the gateway can enrich a request with cost per call, cache hit type, fallback reason, and virtual-key attribution, because those values are computed at the gateway and never exist in application code.
- Bifrost, the open-source [AI gateway](https://www.getmaxim.ai/articles/top-5-llm-gateways-in-2026-a-production-ready-comparison/), emits those fields as Prometheus labels, OpenTelemetry span attributes, and request-log fields, and adds 11 microseconds of overhead per request at 5,000 RPS.

Observability pipelines for LLM traffic fail for a specific reason: the telemetry an LLM request produces looks nothing like the telemetry an HTTP service produces, and pipelines built for the second are fed the first. A single chat completion can carry a 40 KB prompt, a customer's email address in the user turn, and a dollar cost that no application-side instrumentation can compute. [Bifrost](https://www.getmaxim.ai), the [open-source AI gateway built in Go](https://github.com/maximhq/bifrost) by Maxim AI, sits at the one point in the request path where all of those can be handled before the data reaches Prometheus, an OpenTelemetry Collector, or a data warehouse. This post walks through the four pipeline stages and what the gateway contributes at each.

## What Is an Observability Pipeline for LLM Traffic?

An observability pipeline is a processing layer that sits between the systems producing telemetry and the backends storing it, and that collects, filters, enriches, and routes logs, metrics, and traces in transit. For LLM traffic, the pipeline's collection point is the AI gateway, because the gateway is the only component that sees every model call from every application.

The four stages are the same ones the [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) formalizes as receivers, processors, and exporters. What changes for LLM telemetry is the content: a language model request produces the full conversation history, tool definitions, model parameters, streamed output chunks, token counts, and a cost figure that depends on a pricing table the application never sees. The companion guide to [LLM observability at the gateway](https://www.getmaxim.ai/articles/llm-observability-at-the-gateway-what-to-measure-and-where/) covers what to measure; this post covers how the data should move. Bifrost participates from the collect stage onward: its [built-in request logging](https://docs.getbifrost.ai/features/observability/default) captures inputs, outputs, tokens, cost, and latency for every request asynchronously, and its Prometheus and OpenTelemetry plugins export the same data in the formats downstream tooling expects.

## Why LLM Telemetry Breaks Conventional Observability Tooling

Conventional observability tooling assumes small, structured, low-cardinality events that contain no regulated data. LLM traffic violates all of those assumptions at once, so a pipeline tuned for microservice telemetry will drop LLM data, exceed its storage budget, or store personal data it was never meant to hold.

| Dimension | Conventional service telemetry | LLM request telemetry |
| --- | --- | --- |
| Payload size per event | Hundreds of bytes | Tens of kilobytes (prompt, history, tool schemas, output) |
| Label cardinality | Bounded (route, status, region) | Unbounded (model, user, session, prompt version, tenant) |
| Sensitive content | Rare, usually in headers | Routine, inside the message body |
| Cost of the event itself | Negligible | Proportional to tokens |
| Where call cost is known | Application | Only at the gateway, via the pricing catalog |

Each row is a pipeline design problem. Payload size argues for filtering content out of spans and offloading bodies to object storage. Cardinality argues for a fixed label set on metrics and unbounded detail only in logs and traces. Sensitive content argues for redaction before export. Cost-of-call being a gateway-only fact argues for enrichment at the gateway. The [LLM monitoring metrics, audit logs, and controls](https://www.getmaxim.ai/articles/llm-monitoring-metrics-audit-logs-and-controls/) an enterprise needs depend on those four decisions.

Bifrost keeps the pipeline's own overhead out of the request path: the logging plugin writes in background goroutines and adds under 0.1 ms, and the gateway adds [11 microseconds of overhead per request at 5,000 RPS](https://www.getmaxim.ai/resources/benchmarks) in sustained benchmarks.

## Stage 1: Collect LLM Telemetry at the AI Gateway

Collection is the stage where the pipeline decides what it will ever be able to know. For LLM observability, the gateway is the correct collection point because it is the single place that sees every request, provider response, retry, and routing decision, whichever SDK, framework, or coding agent produced the call.

Application-side instrumentation captures only what the application knows: the prompt it sent and the response it received. It cannot see that the request was retried on a rate-limited key, served by a fallback provider, or answered from cache. The [Bifrost AI gateway](https://www.getmaxim.ai) captures those events because it performs them, in three parallel outputs:

- **Request logs**, stored in SQLite, Postgres, or ClickHouse through the [log store](https://docs.getbifrost.ai/architecture/framework/log-store), with full content, `attempt_trail` for every retry, and `routing_engines_used` for every routing decision
- **Prometheus metrics** from the [telemetry plugin](https://docs.getbifrost.ai/features/observability/prometheus), exposed at `/metrics` or pushed to a Push Gateway, with a fixed, bounded label set
- **OpenTelemetry traces** from the [OTel plugin](https://docs.getbifrost.ai/features/observability/otel), emitted as `gen_ai.*` spans over OTLP/HTTP or OTLP/gRPC to any collector

Metrics answer aggregate questions cheaply, traces show one request end to end, and logs hold the payload. A [complete guide to LLM logging and OTel tracing in Bifrost](https://www.getmaxim.ai/articles/complete-guide-to-llm-logging-otel-tracing-and-observability-in-bifrost/) covers configuration for each; for pipeline design, the point is that all three originate from the same hook, so their fields agree by construction. Caller context is carried too: any `x-bf-lh-` request header is captured into log metadata, and forwarded headers appear on the `llm.call` span as `gen_ai.request.extra_header.<name>` attributes.

## Stage 2: Filter Payloads, Plugin Spans, and Content

Filtering is where the pipeline controls its own cost. An LLM span carrying a 40 KB prompt is two orders of magnitude larger than a conventional span of a few hundred bytes, and a trace recording every plugin hook carries sixteen plugin spans per request (eight built-in plugins, two hooks each). The filter stage decides what leaves the gateway.

Bifrost exposes three filtering controls at the source, where the bytes never cross the network:

| Control | Scope | Effect |
| --- | --- | --- |
| `disable_content_logging` on the OTel profile | Every exported span | Drops messages, prompts, reasoning, tool definitions, and tool arguments; keeps model, provider, tokens, cost, latency, status, and governance attribution |
| `disable_root_span_content` | Root span only | Removes duplicated input and output from the root span; child spans keep full content |
| `plugin_span_filter` (`include` or `exclude`) | Plugin hook spans | Exports spans only for the listed plugins; filtered children are re-parented so the trace stays connected |

The client-level `disable_content_logging` flag governs only the Bifrost log store, so a team that wants content out of both the local database and the OTLP export sets both flags. For payloads that must be retained but not in the hot database, [log exports](https://docs.getbifrost.ai/enterprise/log-exports) stream request and response bodies to S3 or GCS object storage while the logs database keeps only metadata and pointers; `object_storage_exclude_fields` can skip raw prompts entirely. Only S3 and GCS are supported today; Azure Blob and warehouse destinations are not implemented. Sensitive data is also a filter-stage decision that belongs at the gateway rather than in a collector processor, as the guide to [gateway-level PII redaction before provider transmission](https://www.getmaxim.ai/articles/gateway-level-pii-redaction-before-provider-transmission/) explains.

## Stage 3: Enrich With Cost, Cache Hits, Fallback Reason, and Virtual Key

Enrichment is the stage that justifies putting the pipeline's entry point at the gateway. Four fields that every cost report, incident review, and chargeback model depends on are computed at the gateway and exist nowhere else: the dollar cost of the call, whether the response came from cache, why a fallback fired, and which virtual key the request ran under.

A downstream processor cannot add these because it lacks the inputs: the pricing catalog and exact token counts, the cache lookup result, the failed attempt's error type, and the [governance layer](https://www.getmaxim.ai/resources/governance) that authenticated the request. Bifrost has all four when the response completes, and writes them to every output.

| Enrichment field | Prometheus | OpenTelemetry trace | Request log |
| --- | --- | --- | --- |
| Cost per call (USD) | `bifrost_cost_total` | `gen_ai.usage.cost` | `cost`, filterable by `min_cost` / `max_cost` |
| Cache hit and type | `bifrost_cache_hits_total` (by type) | Not a named span attribute; use metrics or logs | `cache_debug` with hit type, similarity, threshold |
| Fallback and retry reason | `fallback_index`, `routing_engine_used` (`core` on fallback or retry), `bifrost_key_rotation_events_total` | Error type and status on failed spans | `attempt_trail` with `fail_reason`, `routing_engines_used` |
| Virtual-key attribution | `virtual_key_id` / `virtual_key_name`, team, customer, project | Governance attribution retained even with content disabled | Virtual key, team, customer, project per row |

The cost figure comes from the [Model Catalog](https://docs.getbifrost.ai/architecture/framework/model-catalog), which resolves pricing for the provider and model that actually served the request, including a fallback model. The cache signal distinguishes a direct hash hit from a semantic hit; the [semantic caching](https://docs.getbifrost.ai/features/semantic-caching) plugin records the embedding model, similarity score, and threshold in `cache_debug`, so a pipeline can separate exact-match savings from similarity-match savings, as the deep dive on [semantic caching for LLMs](https://www.getmaxim.ai/articles/what-is-semantic-caching-a-technical-deep-dive/) explains.

Fallback enrichment is deliberately categorical. The routing trail records only the error type (`rate_limit_error`, `authentication_error`, `billing_error`) and HTTP status for each failed attempt, never the upstream provider message, because providers can echo API keys or user input in error bodies, so [automatic fallbacks](https://docs.getbifrost.ai/features/fallbacks) are auditable without pushing secrets into the trace store. Virtual-key attribution turns a telemetry pipeline into a chargeback pipeline: because [virtual keys](https://docs.getbifrost.ai/features/governance/virtual-keys) carry team, customer, and project identity, every sample, span, and log row arrives already labeled with who to bill.

## Stage 4: Route Signals to Prometheus, the OpenTelemetry Collector, and Data Stores

Routing sends each signal to the backend suited to its shape and retention. Metrics belong in a time-series database, traces in a trace backend reached through an OpenTelemetry Collector, and full payloads in object storage or a warehouse. A single LLM request in Bifrost can fan out to all three without any backend receiving data it cannot use.

| Signal | Bifrost exporter | Typical destination | Note |
| --- | --- | --- | --- |
| Metrics | [Prometheus plugin](https://docs.getbifrost.ai/features/observability/prometheus) (`/metrics` scrape or Push Gateway) | Prometheus, Grafana | Push so nodes behind a load balancer are not missed |
| Traces and metrics | [OpenTelemetry export](https://docs.getbifrost.ai/features/observability/otel) over OTLP/HTTP or gRPC | Any OpenTelemetry Collector, Grafana Cloud, New Relic, Honeycomb | Per-signal headers target separate trace and metrics endpoints |
| APM and LLM observability | [Datadog connector](https://docs.getbifrost.ai/features/observability/datadog) | Datadog APM, LLM Observability, DogStatsD | `custom_tags` applied to all traces and metrics |
| Streaming and warehouse | [Kafka](https://docs.getbifrost.ai/features/observability/kafka), Pub/Sub, [BigQuery](https://docs.getbifrost.ai/features/observability/bigquery), Splunk | Data lake, SQL analytics, SIEM | One JSON message per completed trace, keyed by trace ID |
| Raw payloads | Log exports (object storage) | S3, GCS | Database keeps metadata and pointers only |

The Prometheus path is the most common starting point and is covered in [LLM observability with Prometheus metrics and dashboards](https://www.getmaxim.ai/articles/llm-observability-with-prometheus-metrics-and-dashboards/). Prometheus observability for a gateway cluster has one trap: scraping `/metrics` through a load balancer samples whichever node answered, so [clustered deployments](https://docs.getbifrost.ai/enterprise/clustering) should push metrics through the Push Gateway or the OTel plugin's `metrics_enabled` mode to a central collector.

The OpenTelemetry path carries the richest data. Bifrost's `genai_extension` trace format follows the [OpenTelemetry GenAI semantic conventions](https://github.com/open-telemetry/semantic-conventions-genai), so `gen_ai.provider.name`, `gen_ai.request.model`, and token usage attributes are named the way collector processors and trace backends expect. From the collector, the [transform, filter, and attributes processors](https://opentelemetry.io/docs/collector/transforming-telemetry/) apply a second round of shaping before export.

The walkthrough of [OpenTelemetry for LLM observability](https://www.getmaxim.ai/articles/opentelemetry-for-llm-observability-traces-and-metrics/) shows that second round with a working collector config. Regulated deployments that need these connectors inside a private network run them from the [in-VPC enterprise build](https://www.getmaxim.ai/enterprise).

## Controlling High Cardinality Metrics in LLM Pipelines

High cardinality metrics are the fastest way for an LLM observability pipeline to exceed its storage budget. Every unique combination of label values creates a new time series, and LLM traffic offers thousands of candidates: model names, prompt versions, users, sessions, tenants, and tool names. The pipeline has to fix a bounded label set for metrics and push everything else into traces and logs.

The [Prometheus instrumentation guidelines](https://prometheus.io/docs/practices/instrumentation/) recommend keeping the cardinality of most metrics below 10 and treating anything that could exceed 100 as a signal to move that dimension out of monitoring into a general-purpose processing system. LLM telemetry crosses that line the moment user or session identifiers become labels.

Bifrost enforces the boundary at the source:

- The HTTP `path` label carries the matched route template ( `/v1/messages/batches/{batch_id}`), not the raw URL, so cardinality is bounded by registered routes rather than by every model or resource ID in a URL
- The raw complexity score from semantic routing is excluded from labels because it is unbounded; only `complexity_tier` ( `SIMPLE`, `MEDIUM`, `COMPLEX`) is labeled, and the score stays in request logs and trace attributes
- Governance identity (virtual key, team, customer, project) is labeled because it is bounded by the entities an organization configures, which is exactly the set a cost report needs

The rule this encodes: label what an organization provisions in its [governance configuration](https://www.getmaxim.ai/resources/governance), and log or trace what users generate at runtime. A per-session breakdown is a trace query, not a Prometheus query.

## Keeping Personal Data Out of LLM Spans and Logs

Personal data in LLM telemetry is routine, because the user's message is the payload. A pipeline that exports spans with content to a third-party trace backend has copied every email address and support transcript into a system that was never in scope for the original data-processing agreement. Redaction therefore has to happen before the data leaves the gateway and apply consistently to logs, traces, and connector exports. Bifrost's [guardrail redaction](https://docs.getbifrost.ai/enterprise/guardrails/redaction) offers three modes:

- `**runtime**` rewrites detected text in the live request and response with a `replace`, `mask`, or `hash` strategy, so the provider, the logs, and every export receive the redacted form
- `**logs_only**` leaves runtime content untouched but stores reversible placeholders such as `[EMAIL-1]` in Bifrost logs and sends placeholderized content to trace-export connectors
- `**runtime_reversible**` redacts at runtime with reversible placeholders so a user with the `Logs:Reveal` permission can recover originals inside Bifrost, while connectors receive placeholders only

In every mode the reveal mapping stays inside Bifrost, and Prometheus receives no request or response content at all. Detection can come from an external guardrail provider or from the in-process [custom regex guardrail](https://docs.getbifrost.ai/enterprise/guardrails/custom-regex), whose built-in PII Detection template covers email addresses, US phone numbers, US Social Security numbers, and credit-card-like numbers; it is pattern-based and does not detect personal names. The [guardrail options](https://www.getmaxim.ai/resources/guardrails) page covers the managed and provider-backed detectors.

One boundary matters for compliance teams: Bifrost [audit logs](https://docs.getbifrost.ai/enterprise/audit-logs) record administrative activity (who changed which configuration, when), with HMAC-signed entries and S3 or GCS archival. They are not a copy of request traffic, which lives in the request logs and the pipeline above; the two should be retained under separate policies. That division is part of deciding [what to measure and where at the gateway](https://www.getmaxim.ai/articles/llm-observability-at-the-gateway-what-to-measure-and-where/), and regulated deployments formalize it in the [Bifrost Enterprise](https://www.getmaxim.ai/enterprise) tier.

## Frequently Asked Questions

### What is the difference between an observability pipeline and an OpenTelemetry Collector?

An OpenTelemetry Collector is one implementation of an observability pipeline: its receivers, processors, and exporters map to the collect, filter and enrich, and route stages. For LLM traffic, the collector cannot enrich a span with cost, cache hit type, or fallback reason because it never had those inputs; the gateway supplies them, and the collector shapes and forwards what the gateway emits.

### How do you keep PII out of LLM traces?

Redact at the gateway before export, and apply the same policy to logs and connectors. Bifrost's `runtime` redaction mode rewrites detected text before the provider call so every downstream system receives the redacted form; `logs_only` and `runtime_reversible` send placeholders to connectors while keeping the reveal mapping inside Bifrost. For content that should never leave, `disable_content_logging` on the OTel profile drops message bodies while keeping cost, tokens, and attribution.

### How does Bifrost export Prometheus metrics from a multi-node cluster?

Bifrost supports pull-based scraping of `/metrics` on each node and push-based export to a Prometheus Push Gateway or an OTLP metrics endpoint. For clusters behind a load balancer, push is the reliable option, because a scrape that reaches the balancer samples only the node that answered. Push interval, job name, instance ID, and basic auth are configurable.

## Getting Started with Bifrost Observability Pipelines

Observability pipelines for LLM traffic succeed when the collect stage sits where the cost, cache, fallback, and attribution context already exists: the AI gateway. Bifrost emits all four fields to metrics, traces, and logs, filters content and plugin spans at the source, redacts personal data before export, and routes each signal to Prometheus, any OpenTelemetry Collector, Datadog, Kafka, BigQuery, or object storage without changing application code. Start with the [Bifrost resources hub](https://www.getmaxim.ai/resources) for configuration references, or [book a demo](https://www.getmaxim.ai/book-a-demo) to design an observability pipeline with the Bifrost team for your provider mix and compliance requirements.

## Read next

[![Top AI Observability Tools for LLM Monitoring in 2026](https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/size/w720/2026/10/top-ai-observability-tools-for-llm-monitoring-in-2026-bifrost-isometric.png) Compare the top AI observability tools for LLM monitoring in 2026 on latency and error SLOs, cost alerts, failover visibility, and on-call alert routing.](https://www.getmaxim.ai/articles/top-ai-observability-tools-for-llm-monitoring-in-2026/)

[![Top 7 LLM Observability Tools in 2026](https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/size/w720/2026/10/top-7-llm-observability-tools-in-2026-bifrost-isometric.png) Compare the top LLM observability tools in 2026 by capture layer: gateway capture, SDK tracing, and OpenTelemetry libraries, with open-source options.](https://www.getmaxim.ai/articles/top-7-llm-observability-tools-in-2026/)

[![Top 5 Enterprise AI Observability Platforms in 2026](https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/size/w720/2026/10/top-5-enterprise-ai-observability-platforms-in-2026-bifrost-isometric.png) An AI observability platform captures traces, metrics, and cost for every LLM request and delivers them to the tools teams already run. This guide compares Bifrost, Datadog LLM Observability, New Relic AI Monitoring, Dynatrace, and Grafana Cloud on fit with an existing telemetry stack.](https://www.getmaxim.ai/articles/top-5-enterprise-ai-observability-platforms-in-2026/)

## More in Observability

[![MCP Tool Calling at Scale: Cutting Token Cost](https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/size/w720/2026/09/claude-code-monitoring-how-to-track-coding-agents-bifrost-isometric.png) TL;DR \* Coding agents authenticate directly to a model provider by default, so a team running Claude Code and Cursor has no single place where usage, cost, or prompts can be read. \* Routing coding agents through an AI gateway makes every request observable without installing anything on a developer'](https://www.getmaxim.ai/articles/claude-code-monitoring-how-to-track-coding-agents/)

[![Open Source Observability Platform for LLM and Agent Workloads](https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/size/w720/2026/09/open-source-observability-platform-for-llm-and-agent-workloa-bifrost-isometric.png) TL;DR \* An open source observability platform for LLM traffic is assembled from four building blocks: the OpenTelemetry Collector for pipelines, Prometheus for metrics, Grafana for dashboards, and Jaeger (or another OTLP backend) for traces. \* A generic stack records HTTP status codes and latency but has no native concept of](https://www.getmaxim.ai/articles/open-source-observability-platform-for-llm-and-agent-workloads/)

```json
{
    "@context": "https://schema.org",
    "@type": "Article",
    "publisher": {
        "@type": "Organization",
        "name": "Maxim Articles",
        "url": "https://www.getmaxim.ai/articles/",
        "logo": {
            "@type": "ImageObject",
            "url": "https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/size/w256h256/2025/08/thumbnail.png",
            "width": 60,
            "height": 60
        }
    },
    "author": {
        "@type": "Person",
        "name": "Kuldeep Paul",
        "image": {
            "@type": "ImageObject",
            "url": "https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/2025/08/1727978381919.jpeg",
            "width": 800,
            "height": 800
        },
        "url": "https://www.getmaxim.ai/articles/author/kuldeep/",
        "sameAs": []
    },
    "headline": "Observability for LLM Traffic: Collect, Filter, Enrich, Route",
    "url": "https://www.getmaxim.ai/articles/observability-for-llm-traffic-collect-filter-enrich-route/",
    "datePublished": "2026-09-08T04:57:00.000Z",
    "dateModified": "2026-10-13T11:04:55.000Z",
    "image": {
        "@type": "ImageObject",
        "url": "https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/size/w1200/2026/09/observability-for-llm-traffic-collect-filter-enrich-route-bifrost-isometric.png",
        "width": 1200,
        "height": 675
    },
    "keywords": "Observability",
    "description": "TL;DR\n\n * An observability pipeline for LLM traffic moves telemetry through four stages: collect at the AI gateway, filter payloads and low-value spans, enrich with cost and routing context, and route each signal to the backend that can use it.\n * LLM telemetry breaks conventional pipelines on four axes: multi-kilobyte prompt payloads, unbounded label cardinality, personal data inside spans, and telemetry cost that scales with token volume.\n * Only the gateway can enrich a request with cost per ",
    "mainEntityOfPage": "https://www.getmaxim.ai/articles/observability-for-llm-traffic-collect-filter-enrich-route/"
}
```