---
description: What to measure for LLM observability at the AI gateway, covering tokens, cost, time to first token, tail latency, errors, fallbacks and cache hits.
title: "LLM Observability at the Gateway: What to Measure, and Where"
image: https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/size/w1200/2026/09/llm-observability-at-the-gateway-what-to-measure-and-where-bifrost-isometric.png
---

Try Bifrost Enterprise free for 14 days. [Request access](https://www.getmaxim.ai/#enterprise-trial)

**TL;DR**

- LLM observability is the practice of capturing tokens, cost, latency, errors, fallbacks, cache behavior, and tool-call activity for every model request, then routing that data into the same metrics and tracing systems your platform team already runs.
- The AI gateway is the correct measurement point because it is the single choke point every LLM request crosses, which removes the need to instrument each application, SDK, or agent independently.
- Bifrost, the [open-source AI gateway](https://www.getmaxim.ai) built by Maxim AI, exports the full metric set natively through a Prometheus `/metrics` endpoint, an OpenTelemetry (OTLP) plugin, and a Datadog connector, with 11 microseconds of overhead per request at 5,000 requests per second.
- The metric set worth instrumenting on day one is input and output tokens, cost, time to first token (TTFT), tail latency, error rate by provider, fallback and retry counts, cache hit ratio, and tool-call duration for MCP-invoked tools.
- Time to first token searches are up 88% year over year, reflecting how streaming latency and per-request cost have become the two dominant reliability signals for teams running LLM workloads in production.

Production LLM traffic fails in ways HTTP monitoring does not catch: a provider returns 200 OK while streaming its first token 4 seconds late, another silently swaps a cheaper distilled model into a route, and cost per conversation triples over a weekend nobody looked at Grafana. [Bifrost](https://www.getmaxim.ai), the [open-source AI gateway](https://github.com/maximhq/bifrost) built in Go by Maxim AI, gives platform teams one place to measure and export those signals across every provider and application.

This post walks through the signals worth instrumenting for LLM observability, why the gateway is the right layer to capture them, and how Bifrost's [built-in observability](https://docs.getbifrost.ai/features/observability/default), plus the [Prometheus endpoint](https://docs.getbifrost.ai/features/observability/prometheus) and [OpenTelemetry plugin](https://docs.getbifrost.ai/features/observability/otel), fit into existing tooling.

## What Is LLM Observability?

LLM observability is the practice of capturing tokens, cost, latency, errors, fallbacks, cache behavior, and tool-call activity for every model request, and routing that telemetry into the metrics, tracing, and logging systems the rest of the platform already uses. It differs from generic API monitoring in what it records: token counts and USD cost per request, streaming timings such as time to first token, per-provider error attribution across a fallback chain, cache hit ratio by lookup mode, and duration of tools invoked by the model through the [Model Context Protocol (MCP)](https://docs.getbifrost.ai/mcp/overview).

The goal is the same as any observability program: detect regressions, attribute cost, debug failed requests, and answer capacity questions with data. LLM observability adds the fields that make those questions answerable for AI workloads specifically. A five-nines HTTP dashboard says nothing about whether a model quietly started truncating outputs, whether a provider is silently rate-limiting one [virtual key](https://docs.getbifrost.ai/features/governance/virtual-keys), or whether a new MCP tool is timing out 3% of the time.

## Why the AI Gateway Is the Right Measurement Point

The [AI gateway](https://www.getmaxim.ai/articles/top-5-llm-gateways-in-2026-a-production-ready-comparison/) is the right measurement point because every LLM request from every application in the organization crosses it exactly once. Instrumenting there produces a complete, provider-agnostic view with no per-application SDK changes, no per-language exporter to install, and no gap when a team ships a new service. Anything above the gateway (application SDKs) or below it (provider APIs) captures only a slice.

Three practical consequences follow from that placement. First, the gateway sees which model actually served the request after routing, fallback, and retries, so its metrics reflect what happened, not what the client asked for. Second, it can attribute every request to a virtual key, team, customer, or [business unit](https://docs.getbifrost.ai/features/governance), because governance labels are enforced there. Third, it can time the parts of the pipeline the client cannot see: queue wait, cache lookup, provider selection, and MCP tool execution. Bifrost adds 11 microseconds of overhead per request at 5,000 RPS in the [published benchmarks](https://www.getmaxim.ai/resources/benchmarks), so measurement at the gateway is not a latency tax.

### Gateway-Side vs SDK-Side vs Provider-Side Observability

Not every layer answers every question. The table below compares the three common measurement points on the questions platform teams actually ask.

| Question | Gateway-side (Bifrost) | SDK-side (per-app instrumentation) | Provider-side (OpenAI usage, Anthropic dashboard) |
| --- | --- | --- | --- |
| Tokens and cost across all apps and providers | Complete, one place | Per app; hard to unify across teams | Per provider only |
| Which model actually served the request after fallback | Yes, records the served provider and model | Sees only the request it made | Yes, but per provider |
| TTFT for streaming | Yes, `bifrost_stream_first_token_latency_seconds` | Only if the SDK measures it | Not exposed |
| Fallback and retry attempts | Yes, per-attempt trail | No | No |
| Cache hit ratio (direct and semantic) | Yes | No (cache lives at gateway) | No |
| MCP tool-call duration and error type | Yes, `bifrost_mcp_client_operation_duration_seconds` | Only if the app runs the tool | Not applicable |
| Attribution by virtual key, team, customer | Yes, native labels | Requires cross-service correlation | No |
| Deploy without touching every application | Yes | No, every app must add code | Not applicable |

SDK-side instrumentation still has a role for application-level spans (the user turn, the retrieval step). Provider dashboards remain useful for reconciling invoices. Gateway-side is the layer that answers infrastructure questions across every model and app in one place.

## The Signals to Instrument at the Gateway

Nine metric families cover the operational surface of an LLM workload. Each is exported by Bifrost through Prometheus, OTLP, or the Datadog connector, and can be scraped, pushed, or streamed into whatever stack the platform team already runs. The table below maps each signal to why it matters, where it is measured, and the exact metric name Bifrost emits.

| Signal | Why it matters | Where measured | Bifrost / Prometheus metric |
| --- | --- | --- | --- |
| Input tokens | Drives cost on prompt-heavy workloads; catches prompt bloat | Gateway | `bifrost_input_tokens_total` |
| Output tokens | Drives cost on generation-heavy workloads; catches runaway completions | Gateway | `bifrost_output_tokens_total` |
| USD cost | Directly connects usage to spend; enables per-team chargeback | Gateway (pricing table applied post-response) | `bifrost_cost_total` |
| Time to first token (TTFT) | The dominant perceived latency for streaming UX | Gateway (streaming path) | `bifrost_stream_first_token_latency_seconds` |
| Inter-token latency | Signals throttling, model swap, or provider degradation | Gateway (streaming path) | `bifrost_stream_inter_token_latency_seconds` |
| Upstream latency (full) | Provider-level p50/p95/p99 for capacity and SLO tracking | Gateway | `bifrost_upstream_latency_seconds` |
| Error rate by provider | Attributes failures to the right upstream and status code | Gateway | `bifrost_error_requests_total` (with `status_code`) |
| Fallback / retry activity | Confirms failover fired; catches silent degradation | Gateway | `bifrost_request_retries`, `bifrost_key_rotation_events_total` |
| Cache hit ratio | Confirms cost savings; catches cache-key misconfiguration | Gateway | `bifrost_cache_hits_total` (with `cache_type` = direct or semantic) |
| MCP tool-call duration and errors | Isolates agent latency in tool code, not the model | Gateway (MCP client) | `bifrost_mcp_client_operation_duration_seconds` |

Every metric above carries a common label set including `provider`, `model`, `virtual_key_id`, `team_id`, and `customer_id`, so any of them can be sliced by tenant or team without extra work.

### Tokens and Cost

Input and output tokens are counted per response as separate counters, so a runaway completion (large output on a small prompt) and a bloated prompt template (large input on a small output) each stand out. Cost is emitted as `bifrost_cost_total` in USD, computed against the gateway's pricing table after the response returns. Because the counter carries `virtual_key_id` and `team_id` labels, per-team chargeback is a Prometheus query rather than a nightly reconciliation script.

### Time to First Token and Tail Latency

TTFT is the streaming metric that correlates most directly with user-perceived latency. Bifrost records it as a histogram (`bifrost_stream_first_token_latency_seconds`) alongside inter-token latency and full upstream latency, so `histogram_quantile(0.95, ...)` on any of them is a one-line PromQL query. The `bifrost_active_requests` gauge covers in-flight concurrency. A companion post on [monitoring latency and cost in LLM operations](https://www.getmaxim.ai/articles/monitoring-latency-and-cost-in-llm-operations-essential-metrics-for-success/) breaks the p95 and p99 patterns down further.

### Errors, Fallbacks, and Retries

`bifrost_error_requests_total` breaks failures down by `status_code`, `provider`, and `model`, which is what matters when triaging a spike. Retry activity is captured in `bifrost_request_retries` (a histogram bucketed 0/1/2/3/5/10), and key rotations (a per-key failure that pushed Bifrost to a different API key) are counted in `bifrost_key_rotation_events_total` with a `fail_reason` label of `rate_limit_error`, `authentication_error`, or `billing_error`. That is enough to distinguish "our provider is unhealthy" from "one of our API keys is exhausted." Fallback chains are configured on the [Retries and Fallbacks](https://docs.getbifrost.ai/features/fallbacks) page.

### Cache Hit Ratio

Bifrost's [semantic caching](https://docs.getbifrost.ai/features/semantic-caching) supports two lookup paths: direct hash matching (deterministic exact replay) and semantic similarity matching (embedding-based). `bifrost_cache_hits_total` carries a `cache_type` label so hit ratio can be tracked per path. A sudden drop typically points to a missing `x-bf-cache-key` header, not to a cache outage, and the label breakdown surfaces that in seconds.

### MCP Tool Calls

Agent workloads spend meaningful time inside tools, not inside the model. `bifrost_mcp_client_operation_duration_seconds` is a histogram of MCP tool-call duration, labeled by `mcp_client`, `mcp_tool_name`, and `error_type` (empty on success, `auth_required` or `_OTHER` on failure). It captures the [tool-execution](https://docs.getbifrost.ai/mcp/tool-execution) side of an agent turn separately from the model call, so a slow agent can be attributed to a specific tool without adding tracing to the tool code. Related: the [seven metrics worth tracking for AI agent observability](https://www.getmaxim.ai/articles/7-metrics-you-should-track-for-ai-agent-observability/).

## Where to Export the Data

Three transports cover essentially every observability stack in production. All three ship in Bifrost and can run simultaneously; the choice depends on where dashboards already live.

### Prometheus Scrape or Push Gateway

The Prometheus route is the shortest path to a working dashboard. Bifrost exposes a `/metrics` endpoint whenever the [telemetry plugin](https://docs.getbifrost.ai/features/telemetry) is enabled (on by default), and any [Prometheus server](https://prometheus.io/docs/prometheus/latest/getting_started/) can be pointed at it with a standard `scrape_configs` entry. A companion piece covers [Prometheus metrics and dashboards for LLM traffic](https://www.getmaxim.ai/articles/llm-observability-with-prometheus-metrics-and-dashboards/) in detail. For multi-node clusters where scraping through a load balancer would miss nodes, Bifrost also supports pushing metrics to a Prometheus Push Gateway.

### OpenTelemetry (OTLP)

The [OTel plugin](https://docs.getbifrost.ai/features/observability/otel) exports traces (and, in newer versions, metrics) to any OTLP collector over HTTP or gRPC. Bifrost emits spans following the [OpenTelemetry GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/), so backends that already understand GenAI attributes get rich model, prompt, and token metadata without custom parsing.

Session grouping (`group_traces_by_session`) collapses every request that shares an `x-bf-session-id` header into one trace, the correct view for a multi-turn agent run or a [Claude Code](https://docs.getbifrost.ai/cli-agents/claude-code) session. A deeper walkthrough on [OpenTelemetry traces and metrics for LLM observability](https://www.getmaxim.ai/articles/opentelemetry-for-llm-observability-traces-and-metrics/) covers the collector setup end to end.

### Datadog Connector

For teams standardized on Datadog, the Datadog connector is a native integration built on dd-trace-go v2 rather than a generic OTLP hop. It sends APM traces, Datadog LLM Observability payloads, and DogStatsD metrics in one plugin, and supports both Agent mode (through a local Datadog Agent) and Agentless mode (direct to Datadog's intake APIs). LLM Observability spans land in Datadog's [LLM Obs dashboards](https://docs.datadoghq.com/llm_observability/) with model, prompt, and cost attributes already populated.

## Implementation Walkthrough: Metrics in Ten Minutes

The minimum viable observability setup takes three steps once [Bifrost](https://docs.getbifrost.ai/quickstart/gateway/setting-up) is running. Nothing changes in application code; every metric described above is captured automatically by the telemetry plugin.

**Step 1: Enable telemetry.** The telemetry plugin is on by default. If disabled, re-enable it in `config.json` or through the UI. The `/metrics` endpoint becomes live on the same port Bifrost serves inference on (default `8080`).

**Step 2: Point Prometheus at Bifrost.** A single `scrape_configs` entry is enough:

```yaml
scrape_configs:
  - job_name: 'bifrost'
    static_configs:
      - targets: ['bifrost-host:8080']
    scrape_interval: 15s
```

For clusters behind a load balancer, configure `push_gateway_url` so every node pushes metrics into a shared Push Gateway.

**Step 3: Add the OTel or Datadog plugin as needed.** Enable the `otel` plugin with a `collector_url`, or the `datadog` plugin with an agent address (or an API key for agentless mode). Both are configured entirely in `config.json` and do not require rebuilding the gateway.

Once the plugins are running, three starter queries produce a usable dashboard:

```
# 95th-percentile time to first token, per model
histogram_quantile(0.95,
  sum by (le, model) (rate(bifrost_stream_first_token_latency_seconds_bucket[5m])))

# USD cost per hour, per team
sum by (team_name) (rate(bifrost_cost_total[1h])) * 3600

# Cache hit ratio, per cache type
sum by (cache_type) (rate(bifrost_cache_hits_total[5m]))
  / sum by (cache_type) (rate(bifrost_upstream_requests_total[5m]))
```

## Full Traces, Not Just Metrics

Metrics answer "how many" and "how fast." A trace answers "what happened in this specific request." Bifrost's built-in observability captures a full log for every request, including the input messages, model parameters, chosen provider, output, token counts, cost, and latency, and records the retry `attempt_trail`: an ordered array of every attempt with its `key_id`, `fail_reason`, and whether it triggered a key rotation. That is enough to reconstruct why a request took 8 seconds when the p50 for that model is 400 milliseconds.

For long-term retention and compliance, [audit logs](https://docs.getbifrost.ai/enterprise/audit-logs) record administrative activity (who changed which config, when, and to what) with HMAC-signed events. [Log exports](https://docs.getbifrost.ai/enterprise/log-exports) stream request and response payloads to S3 or GCS so the logs database stays small. Together they cover the audit-trail requirements for SOC 2, GDPR, HIPAA, and ISO 27001 workloads. Related: [LLM monitoring metrics, audit logs, and controls](https://www.getmaxim.ai/articles/llm-monitoring-metrics-audit-logs-and-controls/).

## Governance Labels: Attribution Without Correlation

A metric without attribution answers only fleet-wide questions. Bifrost stamps every request with the identifiers platform teams actually query on: `virtual_key_id`, `team_id`, `customer_id`, and `business_unit_id`, alongside human-readable name pairs. Those labels flow through to Prometheus, OTLP, and Datadog automatically. Per-team cost, per-customer error rate, and per-virtual-key rate-limit hits become one PromQL expression rather than a cross-service correlation job. The [governance resource page](https://www.getmaxim.ai/resources/governance) covers how virtual keys become the primary attribution entity and how [budgets and rate limits](https://docs.getbifrost.ai/features/governance/budget-and-limits) attach to them.

## Beyond Metrics: Evaluation and Session-Level Quality

Gateway metrics answer operational questions with high resolution. Model-quality questions (whether an agent completed the task, whether hallucinations increased after a prompt change, whether a rewrite regressed answer quality on a golden set) sit above the gateway. Maxim AI's evaluation platform reads the same [gateway-side traces](https://www.getmaxim.ai/resources) Bifrost produces and adds session-level scoring, offline evaluations, and simulation. Bifrost measures what happened in production; the evaluation platform measures whether what happened was correct.

## Frequently Asked Questions

### What is observability in LLMs?

Observability in LLMs is the practice of capturing tokens, cost, latency (including time to first token), errors, retries, fallbacks, cache behavior, and tool-call activity for every model request, then routing that telemetry into the metrics, tracing, and logging systems the platform team already runs. It expands generic API observability with the fields specific to LLM workloads, so teams can debug streaming latency, attribute cost to tenants, and confirm that [failover](https://docs.getbifrost.ai/features/fallbacks) actually fired.

### Which LLM observability tool is the best?

The right answer depends on where the observability stack lives. Teams on Prometheus and Grafana get the fastest path by scraping the gateway's `/metrics` endpoint. Teams on Datadog use the native connector for APM traces, LLM Observability, and DogStatsD metrics. Teams on OpenTelemetry point Bifrost at their existing OTLP collector and inherit whatever backend it feeds.

### Does Datadog have LLM observability?

Yes. Datadog offers a dedicated LLM Observability product with dashboards for prompts, completions, tokens, and cost. Bifrost integrates natively with it through the Datadog connector, which uses dd-trace-go v2 and Datadog's LLM Obs SDK to send APM traces, LLM Observability payloads, and DogStatsD metrics from every request through the gateway. The connector supports both Agent mode and Agentless mode.

### What is time to first token (TTFT)?

Time to first token is the elapsed time between when the [LLM gateway](https://www.getmaxim.ai/llm-gateway) forwards a request to a provider and when the first response token arrives on the streaming connection. It is the dominant perceived-latency signal for streaming user interfaces because it determines how long the user stares at an empty cursor before text starts appearing. Bifrost records TTFT as a histogram (`bifrost_stream_first_token_latency_seconds`) labeled by provider and model, so p95 TTFT per model is a one-line PromQL query.

### How does an AI gateway differ from generic API monitoring for LLM traffic?

An AI gateway records fields generic API monitoring does not: token counts (input and output) per request, USD cost per response, TTFT and inter-token latency for streaming, per-provider error attribution across a fallback chain, cache-hit type (direct vs semantic), and MCP tool-call duration. It also stamps every request with virtual key, team, and customer labels for native attribution. Generic APM tools record HTTP status and latency, which is a small fraction of what an LLM workload needs.

### Do I need OpenTelemetry, Prometheus, and Datadog, or just one?

Just one is enough to start, and Bifrost lets teams pick whichever their platform stack already uses. Prometheus is the shortest path if there is no observability backend yet. OpenTelemetry is the right choice when a collector already exists. The Datadog connector fits when Datadog is the standard. All three plugins can run at the same time during a migration.

### Where should governance labels be applied?

At the gateway. Bifrost enforces governance through virtual keys and stamps `virtual_key_id`, `team_id`, `customer_id`, and `business_unit_id` on every metric, span, and log automatically. That means per-team cost, per-customer error rate, or per-virtual-key rate-limit rotations are queries against existing labels, not cross-service joins between the gateway, the identity provider, and the application.

## Get Started with LLM Observability on Bifrost

Every metric in this post is captured by Bifrost natively, and every export path (Prometheus, OpenTelemetry, and the Datadog connector) is configured entirely in `config.json` without touching application code. Teams running regulated or high-scale AI workloads can pair this with [Bifrost Enterprise](https://www.getmaxim.ai/enterprise) for [clustering](https://docs.getbifrost.ai/enterprise/clustering), in-VPC deployment, RBAC, and audit logs.

To see how gateway-side LLM observability fits into your platform stack, [book a demo](https://www.getmaxim.ai/book-a-demo) with the Bifrost team, or start from the [LLM Gateway Buyer's Guide](https://www.getmaxim.ai/resources/buyers-guide) for benchmarks and integration walkthroughs.

## Read next

[![Top AI Observability Tools for LLM Monitoring in 2026](https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/size/w720/2026/10/top-ai-observability-tools-for-llm-monitoring-in-2026-bifrost-isometric.png) Compare the top AI observability tools for LLM monitoring in 2026 on latency and error SLOs, cost alerts, failover visibility, and on-call alert routing.](https://www.getmaxim.ai/articles/top-ai-observability-tools-for-llm-monitoring-in-2026/)

[![Top 7 LLM Observability Tools in 2026](https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/size/w720/2026/10/top-7-llm-observability-tools-in-2026-bifrost-isometric.png) Compare the top LLM observability tools in 2026 by capture layer: gateway capture, SDK tracing, and OpenTelemetry libraries, with open-source options.](https://www.getmaxim.ai/articles/top-7-llm-observability-tools-in-2026/)

[![Top 5 Enterprise AI Observability Platforms in 2026](https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/size/w720/2026/10/top-5-enterprise-ai-observability-platforms-in-2026-bifrost-isometric.png) An AI observability platform captures traces, metrics, and cost for every LLM request and delivers them to the tools teams already run. This guide compares Bifrost, Datadog LLM Observability, New Relic AI Monitoring, Dynatrace, and Grafana Cloud on fit with an existing telemetry stack.](https://www.getmaxim.ai/articles/top-5-enterprise-ai-observability-platforms-in-2026/)

## More in Observability

[![AI Observability Platform for Monitoring LLM Costs](https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/size/w720/2026/09/ai-observability-platform-for-monitoring-llm-costs-bifrost-isometric.png) A guide to monitoring LLM costs with an AI observability platform, covering the cost signals to track and how to turn them into budgets.](https://www.getmaxim.ai/articles/ai-observability-platform-for-monitoring-llm-costs/)

[![MCP Tool Calling at Scale: Cutting Token Cost](https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/size/w720/2026/09/claude-code-monitoring-how-to-track-coding-agents-bifrost-isometric.png) TL;DR \* Coding agents authenticate directly to a model provider by default, so a team running Claude Code and Cursor has no single place where usage, cost, or prompts can be read. \* Routing coding agents through an AI gateway makes every request observable without installing anything on a developer'](https://www.getmaxim.ai/articles/claude-code-monitoring-how-to-track-coding-agents/)

```json
{
    "@context": "https://schema.org",
    "@type": "Article",
    "publisher": {
        "@type": "Organization",
        "name": "Maxim Articles",
        "url": "https://www.getmaxim.ai/articles/",
        "logo": {
            "@type": "ImageObject",
            "url": "https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/size/w256h256/2025/08/thumbnail.png",
            "width": 60,
            "height": 60
        }
    },
    "author": {
        "@type": "Person",
        "name": "Kuldeep Paul",
        "image": {
            "@type": "ImageObject",
            "url": "https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/2025/08/1727978381919.jpeg",
            "width": 800,
            "height": 800
        },
        "url": "https://www.getmaxim.ai/articles/author/kuldeep/",
        "sameAs": []
    },
    "headline": "LLM Observability at the Gateway: What to Measure, and Where",
    "url": "https://www.getmaxim.ai/articles/llm-observability-at-the-gateway-what-to-measure-and-where/",
    "datePublished": "2026-09-03T20:02:33.000Z",
    "dateModified": "2026-10-13T11:01:41.000Z",
    "image": {
        "@type": "ImageObject",
        "url": "https://storage.ghost.io/c/84/03/8403f2f6-141c-411a-8f55-a32d4291533e/content/images/size/w1200/2026/09/llm-observability-at-the-gateway-what-to-measure-and-where-bifrost-isometric.png",
        "width": 1200,
        "height": 675
    },
    "keywords": "Observability",
    "description": "A guide to LLM observability at the gateway, covering the signals to instrument and how to export them to Prometheus, OpenTelemetry or Datadog.",
    "mainEntityOfPage": "https://www.getmaxim.ai/articles/llm-observability-at-the-gateway-what-to-measure-and-where/"
}
```