OpenTelemetry for LLM Observability: Traces and Metrics

TL;DR

• LLM observability is the practice of capturing traces, metrics, and logs from model calls so teams can debug latency, cost, token usage, and errors across every provider in production.
• OpenTelemetry (OTLP) is the vendor-neutral standard for that telemetry, and its GenAI semantic conventions define a common set of span attributes (model name, token counts, provider) that every backend understands.
• Instrumenting LLM traffic at the gateway layer emits OTel traces and Prometheus metrics for every request without touching application code.
• The Bifrost gateway exports LLM traces to OTLP collectors (Grafana, Datadog, New Relic, Honeycomb) using GenAI semantic conventions, and exposes native Prometheus metrics at a /metrics endpoint.
• A single trace can group a multi-turn agent run by session ID, giving one connected view of user request, tool calls, and each model span.

An AI application that calls three or more LLM providers in production generates traffic that standard application performance monitoring does not describe: token counts, per-model cost, cache hit rates, streaming latency, and provider-level errors. LLM observability closes that gap by capturing traces and metrics for every model call, and OpenTelemetry is the open standard that carries them. Bifrost, the open-source AI gateway built in Go by Maxim AI, emits OpenTelemetry (OTLP) traces and native Prometheus metrics for all LLM traffic that passes through it, so teams get provider-agnostic telemetry without instrumenting each service by hand. This guide explains how OpenTelemetry models LLM calls as spans and metrics, how GenAI semantic conventions standardize those spans, and how to export the data to any OTLP-compatible backend.

What Is LLM Observability?

LLM observability is the practice of collecting traces, metrics, and logs from large language model calls so engineers can measure latency, cost, token usage, and error rates and debug behavior across every provider and model in production. It extends traditional observability with the dimensions that only LLM traffic has: prompt and completion tokens, per-request dollar cost, cache hits, tool calls, and streaming timing.

As the OpenTelemetry project notes in its own introduction to observability for LLM applications, generative-AI workloads introduce telemetry that traditional monitoring was never designed to capture. Standard APM tools track HTTP status codes and request duration. They do not know that one request cost $0.02 and used 4,100 tokens, or that a retry switched providers after a rate-limit error. LLM observability records those facts as structured telemetry, so a spike in cost or a drop in cache hit rate becomes a query rather than a guess. The three signals that make this work, traces, metrics, and logs, are exactly the signals OpenTelemetry was built to carry, which is why the two topics converge. For a broader survey of the discipline, see this overview of how to monitor, trace, and improve LLM-powered applications.

Why OpenTelemetry for LLM Observability

OpenTelemetry is a vendor-neutral, open-source standard for generating, collecting, and exporting telemetry data, and it is the practical foundation for LLM observability because it decouples instrumentation from any single monitoring vendor. Instrument once against the OpenTelemetry Protocol (OTLP), and the same traces flow to Grafana, Datadog, New Relic, or a self-hosted collector without rewriting a line.

For LLM traffic specifically, three properties matter. OpenTelemetry defines a common wire format (OTLP), so a trace produced by a gateway in Go is readable by a backend written in any language. It defines shared semantic conventions, so a span attribute like gen_ai.request.model means the same thing everywhere. And it separates the producer of telemetry from the consumer, so switching observability backends is a configuration change, not a re-instrumentation project. That portability is the reason a growing share of LLM tooling standardizes on OTLP rather than a proprietary agent. Teams evaluating where instrumentation should live can weigh the trade-offs in this LLM gateway buyer's guide.

Traces, Spans, and Metrics for LLM Calls

OpenTelemetry represents an LLM call as a span inside a trace, and aggregate behavior as metrics. A span is a single timed operation with structured attributes; a trace is a tree of spans representing one end-to-end request; metrics are numeric time series such as request counts, latencies, and token totals. LLM tracing captures the shape of individual requests, while metrics capture the population.

For an LLM request, the span records the operation start and end (latency), the model and provider, the input and output token counts, and success or error status. When an agent makes several model and tool calls to satisfy one user turn, each becomes a child span under a shared parent, so the full run reads as one connected tree. This is the difference between LLM tracing and LLM monitoring: tracing answers "what happened in this specific request," and monitoring answers "how is the system behaving in aggregate." Both are needed, and both come from the same instrumentation.

Bifrost captures this data for every request without any change to application code, recording inputs, outputs, tokens, cost, and latency through an asynchronous built-in observability pipeline that adds no latency to the request path.

GenAI Semantic Conventions for LLM Spans

The OpenTelemetry GenAI semantic conventions are a standardized set of span attribute names for generative-AI operations, so that a model name, token count, or provider is recorded under the same key regardless of who produced the telemetry. Without a shared convention, one tool writes model, another writes llm.model_name, and dashboards break every time the source changes.

The conventions define attributes such as the generative-AI system (provider), the requested model, the operation type, and input and output token counts, attached to the span that represents the model call. Because the names are standardized, a backend can build a token-cost dashboard that works across every instrumented service automatically. Bifrost exports traces in the genai_extension format, which follows these GenAI semantic conventions and carries rich LLM metadata on each span. It also supports the standard OTEL_RESOURCE_ATTRIBUTES environment variable, so resource-level tags like deployment.environment=production or service.version attach to every emitted span for filtering and alerting. Details of the format live in the OpenTelemetry integration docs.

Exporting LLM Telemetry to Grafana, Datadog, and Other Backends

OpenTelemetry telemetry is exported over OTLP to any compatible collector, which means the same LLM traces can land in Grafana Cloud, Datadog, New Relic, Honeycomb, or a self-hosted OpenTelemetry Collector by pointing at a different endpoint. The producer of the telemetry does not need to know which backend consumes it.

Bifrost sends LLM traces to an OTLP collector endpoint over HTTP or gRPC, with per-signal headers so a collector that needs a credential on only the trace or only the metrics endpoint is handled cleanly. Sensitive collector URLs and API keys read from environment variables at runtime using an env. prefix, keeping credentials out of stored configuration. For teams standardized on Datadog, Bifrost also offers a native Datadog connector that uses Datadog's own SDKs for APM traces, LLM Observability, and metrics, going beyond generic OTLP to reach Datadog-specific features like LLM Observability dashboards. Both paths give operational visibility without a proprietary agent embedded in every service. Comparison approaches for different platforms are covered in this rundown of AI observability platforms for LLM applications.

How Bifrost Emits OpenTelemetry and Prometheus Natively

Bifrost emits OpenTelemetry traces and Prometheus metrics natively, so LLM observability is a configuration step rather than an instrumentation project. The OTel plugin exports traces to any OTLP collector, and the telemetry plugin exposes a Prometheus /metrics endpoint that is enabled by default. Both run asynchronously, adding no latency to request processing.

On the metrics side, Bifrost tracks HTTP transport metrics (request counts, duration, and size histograms) and upstream provider metrics for every model call. The provider metrics include bifrost_upstream_requests_total, bifrost_upstream_latency_seconds, bifrost_input_tokens_total, bifrost_output_tokens_total, bifrost_cost_total in USD, and bifrost_cache_hits_total by cache type, which distinguishes direct hits from semantic caching, labeled by provider, model, and virtual key. That labeling turns a Prometheus query into per-team or per-model cost and latency breakdowns. A simple scrape configuration adds Bifrost as a target:

scrape_configs:
  - job_name: 'bifrost'
    static_configs:
      - targets: ['bifrost-host:8080']
    scrape_interval: 15s

For multi-node clusters, Bifrost pushes to a Prometheus Push Gateway so metrics from every node are captured regardless of load-balancer routing.

The full metric set and label reference is documented in the telemetry guide. Because Bifrost adds only 11 microseconds of overhead per request at 5,000 requests per second, this telemetry does not come at the cost of throughput.

Instrumenting LLM Traffic at the Gateway Layer

Instrumenting LLM traffic at the gateway layer means the gateway emits traces and metrics for every request that flows through it, so a single integration covers every application, language, and provider behind it. The alternative, adding an observability SDK to each service, produces inconsistent coverage and re-instrumentation work every time a new service or provider is added.

Because Bifrost is a drop-in replacement that only requires changing the base URL in existing code, routing traffic through it instruments every downstream LLM call across 1,000+ supported models at once. A request that carries an x-bf-session-id header has its trace tagged with the OTel-conventional session.id, and enabling group_traces_by_session collapses a multi-turn agent run into one trace so the whole conversation reads as a single tree. Coding agents such as Claude Code and Codex CLI are recognized automatically, so agent runs populate session.id without a Bifrost-specific header.

The gateway approach also centralizes governance and cost control alongside observability, since the same virtual keys that scope budgets also label the telemetry.

For enterprises running regulated or air-gapped workloads, the same telemetry stays inside the network with an in-VPC deployment, and logs and metrics can be shipped to internal systems with log exports.

Teams wanting a Bifrost-specific walkthrough of logging and OTel tracing can follow the complete guide to LLM logging and OTel tracing in Bifrost, and those setting a standard for their stack can review these LLM observability best practices.

Frequently Asked Questions

What is observability in LLMs?

Observability in LLMs is the ability to understand a language model application's behavior from the telemetry it emits: traces of individual requests, metrics such as latency, cost, and token usage, and structured logs of inputs and outputs. Bifrost captures all of this through built-in request logging and monitoring, so teams can debug slow or failed model calls and monitor cost and reliability across providers rather than inferring problems from user reports.

How does OpenTelemetry work for LLM observability?

OpenTelemetry represents each LLM call as a span with standardized GenAI attributes (model, provider, token counts, latency) and exports those spans over the OpenTelemetry Protocol (OTLP) to any compatible backend. Because instrumentation targets the open OTLP standard rather than a vendor SDK, the same LLM traces can be sent to Grafana, New Relic, Honeycomb, or a self-hosted collector by changing only the endpoint.

What are the GenAI semantic conventions?

The GenAI semantic conventions are OpenTelemetry's standardized attribute names for generative-AI spans, such as the model requested, the provider, the operation type, and input and output token counts. Standardizing these names means a token-cost or latency dashboard works across every instrumented service automatically, because the same data is always recorded under the same keys regardless of which tool produced it.

Does Datadog have LLM observability?

Yes. Datadog offers an LLM Observability product for monitoring AI applications. Bifrost integrates with it through a native Datadog connector that uses Datadog's own SDKs for APM traces, LLM Observability, and metrics, reaching Datadog-specific features like LLM Observability dashboards. Bifrost can alternatively export the same LLM telemetry to Datadog over standard OTLP if a generic pipeline is preferred.

What is the difference between LLM tracing and LLM monitoring?

LLM tracing captures the detailed shape of individual requests as spans, so an engineer can inspect exactly what happened in one slow or failed model call. LLM monitoring aggregates behavior into metrics such as request counts, latency histograms, cost, and token totals for dashboards and alerting. Tracing answers what happened in a specific request; monitoring answers how the system is behaving overall. Both derive from the same instrumentation.

How do you monitor LLM traffic without changing application code?

Route LLM traffic through a gateway that emits telemetry for every request. Bifrost is a drop-in replacement that only requires changing the base URL, and it exports OpenTelemetry traces and Prometheus metrics for all traffic passing through it. Browse the Bifrost resources hub for guides on wiring this into a broader observability stack. This instruments every downstream service and provider at once, with no observability SDK added to individual applications and no re-instrumentation when a new service is introduced.

Getting Started with LLM Observability on Bifrost

OpenTelemetry gives LLM observability a portable foundation: instrument once against OTLP, and the same traces and metrics reach any backend your team runs. Bifrost puts that foundation at the gateway layer, exporting GenAI-convention traces to OTLP collectors and native Prometheus metrics for every LLM call, so visibility into cost, latency, and token usage arrives without per-service instrumentation. To see how Bifrost can standardize LLM observability across your providers and environments, book a demo with the Bifrost team.