LLM observability
LLM observability is recording what an AI application does in production, including prompts, responses, tool calls, latency, token use and cost, as traces you can search, so you can debug failures and see how the system behaves over time.
Also called: tracing, LLM monitoring, LLM tracing, traces, spans
By Divya Sharma · Checked Nov 15, 2026 · How we review
A normal web service fails loudly: an exception, a 500 error. An LLM application often fails quietly, with a confident wrong answer or an agent that loops through many tool calls before giving up. Observability tools capture enough detail to see what happened and why.
LLM tracing
The core unit is the trace. OpenTelemetry, an open standard many of these tools support, defines a trace as the path of a request through an application, and a span as one unit of work within it. In an LLM app one user question becomes a trace, and its spans might be the retrieval query, the reranker, each model call and each tool call, nested in the order they ran. Each span records its inputs, outputs, timing and attributes such as model name and token counts. OpenTelemetry also maintains semantic conventions for generative AI so that different tools label these fields the same way.
What teams do with it
Debug a single bad answer by reading the full trace. Spot slow steps and rising token costs. Group failures to find patterns. Attach user feedback to traces, and turn real failures into test cases for LLM evaluation and regression testing. Many platforms combine tracing with evals, for example Langfuse, Arize Phoenix and LangSmith. The observability stage guide compares them.
Limits and cautions
Traces store whatever users typed, which often includes personal or confidential data, so apply PII redaction and access controls to them as you would to a production database. Logging every token of every request also costs storage, so teams sample busy traffic.
Related terms
- LLM evaluation
- LLM evaluation, often called evals, is measuring how well a model or AI application performs a task by running it on a set of test cases and scoring the outputs against defined criteria, by code, by humans or by another model.
- Regression testing (LLM apps)
- Regression testing for LLM apps means re-running a fixed set of evaluation cases after every change to a prompt, model, retrieval setup or tool, to catch cases that used to work and now fail.
- LLM gateway
- An LLM gateway is a proxy server between your applications and model providers that exposes one API and handles keys, routing, fallbacks, rate limits, logging and spend tracking in a single place.
- Token pricing
- Token pricing is how most AI model APIs charge: a set price per million input tokens you send and a separate, usually higher, price per million output tokens the model generates, with discounts for cached input and batch jobs.
- PII redaction
- PII redaction is the detection and removal or masking of personally identifiable information, such as names, phone numbers, addresses and card numbers, from text or images before it is sent to a model, stored in logs or shown to others.