Skip to content
ReviewRamble

The weekly brief

AI, reviewed, in your inbox.

  • New reviews of tools, models, MCP servers and repos, with the verdict up front
  • Re-checked prices and ratings when something changes
  • New analysis pieces as they are published

Models, agents, MCP servers and the tools actually worth your time.

One email a week. Unsubscribe with the link in any brief. How we use your email

LLM observability

LLM observability is recording what an AI application does in production, including prompts, responses, tool calls, latency, token use and cost, as traces you can search, so you can debug failures and see how the system behaves over time.

Also called: tracing, LLM monitoring, LLM tracing, traces, spans

By Divya Sharma · Checked Nov 15, 2026 · How we review

A normal web service fails loudly: an exception, a 500 error. An LLM application often fails quietly, with a confident wrong answer or an agent that loops through many tool calls before giving up. Observability tools capture enough detail to see what happened and why.

LLM tracing

The core unit is the trace. OpenTelemetry, an open standard many of these tools support, defines a trace as the path of a request through an application, and a span as one unit of work within it. In an LLM app one user question becomes a trace, and its spans might be the retrieval query, the reranker, each model call and each tool call, nested in the order they ran. Each span records its inputs, outputs, timing and attributes such as model name and token counts. OpenTelemetry also maintains semantic conventions for generative AI so that different tools label these fields the same way.

What teams do with it

Debug a single bad answer by reading the full trace. Spot slow steps and rising token costs. Group failures to find patterns. Attach user feedback to traces, and turn real failures into test cases for LLM evaluation and regression testing. Many platforms combine tracing with evals, for example Langfuse, Arize Phoenix and LangSmith. The observability stage guide compares them.

Limits and cautions

Traces store whatever users typed, which often includes personal or confidential data, so apply PII redaction and access controls to them as you would to a production database. Logging every token of every request also costs storage, so teams sample busy traffic.

Related terms

LLM evaluation
LLM evaluation, often called evals, is measuring how well a model or AI application performs a task by running it on a set of test cases and scoring the outputs against defined criteria, by code, by humans or by another model.
Regression testing (LLM apps)
Regression testing for LLM apps means re-running a fixed set of evaluation cases after every change to a prompt, model, retrieval setup or tool, to catch cases that used to work and now fail.
LLM gateway
An LLM gateway is a proxy server between your applications and model providers that exposes one API and handles keys, routing, fallbacks, rate limits, logging and spend tracking in a single place.
Token pricing
Token pricing is how most AI model APIs charge: a set price per million input tokens you send and a separate, usually higher, price per million output tokens the model generates, with discounts for cached input and batch jobs.
PII redaction
PII redaction is the detection and removal or masking of personally identifiable information, such as names, phone numbers, addresses and card numbers, from text or images before it is sent to a model, stored in logs or shown to others.

Sources