Skip to content
ReviewRamble

The weekly brief

AI, reviewed, in your inbox.

  • New reviews of tools, models, MCP servers and repos, with the verdict up front
  • Re-checked prices and ratings when something changes
  • New analysis pieces as they are published

Models, agents, MCP servers and the tools actually worth your time.

One email a week. Unsubscribe with the link in any brief. How we use your email

τ-bench

τ-bench (tau-bench) is a benchmark that tests AI agents in simulated customer-service conversations, where a language model plays the user and the agent must use domain API tools while following written policy rules.

Also called: tau-bench, τ²-bench, tau2-bench

By Divya Sharma · Checked Nov 15, 2026 · How we review

Real agents talk to people and have to follow company rules, not just call tools. τ-bench, published in 2024, tests both. A language model plays a customer with a goal, such as changing a flight or returning an order. The agent under test has domain-specific API tools and a policy document, and must reach the right outcome without breaking the rules.

How it is graded

At the end of each conversation, the benchmark compares the database state with the expected goal state. This checks what the agent actually did, not how it phrased things. The paper also introduced pass^k, the chance an agent succeeds on all k repeated trials of the same task. At release, even strong function-calling agents such as GPT-4o succeeded on under half the tasks, and pass^8 in the retail domain was below 25%.

τ²-bench

τ²-bench, from 2025, adds a dual-control setting. In its telecom domain, both the agent and the simulated user can use tools to change a shared environment, as in technical support where the agent must guide the customer to act. Agents scored much worse when they had to coordinate with an active user than when they worked alone.

Why builders care

The benchmark is a good model for testing your own AI agent: a simulated user, real tools, a policy and an outcome check. Its focus on consistency also shows why one successful demo is not enough; see pass@k for the contrast between best-of-k and every-time success.

Related terms

AI benchmark
An AI benchmark is a fixed, public set of tasks with an agreed scoring method, used to compare models on the same test, for example coding issues solved, exam questions answered or retrieval quality across many datasets.
Tool calling
Tool calling, also called function calling, is a model API feature where you describe functions with a schema, the model replies with a structured request to call one, and your code runs it and returns the result.
pass@k
pass@k is a metric for code benchmarks that gives the chance at least one of k generated solutions to a problem passes its unit tests; pass@1 is the chance that a single attempt succeeds.
AI agent
An AI agent is a system in which a language model decides its own next steps: it calls tools, reads the results and repeats until the task is done, instead of following a fixed script.
Voice agent
A voice agent is an AI agent that holds a spoken conversation in real time, listening to a person, deciding what to say or do, and answering in a synthetic voice, often over a phone line or in an app.

Sources