Robots Atlas>ROBOTS ATLAS
Evaluation

τ-bench

2024ActivePublished: 17 November 2026Updated: 17 November 2026Published
Key innovation
A benchmark that, instead of one-shot tasks, measures an agent's reliability in multi-turn conversations with a simulated user and API tools, introducing the pass^k metric that evaluates behavioural consistency across multiple independent trials.
Category
Evaluation
Abstraction level
Benchmark
Operation level
Agent runtimeTooling
Use cases
Evaluating reliability of tool agents across trials (pass^k)Testing policy-rule adherence by conversational agentsComparing agent strategies: tool-calling, Act, ReActEvaluating customer support in retail and airline domains

How it works

In each task the agent holds a conversation with a user simulated by a language model (by default gpt-4o with the llm strategy; react, verify and reflection are also available). The agent has access to a set of domain-specific API tools and to a policy-rules document it must follow. The interaction mutates a database state; at the end that state is compared against an annotated goal state, yielding a binary score for a single trial. To assess reliability, each task is run k times and pass^1…pass^k is reported — the fraction of tasks solved consistently across all k independent trials; values fall as k grows. The repository also ships auto_error_identification.py, which uses an LLM to assign fault (user, agent, or environment) and classify the fault type.

Problem solved

Prior agent benchmarks tested single, one-shot tasks and did not check real user interaction or consistent adherence to domain rules across many turns. τ-bench addresses this by evaluating an agent in a dynamic conversation with tools and policy, and by measuring reliability (pass^k), because in practice an agent must act correctly not once but repeatedly.

Evolution

Original paper · 2024 · arXiv preprint (2024) · Shunyu Yao
τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
Shunyu Yao, Noah Shinn, Pedram Razavi, Karthik Narasimhan
2024
Release of τ-bench (retail, airline)
Inflection point

Sierra releases τ-bench with the retail and airline domains and the pass^k metric; shows that top agents solve fewer than 50% of tasks.

2025
τ²-bench (dual-control environment)

A follow-up evaluating conversational agents when both agent and user can act on a shared environment (arXiv:2506.07982).

2026
τ³-bench (banking domain, voice modality)

The latest version adds a banking domain and a voice evaluation modality and fixes the retail/airline tasks; it officially supersedes the original τ-bench repository.