τ-bench
How it works
In each task the agent holds a conversation with a user simulated by a language model (by default gpt-4o with the llm strategy; react, verify and reflection are also available). The agent has access to a set of domain-specific API tools and to a policy-rules document it must follow. The interaction mutates a database state; at the end that state is compared against an annotated goal state, yielding a binary score for a single trial. To assess reliability, each task is run k times and pass^1…pass^k is reported — the fraction of tasks solved consistently across all k independent trials; values fall as k grows. The repository also ships auto_error_identification.py, which uses an LLM to assign fault (user, agent, or environment) and classify the fault type.
Problem solved
Prior agent benchmarks tested single, one-shot tasks and did not check real user interaction or consistent adherence to domain rules across many turns. τ-bench addresses this by evaluating an agent in a dynamic conversation with tools and policy, and by measuring reliability (pass^k), because in practice an agent must act correctly not once but repeatedly.
Implementation
Evolution
Sierra releases τ-bench with the retail and airline domains and the pass^k metric; shows that top agents solve fewer than 50% of tasks.
A follow-up evaluating conversational agents when both agent and user can act on a shared environment (arXiv:2506.07982).
The latest version adds a banking domain and a voice evaluation modality and fixes the retail/airline tasks; it officially supersedes the original τ-bench repository.