This is the paper introducing τ-bench, by Shunyu Yao (work done during an internship), Noah Shinn, Pedram Razavi and Karthik Narasimhan of Sierra. It was posted to arXiv as 2406.12045 on 17 June 2024. Code and data are released at github.com/sierra-research/tau-bench.
Summary
The authors argue that existing agent benchmarks "feature simplified instruction-following setups" in which the agent receives all information up front, with no human in the loop and no domain guidelines to consult, and so fail to test two properties they consider necessary for deployment: interacting with human users to gather information and authorization, and following domain-specific rules consistently. τ-bench ("Tool-Agent-User Interaction Benchmark") emulates conversations between a language-model-simulated user and an agent equipped with database API tools and a policy document.
Formulation
Each task is a partially observable Markov decision process in which the agent interacts with databases through API tools and with a simulated user through natural-language messages. The database state is hidden from both agent and user and is read or written only through tool calls. Each domain has a policy explaining the databases, procedures and restrictions; some restrictions are enforced by the API (for example, an unknown payment ID returns an error) and others are not, so the agent must apply them itself. The user is simulated by gpt-4-0613 from a system prompt containing the task instruction and the conversation so far; the user cannot see the agent's tool interactions. The reward is computed at the end of the episode by comparing the final database state with the annotated goal state and checking that required information appears in the agent's messages, so that different conversations reaching the same end state are all credited.
Construction and domains
Construction ran in three stages: manual design of the simplest workable database schemas, APIs and policies, drawing on and simplifying real-world counterparts; automatic data generation, in which GPT-4 wrote a code snippet to sample database entries that the authors then debugged; and manual task annotation validated by agent runs, iterating on each user instruction with a gpt-4-turbo function-calling agent until the outcome was unambiguous, including running every retail task in more than 40 trials. τ-retail has 500 users, 50 products and 1,000 orders, 7 write and 8 non-write tools, and 115 tasks covering cancelling or modifying pending orders, returning or exchanging delivered orders, changing addresses and providing information; each pending order may be cancelled or modified only once and each delivered order returned or exchanged only once. τ-airline has 500 users, 300 flights between 20 US cities and 2,000 reservations, 6 write and 7 non-write tools, and 50 tasks for booking, modifying, cancelling and refunding, with a more complex policy whose baggage, payment and change rules vary by membership tier and cabin class. The authors describe trading "quantity for quality": a small set of tasks run for multiple trials.
The pass^k metric
Because the user simulator is stochastic, the same task yields different conversations. The paper introduces pass^k, the probability that all k independent trials of a task succeed, averaged over tasks, as a measure of consistency, in contrast to pass@k, which credits any success.
Results
Twelve models were run through function calling (Llama-3 through text-formatted ReAct), with the agent at temperature 0.0, the user at 1.0, a limit of 30 agent actions per task and at least three trials. Pass^1 on retail and airline respectively: gpt-4o 61.2 and 35.2 (average 48.2), gpt-4-turbo 57.7 and 32.4, gpt-4-32k 56.5 and 33.0, gpt-3.5-turbo 20.0 and 10.8, claude-3-opus 44.2 and 34.7, claude-3-sonnet 26.3 and 27.6, claude-3-haiku 19.0 and 14.4, gemini-1.5-pro 21.7 and 14.0, gemini-1.5-flash 17.4 and 26.0, mistral-large 30.7 and 22.4, mixtral-8x22b 17.7 and 31.6, meta-llama-3-70B 14.8 and 14.4. Native function calling outperformed text-formatted ReAct and Act, and reasoning traces helped the text-formatted methods. For gpt-4o, pass^8 on retail fell below 25% despite a pass^1 above 60%. Running one trial of every retail task with a gpt-4o agent and gpt-4 user cost about $200 ($0.38 agent and $0.23 user per task), 95.9% of the agent cost being input tokens from the long system prompt.
Of 115 gpt-4o retail trajectories, 40 failed; four failures were traced to instruction typos or ambiguity and fixed, leaving 36 agent failures. About 55% involved wrong tool arguments or wrong information given to the user, 25% wrong decisions from misunderstanding domain rules (for example, calling the exchange tool before collecting every item, when the policy allows one call), and 19% partial resolution of compound requests. Removing the policy document reduced gpt-4o's pass^1 from 61.2 to 56.8 on retail but from 33.2 to 10.8 on airline, which the authors read as showing that retail success rests largely on common-sense tool use while airline requires the policy.
Limitations stated by the authors
The simulated user may carry typos or ambiguities, may lack domain knowledge (as real users do), and may be limited in reasoning and long-context memory; the schemas and policies are simplified; task curation is biased by tuning user prompts against a gpt-4-turbo agent; and success could be supplemented by further checks such as LM-judged rule following. The authors conclude that function-calling agents "lack sufficient consistency and rule-following ability to reliably build real-world applications."
Key claims
- Two domains: retail (115 tasks; 500 users, 50 products, 1,000 orders) and airline (50 tasks; 500 users, 300 flights, 2,000 reservations); user simulated by gpt-4-0613; reward from final database state plus required information in messages.
- pass^k defined as the probability of k consecutive successes on a task.
- Best pass^1: gpt-4o 61.2% retail, 35.2% airline; gpt-4o retail pass^8 below 25%.
- Policy ablation: gpt-4o airline 33.2 → 10.8 without the policy; retail 61.2 → 56.8.
- Failure breakdown of 36 gpt-4o retail failures: about 55% wrong argument or information, 25% wrong decision, 19% partial resolution.
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering (arxiv.org); the abstract page (arxiv.org) records a single version submitted 17 June 2024.
Relationships
- supports: τ-bench — the primary text for the original domains, metric and launch results.
- supports: Sierra — the company's research output.
- related: τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment (Barres et al., 2025) — the successor adding the dual-control telecom domain.
- instance-of: AI Benchmarks and Evaluation
- related: Agentic AI