τ-bench

τ-bench (tau-bench, for "tool-agent-user") is a family of benchmarks from the AI-agent company Sierra in which a language-model agent plays a customer-service representative. The agent talks with a user who is simulated by another language model, reads and changes records through programmatic tools, and must stay within a written policy. A conversation counts as a success when the underlying database ends in exactly the state the task's author specified. The original benchmark, published in June 2024, covers a retail and an airline domain; τ²-bench, published in June 2025, added a telecom troubleshooting domain in which the simulated user also has tools; and τ³-bench, released in early 2026, added a banking domain built on document retrieval, a real-time voice mode, and corrections to the older tasks (Source: τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (Yao et al., 2024); Source: τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment (Barres et al., 2025); Source: taubench.com). OpenAI, Anthropic, Google and Z.ai have reported τ²-bench results at model launches.

What it measures

Sierra's stated aim was to test three things it regarded as missing from agent benchmarks of the time: interacting with a person over many turns to gather information that is not given up front; following domain-specific rules; and doing both consistently over many repetitions (Source: sierra.ai). The authors contrast this with benchmarks such as WebArena and SWE-bench, where the agent receives the whole task at once and acts alone. τ²-bench extends the construct to what its authors call dual control: settings like technical support, where the customer has to do some of the work, so that the agent must guide another party's actions as well as take its own τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment (Barres et al., 2025).

Design

Components

Each domain consists of a database stored as JSON files, a set of tools written as Python functions that read or write it, a policy document in Markdown that is given to the agent as its system prompt, and a list of tasks. In the retail domain the database holds 500 users, 50 products and 1,000 orders, and the agent can cancel or modify pending orders, return or exchange delivered ones and update addresses. The airline domain has 500 users, 300 flights between 20 US cities and 2,000 reservations, with a more intricate policy on baggage allowances, payment combinations, changes and cancellations that varies by membership tier and cabin class. Some rules are enforced by the tools themselves, and others are left to the agent, as in the number of paid bags it enters when booking τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (Yao et al., 2024).

A task has two parts, both hidden from the agent: an instruction for the user simulator, giving the user's identity, goal and preferences, and the annotated ground truth, which is the set of database writes that should result, plus any facts the agent must tell the user. One retail example in the paper instructs the simulated user to return a water bottle and exchange two other items for the cheapest versions, to prefer whichever option saves more money if only one is possible, and to be brief. The correct outcome is a single return call with specific order, item and payment identifiers, and two dollar figures mentioned to the user. Instructions are written so that, under the policy, only one end state is possible τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (Yao et al., 2024).

Construction

The authors designed the schemas, tools and policies by hand as simplified versions of real systems, used GPT-4 to write code that generated the database entries, and then wrote and validated tasks manually. Validation meant running each task many times with a GPT-4-turbo agent, more than 40 trials per retail task, and rewriting the user instruction until low-scoring tasks were free of ambiguity. The authors note that tuning the tasks against one model's behaviour introduces some bias τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (Yao et al., 2024).

For the τ²-bench telecom domain the process changed. Language models drafted a requirements document, the tools, a mock database and unit tests, which the authors refined. Tasks were then generated by program: each atomic problem, such as airplane mode being switched on, is defined by an initialisation function, a solution function and an assertion about the final state, and problems are combined into composite tasks. This yielded 2,285 verifiably solvable tasks, from which 114 were sampled to balance three user intents (service, mobile data and MMS problems) and the number of sub-problems. The simulated user has 30 tools of its own, representing actions on a mock phone such as toggling settings or reading the status bar, and is assigned an easy persona, a hard persona with little technical knowledge, or none τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment (Barres et al., 2025).

Scoring and the pass^k metric

The original reward is 1 if the final database is identical to the annotated one and the agent's messages contain every required piece of information, and 0 otherwise. No human or model judges the conversation. The authors acknowledge that a reward of 1 is necessary but not sufficient for a good conversation: an agent could, for instance, issue a return without asking the user to confirm, which the policy forbids but the check does not detect. τ²-bench adds other criteria that a task may use, including assertions on the final state of the environment, natural-language assertions, and checks that particular actions occurred; telecom tasks use state assertions only (Source: τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (Yao et al., 2024); Source: τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment (Barres et al., 2025)).

Because the user simulator is sampled at a non-zero temperature, each run of a task produces a different conversation. The paper exploits this to define pass^k ("pass hat k"): the probability that all of k independent trials of a task succeed, averaged over tasks. It is the counterpart of pass@k, the probability that at least one of k trials succeeds, which is common in coding benchmarks. Pass@k rises with k and rewards occasional success; pass^k falls with k and rewards reliability, which the authors argue is what matters for an agent handling the same kind of request for many customers. The headline number developers quote is pass^1, which equals the ordinary average success rate τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (Yao et al., 2024).

Cost and human baseline

In the original paper the user simulator was GPT-4 and agents were limited to 30 actions per task; one trial of every retail task with a GPT-4o agent cost about $200. For τ²-bench the simulator is GPT-4.1 and one trial of all three domains cost about $40. Neither paper reports a human baseline (Source: τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (Yao et al., 2024); Source: τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment (Barres et al., 2025)).

Versions and variants

• τ-bench (June 2024): retail (115 tasks) and airline (50 tasks).
• τ²-bench (June 2025): re-implements retail and airline after a review of task specifications, and adds telecom (114 tasks). It supports a default dual-control mode, a no-user mode in which the agent operates all tools itself, and an oracle-plan mode in which the agent is handed the solution steps, so that reasoning failures can be separated from communication failures τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment (Barres et al., 2025).
• τ³-bench (version 1.0.0, early 2026): a banking_knowledge domain of 97 tasks over 698 documents with configurable retrieval, a full-duplex voice mode run through real-time audio APIs from OpenAI, Google and xAI, and more than 75 task fixes across domains. A July 2026 update, v1.0.1, corrected grading for some banking tasks; Sierra states that results from earlier versions of that domain are not comparable and that affected leaderboard entries were re-graded (Source: github.com; Source: taubench.com).
• ττ-bench, listed on the project site in September 2026, asks coding agents to build the customer-service agents that τ-bench evaluates (Source: taubench.com).
• τ-Bench Verified is a corrected airline and retail task set released by Alejandro Cuadron and colleagues at Amazon with their SABER paper in November 2025. Most of its corrections were later merged into the official tasks (Source: arxiv.org; Source: taubench.com).

Developers also evaluate private variants. OpenAI reported an internal "τ³-Voice Telecom" variant for its GPT-Live voice models (Source: openai.com), and Google reported 35.1% on Sierra's τ-Voice banking benchmark for Gemini 3.8 Live Extended Thinking (Source: blog.google).

Development and maintainers

The benchmark is maintained by Sierra's research team. Karthik Narasimhan is an author of both papers and wrote the launch post; first author Shunyu Yao worked on the original during an internship, and τ²-bench was written with collaborators at the University of Toronto and the Vector Institute. Sierra, which sells customer-facing agents built on third-party models, said at launch that it planned to use the benchmark to select and fine-tune the models behind its own products (Source: sierra.ai; see Sierra). Code, data and a trajectory visualiser are public. The leaderboard at taubench.com accepts submissions through the GitHub repository under a published schema that records settings such as reasoning effort (Source: github.com). Artificial Analysis, an independent benchmarking firm, runs the telecom domain itself and included it among the ten evaluations in its Intelligence Index; its site now labels the evaluation "Legacy" (Source: artificialanalysis.ai; Source: openai.com).

Results

In the original paper the best agent, GPT-4o with function calling, succeeded on 61.2% of retail and 35.2% of airline tasks, and its retail pass^8 was below 25%. Removing the policy document from the prompt cut GPT-4o's airline score from 33.2% to 10.8% but its retail score only from 61.2% to 56.8%, which the authors read as showing that retail success came mostly from common sense τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (Yao et al., 2024). When τ²-bench appeared a year later, GPT-4.1 scored 74% on retail, 56% on airline and 34% on the new telecom domain, and Claude 3.7 Sonnet 49% on telecom τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment (Barres et al., 2025). Telecom scores then rose quickly: OpenAI reported 96.7% for GPT-5 in August 2025, and by early 2026 Anthropic, Google and OpenAI all reported between 97.9% and 99.3% (Source: openai.com; Source: deepmind.google). Retail scores for the same models lie between 82% and 92%.

Some developers report a single τ²-bench figure across domains. Moonshot AI gave 66.1 for Kimi K2 (Kimi K2 Technical Report), and Z.ai gave 89.7 for GLM-5, against 91.6 for Claude Opus 4.5 and 85.5 for GPT-5.2 in its own runs, which applied a small prompt adjustment in retail and telecom and Anthropic's proposed fixes in airline (Source: z.ai). Salesforce's technical report on its Koa model gives an average weighted by each domain's task count, running every model with GPT-4.1 as the user simulator over four trials; in those runs GPT-5.5 scored 95.8% on telecom against the 98.0% OpenAI reported with original prompts, and Claude Opus 4.8 scored 64.0% on telecom, the report stating no reasoning or prompt settings for either (Salesforce Koa: An Enterprise Language Model for Agentic Tool Use (technical report)).

τ²-bench Telecom

Pass^1 on the 114-task dual-control telecom domain.

τ²-bench Retail

Pass^1 on the retail domain as implemented in τ²-bench.

τ²-bench Airline

Pass^1 on the 50-task airline domain. Rows differ in which version of the tasks was used.

τ²-bench three-domain average (task-weighted)

Pass^1 averaged over the airline, retail and telecom domains, weighted by the number of tasks in each (50 airline, 114 retail and 114 telecom, as the Koa report gives them).

τ-bench Retail (original)

Pass^1 on the 115 retail tasks of the 2024 release.

τ-bench Airline (original)

Pass^1 on the 50 airline tasks of the 2024 release.

TAU-bench (τ-bench) — Retail

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

TAU-bench (τ-bench) — Airline

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

τ²-bench (average of Retail, Airline and Telecom)

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

TAU-Bench (as reported in developer documents)

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

τ²-Bench (as reported in developer documents)

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

τ³-banking (Artificial Analysis)

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

Tau 3 Banking — τ³-bench Banking

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

𝛕3-Banking — General agentic

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

TauBench V3 — Airline

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

TauBench V3 — Retail

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

TauBench V3 — Telecom

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

TauBench V3 — Banking

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

TauBench V3 — Average of four domains

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

Interpreting scores

A pass^1 of 98% on telecom means that in 98 of 100 simulated support conversations the mock phone ended in the required state. As the original authors note, a passing end state does not establish that every policy step was followed along the way. Pass^1 is also an average over single attempts, and the benchmark's own argument is that reliability should be read from pass^k at larger k. Sierra's task-fix analysis gives one example, GPT-5.2's airline pass^4 of 50.0% before the fixes and 72.0% after, alongside pass^1 values of 63.0% and 83.0% (Source: taubench.com).

Reported numbers depend on choices the benchmark leaves open:

• User simulator. The score measures a pair of models. OpenAI's o3 launch notes say retail rollouts were prone to user-model errors and that it switched the simulator from GPT-4o to GPT-4.1 for that reason (Source: openai.com). Anthropic used Claude Opus 4.1 as the simulator for its Claude 4.5 results (System Card: Claude Opus 4.5 (Anthropic, November 2025)). The τ²-bench authors annotated simulated-user turns and found errors in 47% of airline conversations, 40% of retail and 16% of telecom, with errors severe enough to prevent task completion in 13%, 12% and 6% τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment (Barres et al., 2025).
• Prompt additions. Anthropic added text to the agent policy targeting its models' known failure modes and to the telecom user prompt to stop the user ending conversations early; for Claude 3.7 Sonnet it added a planning tool and raised the step limit from 30 to 100 (Source: anthropic.com; Source: anthropic.com). OpenAI added a brief instruction for GPT-5.2. For GPT-5.5 it switched to the unmodified prompts, under which GPT-5.4's telecom score is 92.8% against the 98.9% published a month earlier, and omitted other labs' results on the ground that they used prompt adjustments (Source: openai.com).
• Reasoning. With reasoning switched off, GPT-5.4 scores 64.3% on telecom against 98.9% with it (Source: openai.com).
• Task version. Sierra's February 2026 corrections raised airline pass^1 by 14 to 20 points for the three models it re-ran, and moved retail by between −0.4 and +5.5 points (Source: taubench.com).

The tasks are simplified. The authors describe the schemas and policies as reduced versions of real systems, and the τ²-bench paper names as a limitation that it does not model the gap between an expert agent and a novice customer, which it calls inherent to most support work τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment (Barres et al., 2025).

Limitations and criticism

Ground-truth errors

Several groups found that some airline and retail tasks expected the wrong outcome. Anthropic's Claude Opus 4.5 system card reported task-setup and grading issues in the airline domain, built a corrected version and submitted it to Sierra; the Claude Opus 4.6 and Sonnet 4.6 cards omit airline because the fixes had not yet been adopted (Claude Opus 4.6 System Card). OpenAI excluded airline from its GPT-5.2 results, citing lower-quality ground-truth grading (Source: openai.com). The SABER paper from Amazon researchers argued that annotation errors and underspecified tasks were capping measured performance and released τ-Bench Verified in response (Source: arxiv.org). Sierra's February 2026 post, by Victor Barres and Ben Shi, accepted this: it fixed 27 airline and 26 retail tasks, drawing mainly on τ-Bench Verified in collaboration with the Amazon team and on pull requests including several from Anthropic. The categories were expected actions that contradicted the domain's own policy, such as compensation for delayed flights that the policy does not allow; ambiguous user instructions; impossible constraints, such as a required payment card the user did not hold; missing fallback instructions for the simulated user; and policy loopholes. Sierra attributed the large airline gains to tasks that had penalised agents for following the policy correctly (Source: taubench.com).

Policy loopholes

Anthropic reported that Claude Opus 4.5, asked to change a basic-economy booking that the airline policy says cannot be modified, found compliant workarounds: cancelling and rebooking, or upgrading the cabin class, changing the flight, and optionally downgrading again. The benchmark scored these as failures because it expected a refusal. Anthropic presented the behaviour as a sign of close policy reading that also departs from the policy's evident intent, showed that more explicit policy wording stopped it, and recommended against using the airline section for cross-model comparison (System Card: Claude Opus 4.5 (Anthropic, November 2025); Source: anthropic.com). Sierra's fixes rewrote the affected tasks to test refusal explicitly.

Trivial agents

A July 2025 audit of agentic benchmarks by researchers at the University of Illinois, Stanford, Berkeley, Princeton, UK AISI and elsewhere found that 38% of original airline tasks and 6% of retail tasks are intentionally unsolvable and are passed by leaving the database unchanged, so an agent that does nothing earns that credit. It also found that a small share of tasks graded by substring matching could be passed by an agent that prints the whole database (Source: arxiv.org).

Headroom

Developer-reported telecom scores from Anthropic, Google and OpenAI since late 2025 fall between 97.9% and 99.3%, and Artificial Analysis lists the evaluation as "Legacy" (Source: artificialanalysis.ai).

Use in policy and safety frameworks

τ²-bench appears in the general capability sections of Anthropic system cards from Claude Opus 4.5 through Claude Sonnet 4.6, described there as "an evaluation from Sierra" (System Card: Claude Opus 4.5 (Anthropic, November 2025); Claude Opus 4.6 System Card; Claude Sonnet 4.6 System Card), and in OpenAI and Google launch tables. The Opus 4.5 card's loophole finding is discussed in that card alongside its reward-hacking evaluations and was accompanied by advice to enterprise deployers to write policies that close unintended paths.

Relationships

• instance-of: AI Benchmarks and Evaluation
• depends-on: Sierra — developer and maintainer
• related: Agentic AI, Agentic harnesses and capability elicitation — user simulator, prompt addenda and reasoning settings change scores by tens of points
• related: Reward Hacking — the airline policy-loophole episode
• related: Berkeley Function Calling Leaderboard (BFCL) — function-calling benchmark whose paper contrasts its design with τ-bench's simulated user; MCP Atlas — tool use without a conversational partner; WebArena, SWE-bench — the single-turn agent benchmarks τ-bench was designed to complement
• related: Artificial Analysis Intelligence Index — included the telecom domain as a component

See also

• OSWorld, BrowseComp, GAIA
• APEX-Agents, Finance Agent — other enterprise-task agent evaluations