This is the paper introducing τ²-bench, the successor to τ-bench, by Victor Barres and Honghua Dong (equal contribution; Dong's work done during an internship), Soham Ray, Xujie Si and Karthik Narasimhan, affiliated to Sierra, the University of Toronto and the Vector Institute. It was posted to arXiv as 2506.07982 on 9 June 2025. Code and data are released at github.com/sierra-research/tau2-bench.
Summary
The authors observe that existing conversational-agent benchmarks, including τ-bench, are "single-control" environments: only the agent can act on the world, while the simulated user supplies preferences and goals. Technical support differs, because the user must take actions such as restarting a phone. τ²-bench introduces a "dual-control" setting in which the simulated user also holds tools, formalised as a decentralised partially observable Markov decision process (Dec-POMDP) with a shared world state, separate agent and user databases, per-player action and observation spaces, and a reward computed from the final state. The design challenge the authors name is keeping the "complexity asymmetry" between agent and user: user tools return only human-readable outputs, user planning is limited to reacting to agent requests, and user behaviour is tied to observable state.
Domains and task generation
The paper re-verifies the retail and airline domains (115 and 50 tasks) and adds telecom, with 5 plans, 9 lines and 4 customers in the agent database, 6 write and 7 read agent tools, and 15 write and 15 read user tools. Telecom was built in five stages: an LLM drafted a product requirements document, function implementations, a mock database and unit tests for the agent's CRM system, refined by hand until tests passed; the same for the user's mock phone device; programmatic task creation; LLM-generated agent policies; and joint manual refinement. Tasks are composed from atomic subtasks, each defined by initialisation, solution and assertion functions (for example, set_airplane_mode(True) as initialisation, toggle_airplane_mode() as solution, assert_service_status("connected") as assertion). Subtasks are grouped so that mutually exclusive alternatives share a group; a composite task takes at most one from each group, and correctness is verified automatically by applying initialisation then solution functions and checking assertions, and by confirming the task is unsolved until all solution functions run. Fifteen subtask groups across three intents (service_issue, mobile_data_issue, mms_issue) yield 2,285 tasks, of which 114 were subsampled for balance. Each telecom task is randomly assigned a persona: none, easy (domain-familiar) or hard (low technical knowledge). Evaluation criteria can include a database check, status assertions, natural-language assertions, a communication check and action matching; telecom uses assertion functions only.
Results
Four models — gpt-4.1-mini, gpt-4.1, o4-mini and claude-3-7-sonnet — were run with gpt-4.1 as the user simulator, four trials per task at temperature 0. One trial of all domains cost about $40 (gpt-4.1 agent $0.086 and user $0.059 per task). GPT-4.1's pass^1 fell from 74% on retail and 56% on airline to 34% on telecom; gpt-4.1-mini, o4-mini and claude-3.7-sonnet reached "around 50%" on telecom, with claude-3.7-sonnet at 49%, on par with its airline score but declining faster as k increased.
An ablation in telecom compared the default dual-control mode with a no-user mode, in which the agent receives a ticket and controls every tool, and an oracle-plan mode, in which the agent is given the required tool-call sequence. Moving from no-user to default reduced pass^1 by 18 points for gpt-4.1 and 25 points for o4-mini, which the authors read as isolating communication and coordination, rather than reasoning, as the bottleneck. Oracle plans helped o4-mini more than gpt-4.1. A workflow-based policy document was also tested. Performance fell with the number of required actions, approaching zero above seven actions in default mode, and with the number of distinct subtasks; the harder intents (mobile_data_issue and mms_issue, which may require resolving a service issue first) drove most failures. Agents did better with the easy persona than the hard one, and no-persona tasks scored on par with or below hard-persona tasks.
User simulator reliability
Two annotators reviewed gpt-4.1 user-simulator traces against the guidelines, instructions, tools and conversational consistency, classifying errors as task-critical or benign. Error rates were 47% on airline (13% critical, 100 conversations), 40% on retail (12% critical, 50 conversations) and 16% on telecom (6% critical, 50 conversations). The authors attribute the improvement to the domain's tools and observable state constraining user behaviour in place of natural-language instruction. The table caption states no critical errors were reported for telecom while the table and text give 3 (6%).
Limitations stated by the authors
The tool-constrained user simulation has not been applied back to retail and airline; extending domains still relies on human experts; and the benchmark "does not explicitly model the expert-novice gap inherent to most customer support tasks."
Key claims
- Dual-control telecom domain formalised as a Dec-POMDP; 114 tasks subsampled from 2,285 programmatically generated ones; 15 subtask groups over three intents; agent tools 6 write and 7 read, user tools 15 write and 15 read.
- GPT-4.1 pass^1: retail 74%, airline 56%, telecom 34%; claude-3.7-sonnet telecom 49%; other models around 50% on telecom.
- No-user to dual-control drop: 18 points (gpt-4.1), 25 points (o4-mini); performance near zero above seven required actions.
- User-simulator error rates: airline 47% (13% critical), retail 40% (12%), telecom 16% (6%).
- One trial of all domains costs about $40 with gpt-4.1 as agent and user.
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering (arxiv.org); the abstract page (arxiv.org) records a single version submitted 9 June 2025.
Relationships
- supports: τ-bench — the primary text for the τ²-bench telecom domain, its modes and launch results.
- supersedes: τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (Yao et al., 2024) — as the current edition of the benchmark; the 2024 paper remains the reference for the original domains and the pass^k metric.
- supports: Sierra
- instance-of: AI Benchmarks and Evaluation
- related: Agentic AI