Voice agents asked to take down a caller's name, phone number or email address get it exactly right far less often than text agents given the same job. In τ-Elicitation, a benchmark released by Sierra and posted to arXiv on 11 September 2026, a matched text agent passed all 200 tasks. Four voice agent configurations passed between 14% and 41% of tasks when each task had to succeed in all three test conditions: regular audio, heavy background noise, and a caller who interrupts and restarts.
Sierra has since renamed the benchmark τ-Entity. It joins the τ-bench family, a set of open benchmarks for AI customer service agents. Read together, the family measures two gaps that a single headline score does not show: between what an agent can do once and what it does every time, and between text and voice.
How the benchmarks work
The original τ-bench, from researchers at Sierra and Princeton and published at ICLR 2025, places an AI agent at a simulated retail or airline support desk. A language model plays the customer. The agent gets API tools and a written policy, such as rules on changing a booking. The benchmark scores each task by comparing the database at the end of the conversation with the expected outcome, so a polite conversation that issues the wrong refund still fails.
τ²-bench, presented at ICML 2026, added a telecom domain where the customer also has tools and must carry out steps the agent asks for. The authors report significant performance drops when agents move from acting alone to guiding a user. τ-Voice, also at ICML 2026, runs 278 tasks from the text benchmarks over real-time audio, with identical tools, policies and scoring.
Sierra sells AI customer service agents, so it is grading a field it competes in. The code is public and the τ-bench, τ²-bench and τ-Voice papers passed peer review, but the results still come from a vendor's lab.
Why pass^k matters
A pass^1 score is the share of tasks an agent completes on a single attempt. τ-bench introduced pass^k, the probability that an agent succeeds on all k attempts at the same task. A contact centre handles the same request many times a day, so consistency is closer to what customers experience than a best run.
The original paper found that GPT-4o succeeded on fewer than half of tasks, and that its pass^8 in the retail domain was below 25%. ServiceNow, which also sells service management software, published a separate voice agent framework, EVA-Bench. Across 12 systems, it found a median gap of 0.44 on its accuracy score between pass@k (at least one success in k trials) and pass^k (success in every trial). A demo shows that an agent can complete a task. Pass^k shows whether it does so every time.
What changes over voice
In τ-Voice, a text agent (GPT-5 with reasoning) completed 85% of tasks. Voice agents completed 31% to 51% with clean audio and 26% to 38% with background noise and varied accents, keeping 30% to 45% of text performance. The authors attribute 79% to 90% of failures to the agent's own behaviour.
Full-Duplex-Bench-v3, a benchmark of tool-using voice agents built from recordings of real people, found a specific weak point. Across the six systems tested, the authors name self-correction, where a speaker changes a detail partway through a request, as one of the most consistent failure modes. GPT-Realtime, the leader on that measure at 0.588, still failed more than 40% of self-correction scenarios. The authors trace this to models committing to a tool parameter before the correction arrives.
Getting the details right
τ-Elicitation tests a routine step in a support call: capturing one piece of information. Its ten entity types include person names, phone numbers, dates, addresses, email addresses and medications. A task passes only if the stored value is an exact match.
Agents verified hard and unfamiliar values more often than easy ones, but repair was the weak step. Of the errors agents verified, only 24% to 37% ended up correct, depending on the model. Across 347 initially wrong captures, agents recovered 10% with no repair step and 35% when they used both read-back and spelling.
A prescribed sequence of spelling, read-back, correction and confirmation raised the three-condition pass rate by 14 to 31 points, at a cost of 21 to 28 seconds per call. That is a direct trade between accuracy and handle time. The authors note the study covers English only, three providers and one simulator, and the paper is a preprint.
What to check before buying
- Benchmark and version. Sierra's original τ-bench repository now warns that its airline and retail tasks are outdated and points to τ³-bench, so scores from different versions do not compare directly.
- Pass^1 or pass^k. A single-attempt score is not a reliability figure, and the number of trials behind a pass^k score matters.
- Text or voice. A text score says little about a phone channel, and voice scores depend on the audio conditions tested.
- Detail capture. How the agent confirms names, account numbers and addresses, and how many seconds that adds per call.
- The buyer's own calls. All of these benchmarks use constructed tasks and test environments, and most use simulated callers. They show where agents fail, not what a given deployment will achieve.
Sources
- Yao, S., Shinn, N., Razavi, P. and Narasimhan, K. "τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains." ICLR 2025. https://arxiv.org/abs/2406.12045
- ICLR 2025 proceedings entry for τ-bench. 2025. https://proceedings.iclr.cc/paper_files/paper/2025/hash/1b126cc38b8638e07bef37e7b2bb72bf-Abstract-Conference.html
- Barres, V., Dong, H., Ray, S., Si, X. and Narasimhan, K. "τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment." 2025. https://arxiv.org/abs/2506.07982
- ICML 2026 listing for τ²-Bench. 2026. https://icml.cc/virtual/2026/oral/71171
- Ray, S., Dhandhania, K., Barres, V. and Narasimhan, K. "τ-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains." 2026. https://arxiv.org/abs/2603.13686
- ICML 2026 listing for τ-Voice. 2026. https://icml.cc/virtual/2026/poster/66590
- Ray, S. and Barres, V. "τ-Elicitation: Benchmarking multi-turn entity extraction in voice agents." 2026. https://arxiv.org/abs/2609.13602
- Sierra. "τ-Entity: Can voice agents get the details right?" 2026. https://sierra.ai/blog/tau-entity-can-voice-agents-get-the-details-right
- Sierra Research. τ-bench repository README. https://github.com/sierra-research/tau-bench
- ServiceNow AI Research. "EVA-Bench: A New End-to-end Framework for Evaluating Voice Agents." 2026. https://arxiv.org/abs/2605.13841
- Lin, G.-T., Chen, C., Chen, Z. and Lee, H. "Full-Duplex-Bench-v3: Benchmarking Tool Use for Full-Duplex Voice Agents Under Real-World Disfluency." 2026. https://arxiv.org/abs/2604.04847
- Sierra. Company website. https://sierra.ai