τ-bench
Source-qualified evaluations, report statistics and safety observations.
gpt-realtime-1.0
| Metric | Value | Unit |
|---|
| τ-bench · airline · pass^1 | 0.36 | percent |
| τ-bench · retail · pass^1 | 0.35964912280701755 | percent |
| τ-bench · telecom · pass^1 | 0.19298245614035087 | percent |
gpt-realtime-1.5
| Metric | Value | Unit |
|---|
| τ-bench · airline · pass^1 | 0.4 | percent |
| τ-bench · retail · pass^1 | 0.447 | percent |
| τ-bench · telecom · pass^1 | 0.21100000000000002 | percent |
gpt-realtime-2
| Metric | Value | Unit |
|---|
| τ-bench · banking_knowledge · pass^1 | 0.10309278350515463 | percent |
gpt-realtime-2
| Metric | Value | Unit |
|---|
| τ-bench · airline · pass^1 | 0.56 | percent |
| τ-bench · retail · pass^1 | 0.39473684210526316 | percent |
| τ-bench · telecom · pass^1 | 0.20175438596491227 | percent |
gpt-realtime-2
| Metric | Value | Unit |
|---|
| τ-bench · airline · pass^1 | 0.66 | percent |
| τ-bench · retail · pass^1 | 0.5 | percent |
| τ-bench · telecom · pass^1 | 0.37719298245614036 | percent |
gpt-realtime-2
| Metric | Value | Unit |
|---|
| τ-bench · airline · pass^1 | 0.58 | percent |
| τ-bench · retail · pass^1 | 0.47368421052631576 | percent |
| τ-bench · telecom · pass^1 | 0.21929824561403508 | percent |
Grok Voice 3.0
| Metric | Value | Unit |
|---|
| τ-bench · airline · pass^1 | 0.8 | percent |
| τ-bench · retail · pass^1 | 0.7631578947368421 | percent |
| τ-bench · telecom · pass^1 | 0.8245614035087719 | percent |
grok-voice-think-fast-1.0 + tool-mentor
| Metric | Value | Unit |
|---|
| τ-bench · airline · pass^1 | 0.7 | percent |
| τ-bench · retail · pass^1 | 0.7631578947368421 | percent |
| τ-bench · telecom · pass^1 | 0.7807017543859649 | percent |
grok-voice-think-fast-1.0
| Metric | Value | Unit |
|---|
| τ-bench · airline · pass^1 | 0.66 | percent |
| τ-bench · retail · pass^1 | 0.6228070175438597 | percent |
| τ-bench · telecom · pass^1 | 0.7368421052631579 | percent |
grok-voice-think-fast-2.0
| Metric | Value | Unit |
|---|
| τ-bench · airline · pass^1 | 0.56 | percent |
| τ-bench · retail · pass^1 | 0.5964912280701754 | percent |
| τ-bench · telecom · pass^1 | 0.7192982456140351 | percent |
Cascaded baseline
| Metric | Value | Unit |
|---|
| τ-bench · airline · pass^1 | 0.48 | percent |
| τ-bench · retail · pass^1 | 0.2894736842105263 | percent |
| τ-bench · telecom · pass^1 | 0.16666666666666663 | percent |
phonic-1
| Metric | Value | Unit |
|---|
| τ-bench · airline · pass^1 | 0.8 | percent |
| τ-bench · retail · pass^1 | 0.7631578947368421 | percent |
| τ-bench · telecom · pass^1 | 0.9473684210526315 | percent |
phonic-1
| Metric | Value | Unit |
|---|
| τ-bench · airline · pass^1 | 0.9 | percent |
| τ-bench · retail · pass^1 | 0.7982456140350878 | percent |
| τ-bench · telecom · pass^1 | 0.9824561403508771 | percent |
Pine Voice 1.0
| Metric | Value | Unit |
|---|
| τ-bench · airline · pass^1 | 0.92 | percent |
| τ-bench · retail · pass^1 | 0.7876106194690266 | percent |
| τ-bench · telecom · pass^1 | 0.9122807017543859 | percent |
Pine Voice Preview
| Metric | Value | Unit |
|---|
| τ-bench · airline · pass^1 | 0.8 | percent |
| τ-bench · retail · pass^1 | 0.8508771929824561 | percent |
| τ-bench · telecom · pass^1 | 0.7543859649122807 | percent |
Pine Voice Preview
| Metric | Value | Unit |
|---|
| τ-bench · airline · pass^1 | 0.7 | percent |
| τ-bench · retail · pass^1 | 0.7017543859649122 | percent |
| τ-bench · telecom · pass^1 | 0.8596491228070176 | percent |
qwen3.5-omni-plus-realtime
| Metric | Value | Unit |
|---|
| τ-bench · airline · pass^1 | 0.54 | percent |
| τ-bench · retail · pass^1 | 0.4649122807017544 | percent |
| τ-bench · telecom · pass^1 | 0.6052631578947368 | percent |
Salesforce Launa v1
| Metric | Value | Unit |
|---|
| τ-bench · airline · pass^1 | 0.66 | percent |
| τ-bench · retail · pass^1 | 0.7192982456140351 | percent |
| τ-bench · telecom · pass^1 | 0.8070175438596491 | percent |