τ-bench
Source-qualified evaluations, report statistics and safety observations.
Claude-3.7-Sonnet
| Metric | Value | Unit |
|---|
| τ-bench · airline · pass^1 | 0.642 | percent |
| τ-bench · airline · pass^2 | 0.589 | percent |
| τ-bench · airline · pass^3 | 0.5539999999999999 | percent |
| τ-bench · airline · pass^4 | 0.521 | percent |
| τ-bench · retail · pass^1 | 0.721 | percent |
| τ-bench · retail · pass^2 | 0.6679999999999999 | percent |
| τ-bench · retail · pass^3 | 0.632 | percent |
| τ-bench · retail · pass^4 | 0.597 | percent |
| τ-bench · telecom · pass^1 | 0.49 | percent |
| τ-bench · telecom · pass^2 | 0.43700000000000006 | percent |
| τ-bench · telecom · pass^3 | 0.401 | percent |
| τ-bench · telecom · pass^4 | 0.37200000000000005 | percent |
Claude Opus 4.1
| Metric | Value | Unit |
|---|
| τ-bench · airline · pass^1 | 0.56 | percent |
| τ-bench · retail · pass^1 | 0.8240000000000001 | percent |
Claude Opus 4
| Metric | Value | Unit |
|---|
| τ-bench · airline · pass^1 | 0.596 | percent |
| τ-bench · retail · pass^1 | 0.8140000000000001 | percent |
Claude-Sonnet-4.5
| Metric | Value | Unit |
|---|
| τ-bench · airline · pass^1 | 0.7 | percent |
| τ-bench · retail · pass^1 | 0.862 | percent |
| τ-bench · telecom · pass^1 | 0.98 | percent |
Claude Sonnet 4
| Metric | Value | Unit |
|---|
| τ-bench · airline · pass^1 | 0.6 | percent |
| τ-bench · retail · pass^1 | 0.805 | percent |
DeepSeek-V3.2
| Metric | Value | Unit |
|---|
| τ-bench · airline · pass^1 | 0.638 | percent |
| τ-bench · retail · pass^1 | 0.8109999999999999 | percent |
| τ-bench · telecom · pass^1 | 0.9620000000000001 | percent |
Gemini 3.0 Pro
| Metric | Value | Unit |
|---|
| τ-bench · airline · pass^1 | 0.73 | percent |
| τ-bench · retail · pass^1 | 0.853 | percent |
| τ-bench · telecom · pass^1 | 0.98 | percent |
GPT-4.1-mini
| Metric | Value | Unit |
|---|
| τ-bench · airline · pass^1 | 0.48700000000000004 | percent |
| τ-bench · airline · pass^2 | 0.389 | percent |
| τ-bench · airline · pass^3 | 0.321 | percent |
| τ-bench · airline · pass^4 | 0.278 | percent |
| τ-bench · retail · pass^1 | 0.614 | percent |
| τ-bench · retail · pass^2 | 0.498 | percent |
| τ-bench · retail · pass^3 | 0.423 | percent |
| τ-bench · retail · pass^4 | 0.369 | percent |
| τ-bench · telecom · pass^1 | 0.489 | percent |
| τ-bench · telecom · pass^2 | 0.387 | percent |
| τ-bench · telecom · pass^3 | 0.312 | percent |
| τ-bench · telecom · pass^4 | 0.261 | percent |
GPT-4.1
| Metric | Value | Unit |
|---|
| τ-bench · airline · pass^1 | 0.56 | percent |
| τ-bench · airline · pass^2 | 0.478 | percent |
| τ-bench · airline · pass^3 | 0.424 | percent |
| τ-bench · airline · pass^4 | 0.381 | percent |
| τ-bench · retail · pass^1 | 0.74 | percent |
| τ-bench · retail · pass^2 | 0.642 | percent |
| τ-bench · retail · pass^3 | 0.581 | percent |
| τ-bench · retail · pass^4 | 0.523 | percent |
| τ-bench · telecom · pass^1 | 0.34 | percent |
| τ-bench · telecom · pass^2 | 0.276 | percent |
| τ-bench · telecom · pass^3 | 0.23800000000000002 | percent |
| τ-bench · telecom · pass^4 | 0.204 | percent |
GPT-5
| Metric | Value | Unit |
|---|
| τ-bench · airline · pass^1 | 0.625 | percent |
| τ-bench · airline · pass^2 | 0.5533333333333332 | percent |
| τ-bench · airline · pass^3 | 0.51 | percent |
| τ-bench · airline · pass^4 | 0.48 | percent |
| τ-bench · retail · pass^1 | 0.8157894736842105 | percent |
| τ-bench · retail · pass^2 | 0.7178362573099416 | percent |
| τ-bench · retail · pass^3 | 0.6469298245614036 | percent |
| τ-bench · retail · pass^4 | 0.5877192982456141 | percent |
| τ-bench · telecom · pass^1 | 0.9583333333333335 | percent |
| τ-bench · telecom · pass^2 | 0.9195906432748537 | percent |
| τ-bench · telecom · pass^3 | 0.8837719298245613 | percent |
| τ-bench · telecom · pass^4 | 0.8508771929824561 | percent |
Kimi-k2
| Metric | Value | Unit |
|---|
| τ-bench · airline · pass^1 | 0.565 | percent |
| τ-bench · retail · pass^1 | 0.706 | percent |
| τ-bench · telecom · pass^1 | 0.6579999999999999 | percent |
o3
| Metric | Value | Unit |
|---|
| τ-bench · airline · pass^1 | 0.52 | percent |
| τ-bench · retail · pass^1 | 0.7390000000000001 | percent |
o4-mini
| Metric | Value | Unit |
|---|
| τ-bench · airline · pass^1 | 0.521 | percent |
| τ-bench · airline · pass^2 | 0.442 | percent |
| τ-bench · airline · pass^3 | 0.389 | percent |
| τ-bench · airline · pass^4 | 0.34700000000000003 | percent |
| τ-bench · retail · pass^1 | 0.6829999999999999 | percent |
| τ-bench · retail · pass^2 | 0.5870000000000001 | percent |
| τ-bench · retail · pass^3 | 0.521 | percent |
| τ-bench · retail · pass^4 | 0.46799999999999997 | percent |
| τ-bench · telecom · pass^1 | 0.502 | percent |
| τ-bench · telecom · pass^2 | 0.42100000000000004 | percent |
| τ-bench · telecom · pass^3 | 0.368 | percent |
| τ-bench · telecom · pass^4 | 0.324 | percent |
Qwen3-Max
| Metric | Value | Unit |
|---|
| τ-bench · airline · pass^1 | 0.595 | percent |
| τ-bench · airline · pass^2 | 0.55 | percent |
| τ-bench · airline · pass^4 | 0.46 | percent |
| τ-bench · retail · pass^1 | 0.722 | percent |
| τ-bench · retail · pass^2 | 0.675 | percent |
| τ-bench · retail · pass^4 | 0.57 | percent |
| τ-bench · telecom · pass^1 | 0.8420000000000001 | percent |
| τ-bench · telecom · pass^2 | 0.7759999999999999 | percent |
| τ-bench · telecom · pass^4 | 0.614 | percent |
Qwen3-Max-Thinking
| Metric | Value | Unit |
|---|
| τ-bench · airline · pass^1 | 0.69 | percent |
| τ-bench · airline · pass^2 | 0.6533333333333333 | percent |
| τ-bench · airline · pass^3 | 0.635 | percent |
| τ-bench · airline · pass^4 | 0.62 | percent |
| τ-bench · retail · pass^1 | 0.7938596491228069 | percent |
| τ-bench · retail · pass^2 | 0.6944444444444445 | percent |
| τ-bench · retail · pass^3 | 0.6271929824561403 | percent |
| τ-bench · retail · pass^4 | 0.5789473684210527 | percent |
| τ-bench · telecom · pass^1 | 0.9824561403508771 | percent |
| τ-bench · telecom · pass^2 | 0.9663742690058479 | percent |
| τ-bench · telecom · pass^3 | 0.9517543859649122 | percent |
| τ-bench · telecom · pass^4 | 0.9385964912280701 | percent |
Nemotron-Orchestrator-8B
| Metric | Value | Unit |
|---|
| τ-bench · airline · pass^1 | 0.56 | percent |
| τ-bench · retail · pass^1 | 0.8420000000000001 | percent |
| τ-bench · telecom · pass^1 | 0.8859999999999999 | percent |
Claude Fable 5
| Metric | Value | Unit |
|---|
| τ-bench · banking_knowledge · pass^1 | 0.3969072164948454 | percent |
| τ-bench · banking_knowledge · pass^2 | 0.32817869415807566 | percent |
| τ-bench · banking_knowledge · pass^3 | 0.3015463917525773 | percent |
| τ-bench · banking_knowledge · pass^4 | 0.28865979381443296 | percent |
Claude Opus 4.5
| Metric | Value | Unit |
|---|
| τ-bench · airline · pass^1 | 0.84 | percent |
| τ-bench · airline · pass^2 | 0.7767000000000001 | percent |
| τ-bench · airline · pass^3 | 0.735 | percent |
| τ-bench · airline · pass^4 | 0.7 | percent |
| τ-bench · banking_knowledge · pass^1 | 0.24739999999999998 | percent |
| τ-bench · banking_knowledge · pass^2 | 0.1718 | percent |
| τ-bench · banking_knowledge · pass^3 | 0.1366 | percent |
| τ-bench · banking_knowledge · pass^4 | 0.1134 | percent |
| τ-bench · retail · pass^1 | 0.7961 | percent |
| τ-bench · retail · pass^2 | 0.674 | percent |
| τ-bench · retail · pass^3 | 0.5877 | percent |
| τ-bench · retail · pass^4 | 0.5175 | percent |
| τ-bench · telecom · pass^1 | 0.9231999999999999 | percent |
| τ-bench · telecom · pass^2 | 0.8611 | percent |
| τ-bench · telecom · pass^3 | 0.8136 | percent |
| τ-bench · telecom · pass^4 | 0.7807 | percent |