τ-bench
Source-qualified evaluations, report statistics and safety observations.
Grok 4.1 fast
| Metric | Value | Unit |
|---|
| τ-bench · banking_knowledge · pass^1 | 0.13140000000000002 | percent |
| τ-bench · banking_knowledge · pass^2 | 0.07730000000000001 | percent |
| τ-bench · banking_knowledge · pass^3 | 0.0567 | percent |
| τ-bench · banking_knowledge · pass^4 | 0.051500000000000004 | percent |
Grok 4.2
| Metric | Value | Unit |
|---|
| τ-bench · banking_knowledge · pass^1 | 0.1804 | percent |
| τ-bench · banking_knowledge · pass^2 | 0.12369999999999999 | percent |
| τ-bench · banking_knowledge · pass^3 | 0.1005 | percent |
| τ-bench · banking_knowledge · pass^4 | 0.0825 | percent |
Grok 4.5
| Metric | Value | Unit |
|---|
| τ-bench · banking_knowledge · pass^1 | 0.4793814432989691 | percent |
| τ-bench · banking_knowledge · pass^2 | 0.39518900343642616 | percent |
| τ-bench · banking_knowledge · pass^3 | 0.3530927835051546 | percent |
| τ-bench · banking_knowledge · pass^4 | 0.31958762886597936 | percent |
Grok 4 fast
| Metric | Value | Unit |
|---|
| τ-bench · banking_knowledge · pass^1 | 0.1572 | percent |
| τ-bench · banking_knowledge · pass^2 | 0.0842 | percent |
| τ-bench · banking_knowledge · pass^3 | 0.0567 | percent |
| τ-bench · banking_knowledge · pass^4 | 0.0412 | percent |
Inkling
| Metric | Value | Unit |
|---|
| τ-bench · banking_knowledge · pass^1 | 0.25 | percent |
| τ-bench · banking_knowledge · pass^2 | 0.16666666666666663 | percent |
| τ-bench · banking_knowledge · pass^3 | 0.13402061855670103 | percent |
| τ-bench · banking_knowledge · pass^4 | 0.1134020618556701 | percent |
Kimi K3
| Metric | Value | Unit |
|---|
| τ-bench · banking_knowledge · pass^1 | 0.3711340206185567 | percent |
| τ-bench · banking_knowledge · pass^2 | 0.25257731958762886 | percent |
| τ-bench · banking_knowledge · pass^3 | 0.20103092783505155 | percent |
| τ-bench · banking_knowledge · pass^4 | 0.17525773195876287 | percent |
Muse Spark 1.1
| Metric | Value | Unit |
|---|
| τ-bench · banking_knowledge · pass^1 | 0.40463917525773196 | percent |
| τ-bench · banking_knowledge · pass^2 | 0.2955326460481099 | percent |
| τ-bench · banking_knowledge · pass^3 | 0.23969072164948454 | percent |
| τ-bench · banking_knowledge · pass^4 | 0.20618556701030927 | percent |
Qwen3.5-397B-A17B
| Metric | Value | Unit |
|---|
| τ-bench · airline · pass^1 | 0.815 | percent |
| τ-bench · airline · pass^2 | 0.7567 | percent |
| τ-bench · airline · pass^3 | 0.715 | percent |
| τ-bench · airline · pass^4 | 0.68 | percent |
| τ-bench · banking_knowledge · pass^1 | 0.09789999999999999 | percent |
| τ-bench · banking_knowledge · pass^2 | 0.0636 | percent |
| τ-bench · banking_knowledge · pass^3 | 0.0541 | percent |
| τ-bench · banking_knowledge · pass^4 | 0.051500000000000004 | percent |
| τ-bench · retail · pass^1 | 0.8443 | percent |
| τ-bench · retail · pass^2 | 0.7442 | percent |
| τ-bench · retail · pass^3 | 0.6645 | percent |
| τ-bench · retail · pass^4 | 0.5965 | percent |
| τ-bench · telecom · pass^1 | 0.9781 | percent |
| τ-bench · telecom · pass^2 | 0.9576 | percent |
| τ-bench · telecom · pass^3 | 0.9386 | percent |
| τ-bench · telecom · pass^4 | 0.9211 | percent |
Qwen 3.8 Max
| Metric | Value | Unit |
|---|
| τ-bench · banking_knowledge · pass^1 | 0.5515463917525774 | percent |
| τ-bench · banking_knowledge · pass^2 | 0.4518900343642611 | percent |
| τ-bench · banking_knowledge · pass^3 | 0.3969072164948454 | percent |
| τ-bench · banking_knowledge · pass^4 | 0.35051546391752575 | percent |
RAFT-30B-A3B
| Metric | Value | Unit |
|---|
| τ-bench · retail · pass^1 | 0.8245614035087719 | percent |
| τ-bench · retail · pass^2 | 0.7295321637426901 | percent |
| τ-bench · retail · pass^3 | 0.6622807017543859 | percent |
| τ-bench · retail · pass^4 | 0.6140350877192983 | percent |
Ava Zero
| Metric | Value | Unit |
|---|
| τ-bench · airline · pass^1 | 0.86 | percent |
| τ-bench · retail · pass^1 | 0.7543859649122807 | percent |
| τ-bench · telecom · pass^1 | 0.7982456140350878 | percent |
gemini-3.1-flash-live-preview-thinking-high
| Metric | Value | Unit |
|---|
| τ-bench · banking_knowledge · pass^1 | 0.1134020618556701 | percent |
gemini-3.1-flash-live-preview-thinking-high
| Metric | Value | Unit |
|---|
| τ-bench · airline · pass^1 | 0.64 | percent |
| τ-bench · retail · pass^1 | 0.45614035087719296 | percent |
| τ-bench · telecom · pass^1 | 0.21929824561403508 | percent |
gemini-3.1-flash-live-preview-thinking-minimal
| Metric | Value | Unit |
|---|
| τ-bench · airline · pass^1 | 0.42 | percent |
| τ-bench · retail · pass^1 | 0.2631578947368421 | percent |
| τ-bench · telecom · pass^1 | 0.17543859649122806 | percent |
gemini-live-2.5-flash-native-audio
| Metric | Value | Unit |
|---|
| τ-bench · airline · pass^1 | 0.3 | percent |
| τ-bench · retail · pass^1 | 0.298 | percent |
| τ-bench · telecom · pass^1 | 0.175 | percent |
gpt-live-1
| Metric | Value | Unit |
|---|
| τ-bench · banking_knowledge · pass^1 | 0.3711340206185567 | percent |
gpt-live-1
| Metric | Value | Unit |
|---|
| τ-bench · airline · pass^1 | 0.82 | percent |
| τ-bench · retail · pass^1 | 0.7894736842105263 | percent |
| τ-bench · telecom · pass^1 | 0.8421052631578947 | percent |
gpt-live-1
| Metric | Value | Unit |
|---|
| τ-bench · airline · pass^1 | 0.78 | percent |
| τ-bench · banking_knowledge · pass^1 | 0.31958762886597936 | percent |
| τ-bench · retail · pass^1 | 0.8421052631578947 | percent |
| τ-bench · telecom · pass^1 | 0.9649122807017544 | percent |