τ-bench
Source-qualified evaluations, report statistics and safety observations.
Claude Opus 4.6
| Metric | Value | Unit |
|---|
| τ-bench · banking_knowledge · pass^1 | 0.2732 | percent |
| τ-bench · banking_knowledge · pass^2 | 0.1873 | percent |
| τ-bench · banking_knowledge · pass^3 | 0.1418 | percent |
| τ-bench · banking_knowledge · pass^4 | 0.1134 | percent |
Claude Opus 4.7
| Metric | Value | Unit |
|---|
| τ-bench · banking_knowledge · pass^1 | 0.4020618556701031 | percent |
| τ-bench · banking_knowledge · pass^2 | 0.3109965635738832 | percent |
| τ-bench · banking_knowledge · pass^3 | 0.2706185567010309 | percent |
| τ-bench · banking_knowledge · pass^4 | 0.24742268041237114 | percent |
Claude Opus 4.8
| Metric | Value | Unit |
|---|
| τ-bench · banking_knowledge · pass^1 | 0.3969072164948454 | percent |
| τ-bench · banking_knowledge · pass^2 | 0.31271477663230246 | percent |
| τ-bench · banking_knowledge · pass^3 | 0.26288659793814434 | percent |
| τ-bench · banking_knowledge · pass^4 | 0.2268041237113402 | percent |
Claude Opus 5
| Metric | Value | Unit |
|---|
| τ-bench · banking_knowledge · pass^1 | 0.48711340206185566 | percent |
| τ-bench · banking_knowledge · pass^2 | 0.3917525773195877 | percent |
| τ-bench · banking_knowledge · pass^3 | 0.34536082474226804 | percent |
| τ-bench · banking_knowledge · pass^4 | 0.31958762886597936 | percent |
Claude Sonnet 4.5
| Metric | Value | Unit |
|---|
| τ-bench · airline · pass^1 | 0.72 | percent |
| τ-bench · airline · pass^2 | 0.62 | percent |
| τ-bench · airline · pass^3 | 0.545 | percent |
| τ-bench · airline · pass^4 | 0.48 | percent |
| τ-bench · banking_knowledge · pass^1 | 0.2526 | percent |
| τ-bench · banking_knowledge · pass^2 | 0.15810000000000002 | percent |
| τ-bench · banking_knowledge · pass^3 | 0.1263 | percent |
| τ-bench · banking_knowledge · pass^4 | 0.10310000000000001 | percent |
| τ-bench · retail · pass^1 | 0.7237 | percent |
| τ-bench · retail · pass^2 | 0.5746 | percent |
| τ-bench · retail · pass^3 | 0.47369999999999995 | percent |
| τ-bench · retail · pass^4 | 0.3947 | percent |
| τ-bench · telecom · pass^1 | 0.8487 | percent |
| τ-bench · telecom · pass^2 | 0.7544 | percent |
| τ-bench · telecom · pass^3 | 0.6886 | percent |
| τ-bench · telecom · pass^4 | 0.6404000000000001 | percent |
PrincipledMachines Cortex 0.1
| Metric | Value | Unit |
|---|
| τ-bench · banking_knowledge · pass^1 | 0.7319587628865979 | percent |
Distyl ButtonAgent
| Metric | Value | Unit |
|---|
| τ-bench · banking_knowledge · pass^1 | 0.3119 | percent |
| τ-bench · banking_knowledge · pass^2 | 0.2147 | percent |
| τ-bench · banking_knowledge · pass^3 | 0.16190000000000002 | percent |
| τ-bench · banking_knowledge · pass^4 | 0.134 | percent |
Gemini 2.5 Pro
| Metric | Value | Unit |
|---|
| τ-bench · banking_knowledge · pass^1 | 0.1366 | percent |
| τ-bench · banking_knowledge · pass^2 | 0.0601 | percent |
| τ-bench · banking_knowledge · pass^3 | 0.0258 | percent |
| τ-bench · banking_knowledge · pass^4 | 0.0103 | percent |
Gemini 3.1 Pro Preview
| Metric | Value | Unit |
|---|
| τ-bench · banking_knowledge · pass^1 | 0.26030000000000003 | percent |
| τ-bench · banking_knowledge · pass^2 | 0.1512 | percent |
| τ-bench · banking_knowledge · pass^3 | 0.1108 | percent |
| τ-bench · banking_knowledge · pass^4 | 0.0928 | percent |
Gemini 3 Flash
| Metric | Value | Unit |
|---|
| τ-bench · airline · pass^1 | 0.825 | percent |
| τ-bench · airline · pass^2 | 0.7633 | percent |
| τ-bench · airline · pass^3 | 0.72 | percent |
| τ-bench · airline · pass^4 | 0.68 | percent |
| τ-bench · banking_knowledge · pass^1 | 0.2732 | percent |
| τ-bench · banking_knowledge · pass^2 | 0.15460000000000002 | percent |
| τ-bench · banking_knowledge · pass^3 | 0.10310000000000001 | percent |
| τ-bench · banking_knowledge · pass^4 | 0.0722 | percent |
| τ-bench · retail · pass^1 | 0.7675 | percent |
| τ-bench · retail · pass^2 | 0.6594 | percent |
| τ-bench · retail · pass^3 | 0.5789 | percent |
| τ-bench · retail · pass^4 | 0.5175 | percent |
| τ-bench · telecom · pass^1 | 0.9123 | percent |
| τ-bench · telecom · pass^2 | 0.8348 | percent |
| τ-bench · telecom · pass^3 | 0.7654000000000001 | percent |
| τ-bench · telecom · pass^4 | 0.7018000000000001 | percent |
Gemini 3 Pro
| Metric | Value | Unit |
|---|
| τ-bench · airline · pass^1 | 0.805 | percent |
| τ-bench · airline · pass^2 | 0.7467 | percent |
| τ-bench · airline · pass^3 | 0.7 | percent |
| τ-bench · airline · pass^4 | 0.66 | percent |
| τ-bench · banking_knowledge · pass^1 | 0.1804 | percent |
| τ-bench · banking_knowledge · pass^2 | 0.0928 | percent |
| τ-bench · banking_knowledge · pass^3 | 0.0567 | percent |
| τ-bench · banking_knowledge · pass^4 | 0.0412 | percent |
| τ-bench · retail · pass^1 | 0.7587999999999999 | percent |
| τ-bench · retail · pass^2 | 0.6345000000000001 | percent |
| τ-bench · retail · pass^3 | 0.5439 | percent |
| τ-bench · retail · pass^4 | 0.47369999999999995 | percent |
| τ-bench · telecom · pass^1 | 0.9101 | percent |
| τ-bench · telecom · pass^2 | 0.845 | percent |
| τ-bench · telecom · pass^3 | 0.7917000000000001 | percent |
| τ-bench · telecom · pass^4 | 0.7456 | percent |
GLM-5.2
| Metric | Value | Unit |
|---|
| τ-bench · banking_knowledge · pass^1 | 0.3711340206185567 | percent |
| τ-bench · banking_knowledge · pass^2 | 0.25257731958762886 | percent |
| τ-bench · banking_knowledge · pass^3 | 0.18298969072164947 | percent |
| τ-bench · banking_knowledge · pass^4 | 0.13402061855670103 | percent |
GLM-5
| Metric | Value | Unit |
|---|
| τ-bench · airline · pass^1 | 0.825 | percent |
| τ-bench · airline · pass^2 | 0.76 | percent |
| τ-bench · airline · pass^3 | 0.72 | percent |
| τ-bench · airline · pass^4 | 0.7 | percent |
| τ-bench · banking_knowledge · pass^1 | 0.09789999999999999 | percent |
| τ-bench · banking_knowledge · pass^2 | 0.0722 | percent |
| τ-bench · banking_knowledge · pass^3 | 0.0567 | percent |
| τ-bench · banking_knowledge · pass^4 | 0.030899999999999997 | percent |
| τ-bench · retail · pass^1 | 0.7368000000000001 | percent |
| τ-bench · retail · pass^2 | 0.6038 | percent |
| τ-bench · retail · pass^3 | 0.511 | percent |
| τ-bench · retail · pass^4 | 0.4386 | percent |
| τ-bench · telecom · pass^1 | 0.8684000000000001 | percent |
| τ-bench · telecom · pass^2 | 0.7631999999999999 | percent |
| τ-bench · telecom · pass^3 | 0.682 | percent |
| τ-bench · telecom · pass^4 | 0.6228 | percent |
GPT-5.2
| Metric | Value | Unit |
|---|
| τ-bench · airline · pass^1 | 0.525 | percent |
| τ-bench · airline · pass^2 | 0.3566666666666667 | percent |
| τ-bench · airline · pass^3 | 0.27 | percent |
| τ-bench · airline · pass^4 | 0.22 | percent |
| τ-bench · banking_knowledge · pass^1 | 0.1263 | percent |
| τ-bench · banking_knowledge · pass^2 | 0.0722 | percent |
| τ-bench · banking_knowledge · pass^3 | 0.051500000000000004 | percent |
| τ-bench · banking_knowledge · pass^4 | 0.0412 | percent |
| τ-bench · retail · pass^1 | 0.75 | percent |
| τ-bench · retail · pass^2 | 0.6213450292397662 | percent |
| τ-bench · retail · pass^3 | 0.5307017543859649 | percent |
| τ-bench · retail · pass^4 | 0.45614035087719296 | percent |
| τ-bench · telecom · pass^1 | 0.5723684210526315 | percent |
| τ-bench · telecom · pass^2 | 0.4283625730994152 | percent |
| τ-bench · telecom · pass^3 | 0.35526315789473684 | percent |
| τ-bench · telecom · pass^4 | 0.30701754385964913 | percent |
GPT-5.2
| Metric | Value | Unit |
|---|
| τ-bench · airline · pass^1 | 0.83 | percent |
| τ-bench · airline · pass^2 | 0.7833 | percent |
| τ-bench · airline · pass^3 | 0.75 | percent |
| τ-bench · airline · pass^4 | 0.72 | percent |
| τ-bench · banking_knowledge · pass^1 | 0.3222 | percent |
| τ-bench · banking_knowledge · pass^2 | 0.23879999999999998 | percent |
| τ-bench · banking_knowledge · pass^3 | 0.20620000000000002 | percent |
| τ-bench · banking_knowledge · pass^4 | 0.1856 | percent |
| τ-bench · retail · pass^1 | 0.8158 | percent |
| τ-bench · retail · pass^2 | 0.6959000000000001 | percent |
| τ-bench · retail · pass^3 | 0.5987 | percent |
| τ-bench · retail · pass^4 | 0.5175 | percent |
| τ-bench · telecom · pass^1 | 0.8969 | percent |
| τ-bench · telecom · pass^2 | 0.8245999999999999 | percent |
| τ-bench · telecom · pass^3 | 0.7675 | percent |
| τ-bench · telecom · pass^4 | 0.7193 | percent |
GPT-5.4
| Metric | Value | Unit |
|---|
| τ-bench · banking_knowledge · pass^1 | 0.3943 | percent |
| τ-bench · banking_knowledge · pass^2 | 0.2955 | percent |
| τ-bench · banking_knowledge · pass^3 | 0.24480000000000002 | percent |
| τ-bench · banking_knowledge · pass^4 | 0.2165 | percent |
GPT-5.5
| Metric | Value | Unit |
|---|
| τ-bench · banking_knowledge · pass^1 | 0.44587628865979384 | percent |
| τ-bench · banking_knowledge · pass^2 | 0.37628865979381454 | percent |
| τ-bench · banking_knowledge · pass^3 | 0.3324742268041237 | percent |
| τ-bench · banking_knowledge · pass^4 | 0.29896907216494845 | percent |
GPT-5.6-sol
| Metric | Value | Unit |
|---|
| τ-bench · banking_knowledge · pass^1 | 0.46907216494845355 | percent |
| τ-bench · banking_knowledge · pass^2 | 0.3728522336769759 | percent |
| τ-bench · banking_knowledge · pass^3 | 0.31958762886597936 | percent |
| τ-bench · banking_knowledge · pass^4 | 0.27835051546391754 | percent |