Qwen: Qwen3 235B A22B Thinking 2507

Qwen3-235B-A22B-Thinking-2507 is a high-performance, open-weight Mixture-of-Experts (MoE) language model optimized for complex reasoning tasks. It activates 22B of its 235B parameters per forward pas…

qwen/qwen3-235b-a22b-thinking-2507

Qwen3-235B-A22B-Thinking-2507 is a high-performance, open-weight Mixture-of-Experts (MoE) language model optimized for complex reasoning tasks. It activates 22B of its 235B parameters per forward pass and natively supports up to 262,144 tokens of context. This "thinking-only" variant enhances structured logical reasoning, mathematics, science, and long-form generation, showing strong benchmark performance across AIME, SuperGPQA, LiveCodeBench, and MMLU-Redux. It enforces a special reasoning mode (</think>) and is designed for high-token outputs (up to 81,920 tokens) in challenging domains.

The model is instruction-tuned and excels at step-by-step reasoning, tool use, agentic workflows, and multilingual tasks. This release represents the most capable open-source variant in the Qwen3-235B series, surpassing many closed models in structured reasoning use cases.

Specifications & capabilities

Input
text
Output
text
Context window · tokens
128000
Knowledge cutoff
2025-06-30
Published model ID
Qwen3-235B-A22B-Thinking-2507
Capabilities
reasoning, structuredOutputs, toolCalling
Description source ↗

Evaluations

1–20 / 75

OpenCompass LLM

83.5%

source id
openCompass
leaderboard name
MMLU-Pro
dimension
general
metric
MMLU-Pro
rank
13
aggregation scope
status
published
direction
higherIsBetter
conditions
suite
OverallTable
catalog
open
snapshot
month
REALTIME
fileName
assets/research-rank/research-data.REALTIME.20260309.json
updateTime
2026-03-09
columnFileName
assets/research-rank/research-column.REALTIME.20260309.json
metric_kind
MMLU-Pro
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
0-data.json
source_revision
117c5effa3066abf39f3e8f61109707b09f0cd95223246c07acd6289f4f7fcd2
metric_population
48
benchmark_evaluation
true
source model id
Qwen3-235B-A22B-Thinking-2507
Original result ↗

OpenCompass LLM

90.9%

source id
openCompass
leaderboard name
AIME2025
dimension
general
metric
AIME2025
rank
14
aggregation scope
status
published
direction
higherIsBetter
conditions
suite
OverallTable
catalog
open
snapshot
month
REALTIME
fileName
assets/research-rank/research-data.REALTIME.20260309.json
updateTime
2026-03-09
columnFileName
assets/research-rank/research-column.REALTIME.20260309.json
metric_kind
AIME2025
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
0-data.json
source_revision
117c5effa3066abf39f3e8f61109707b09f0cd95223246c07acd6289f4f7fcd2
metric_population
48
benchmark_evaluation
true
source model id
Qwen3-235B-A22B-Thinking-2507
Original result ↗

OpenCompass LLM

71.8%

source id
openCompass
leaderboard name
Average
dimension
general
metric
Average
rank
18
aggregation scope
status
published
direction
higherIsBetter
conditions
suite
OverallTable
catalog
open
snapshot
month
REALTIME
fileName
assets/research-rank/research-data.REALTIME.20260309.json
updateTime
2026-03-09
columnFileName
assets/research-rank/research-column.REALTIME.20260309.json
metric_kind
Average
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
0-data.json
source_revision
117c5effa3066abf39f3e8f61109707b09f0cd95223246c07acd6289f4f7fcd2
metric_population
48
benchmark_evaluation
true
source model id
Qwen3-235B-A22B-Thinking-2507
Original result ↗

OpenCompass LLM

18.5%

source id
openCompass
leaderboard name
HLE
dimension
general
metric
HLE
rank
18
aggregation scope
status
published
direction
higherIsBetter
conditions
suite
OverallTable
catalog
open
snapshot
month
REALTIME
fileName
assets/research-rank/research-data.REALTIME.20260309.json
updateTime
2026-03-09
columnFileName
assets/research-rank/research-column.REALTIME.20260309.json
metric_kind
HLE
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
0-data.json
source_revision
117c5effa3066abf39f3e8f61109707b09f0cd95223246c07acd6289f4f7fcd2
metric_population
48
benchmark_evaluation
true
source model id
Qwen3-235B-A22B-Thinking-2507
Original result ↗

OpenCompass LLM

79.8%

source id
openCompass
leaderboard name
GPQA-Diamond
dimension
general
metric
GPQA-Diamond
rank
20
aggregation scope
status
published
direction
higherIsBetter
conditions
suite
OverallTable
catalog
open
snapshot
month
REALTIME
fileName
assets/research-rank/research-data.REALTIME.20260309.json
updateTime
2026-03-09
columnFileName
assets/research-rank/research-column.REALTIME.20260309.json
metric_kind
GPQA-Diamond
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
0-data.json
source_revision
117c5effa3066abf39f3e8f61109707b09f0cd95223246c07acd6289f4f7fcd2
metric_population
48
benchmark_evaluation
true
source model id
Qwen3-235B-A22B-Thinking-2507
Original result ↗

OpenRouter Benchmarks

80.303%

source id
openRouterBenchmarks
leaderboard name
GPQA Diamond · Accuracy · Alibaba
dimension
knowledge
metric
GPQA Diamond · Accuracy
rank
21
sample size
198
aggregation scope
status
published
harness
url
https://github.com/OpenRouterTeam/benchmark-harness
name
OpenRouter benchmark harness
direction
higherIsBetter
run_count
1
conditions
window
newest 90 days; successful visible nonzero runs
dataset
benchmark-leaderboard
benchmark
id
gpqa_diamond
name
GPQA Diamond
record_kind
aggregate
provider_name
Alibaba
attribution
Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
last_run_at
2026-08-22T17:18:01.115Z
license_url
https://creativecommons.org/licenses/by/4.0/
methodology
Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
metric_kind
GPQA Diamond · Accuracy
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
sample_size
198
source_file
benchmark-leaderboard.json
source_page
https://openrouter.ai/benchmarks/gpqa-diamond
measurements
accuracy
0.80303
cost_per_task_usd
0.05439790909090909
output_tokens_per_task
23570.0202020202
latency_seconds_per_task
437.2506313131313
published_at
2026-11-16T01:46:02.653Z
snapshot_date
2026-11-16
source_revision
7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
transformations
TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
metric_population
29
benchmark_evaluation
true
openrouter_permaslug
qwen/qwen3-235b-a22b-thinking-2507
source model id
qwen/qwen3-235b-a22b-thinking-2507
Original result ↗

OpenCompass LLM

70.6%

source id
openCompass
leaderboard name
LiveCodeBenchV6
dimension
general
metric
LiveCodeBenchV6
rank
22
aggregation scope
status
published
direction
higherIsBetter
conditions
suite
OverallTable
catalog
open
snapshot
month
REALTIME
fileName
assets/research-rank/research-data.REALTIME.20260309.json
updateTime
2026-03-09
columnFileName
assets/research-rank/research-column.REALTIME.20260309.json
metric_kind
LiveCodeBenchV6
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
0-data.json
source_revision
117c5effa3066abf39f3e8f61109707b09f0cd95223246c07acd6289f4f7fcd2
metric_population
48
benchmark_evaluation
true
source model id
Qwen3-235B-A22B-Thinking-2507
Original result ↗

OpenRouter Benchmarks

61.333%

source id
openRouterBenchmarks
leaderboard name
tau2-bench Verified Airline · Accuracy · Alibaba
dimension
agentic
metric
tau2-bench Verified Airline · Accuracy
rank
22
sample size
50
aggregation scope
status
published
harness
url
https://github.com/OpenRouterTeam/benchmark-harness
name
OpenRouter benchmark harness
direction
higherIsBetter
run_count
1
conditions
window
newest 90 days; successful visible nonzero runs
dataset
benchmark-leaderboard
benchmark
id
tau_bench_verified_airline
name
tau2-bench Verified Airline
record_kind
aggregate
provider_name
Alibaba
attribution
Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
last_run_at
2026-08-22T16:05:00.942Z
license_url
https://creativecommons.org/licenses/by/4.0/
methodology
Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
metric_kind
tau2-bench Verified Airline · Accuracy
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
sample_size
50
source_file
benchmark-leaderboard.json
source_page
https://openrouter.ai/benchmarks/tau2-bench-airline
measurements
accuracy
0.613333
cost_per_task_usd
0.12290606
output_tokens_per_task
27061.54
latency_seconds_per_task
597.00058
published_at
2026-11-16T01:46:02.653Z
snapshot_date
2026-11-16
source_revision
7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
transformations
TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
metric_population
29
benchmark_evaluation
true
openrouter_permaslug
qwen/qwen3-235b-a22b-thinking-2507
source model id
qwen/qwen3-235b-a22b-thinking-2507
Original result ↗

OpenRouter Benchmarks

74.916%

source id
openRouterBenchmarks
leaderboard name
GPQA Diamond · Accuracy · Venice
dimension
knowledge
metric
GPQA Diamond · Accuracy
rank
29
sample size
198
aggregation scope
status
published
harness
url
https://github.com/OpenRouterTeam/benchmark-harness
name
OpenRouter benchmark harness
direction
higherIsBetter
run_count
1
conditions
window
newest 90 days; successful visible nonzero runs
dataset
benchmark-leaderboard
benchmark
id
gpqa_diamond
name
GPQA Diamond
record_kind
aggregate
provider_name
Venice
attribution
Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
last_run_at
2026-08-22T17:38:59.682Z
license_url
https://creativecommons.org/licenses/by/4.0/
methodology
Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
metric_kind
GPQA Diamond · Accuracy
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
sample_size
198
source_file
benchmark-leaderboard.json
source_page
https://openrouter.ai/benchmarks/gpqa-diamond
measurements
accuracy
0.749158
cost_per_task_usd
0.07988080808080808
output_tokens_per_task
22758.171717171717
latency_seconds_per_task
853.6403434343434
published_at
2026-11-16T01:46:02.653Z
snapshot_date
2026-11-16
source_revision
7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
transformations
TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
metric_population
39
benchmark_evaluation
true
openrouter_permaslug
qwen/qwen3-235b-a22b-thinking-2507
source model id
qwen/qwen3-235b-a22b-thinking-2507
Original result ↗

OpenRouter Benchmarks

63.333%

source id
openRouterBenchmarks
leaderboard name
tau2-bench Verified Airline · Accuracy · Venice
dimension
agentic
metric
tau2-bench Verified Airline · Accuracy
rank
33
sample size
50
aggregation scope
status
published
harness
url
https://github.com/OpenRouterTeam/benchmark-harness
name
OpenRouter benchmark harness
direction
higherIsBetter
run_count
1
conditions
window
newest 90 days; successful visible nonzero runs
dataset
benchmark-leaderboard
benchmark
id
tau_bench_verified_airline
name
tau2-bench Verified Airline
record_kind
aggregate
provider_name
Venice
attribution
Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
last_run_at
2026-08-22T16:17:16.779Z
license_url
https://creativecommons.org/licenses/by/4.0/
methodology
Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
metric_kind
tau2-bench Verified Airline · Accuracy
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
sample_size
50
source_file
benchmark-leaderboard.json
source_page
https://openrouter.ai/benchmarks/tau2-bench-airline
measurements
accuracy
0.633333
cost_per_task_usd
0.21351704000000002
output_tokens_per_task
27870.24
latency_seconds_per_task
877.39536
published_at
2026-11-16T01:46:02.653Z
snapshot_date
2026-11-16
source_revision
7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
transformations
TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
metric_population
38
benchmark_evaluation
true
openrouter_permaslug
qwen/qwen3-235b-a22b-thinking-2507
source model id
qwen/qwen3-235b-a22b-thinking-2507
Original result ↗

OpenCompass LLM

87.8%

source id
openCompass
leaderboard name
IFEval
dimension
general
metric
IFEval
rank
34
aggregation scope
status
published
direction
higherIsBetter
conditions
suite
OverallTable
catalog
open
snapshot
month
REALTIME
fileName
assets/research-rank/research-data.REALTIME.20260309.json
updateTime
2026-03-09
columnFileName
assets/research-rank/research-column.REALTIME.20260309.json
metric_kind
IFEval
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
0-data.json
source_revision
117c5effa3066abf39f3e8f61109707b09f0cd95223246c07acd6289f4f7fcd2
metric_population
48
benchmark_evaluation
true
source model id
Qwen3-235B-A22B-Thinking-2507
Original result ↗

OpenRouter Benchmarks

78.108%

source id
openRouterBenchmarks
leaderboard name
GPQA Diamond · Accuracy · DeepInfra
dimension
knowledge
metric
GPQA Diamond · Accuracy
rank
34
sample size
185
aggregation scope
status
published
harness
url
https://github.com/OpenRouterTeam/benchmark-harness
name
OpenRouter benchmark harness
direction
higherIsBetter
run_count
1
conditions
window
newest 90 days; successful visible nonzero runs
dataset
benchmark-leaderboard
benchmark
id
gpqa_diamond
name
GPQA Diamond
record_kind
aggregate
provider_name
DeepInfra
attribution
Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
last_run_at
2026-08-22T16:06:14.894Z
license_url
https://creativecommons.org/licenses/by/4.0/
methodology
Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
metric_kind
GPQA Diamond · Accuracy
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
sample_size
185
source_file
benchmark-leaderboard.json
source_page
https://openrouter.ai/benchmarks/gpqa-diamond
measurements
accuracy
0.781081
cost_per_task_usd
0.04895784324324324
output_tokens_per_task
21222.45945945946
latency_seconds_per_task
686.6486648648649
published_at
2026-11-16T01:46:02.653Z
snapshot_date
2026-11-16
source_revision
7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
transformations
TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
metric_population
71
benchmark_evaluation
true
openrouter_permaslug
qwen/qwen3-235b-a22b-thinking-2507
source model id
qwen/qwen3-235b-a22b-thinking-2507
Original result ↗

OpenRouter Benchmarks

73.232%

source id
openRouterBenchmarks
leaderboard name
GPQA Diamond · Accuracy · Novita
dimension
knowledge
metric
GPQA Diamond · Accuracy
rank
42
sample size
198
aggregation scope
status
published
harness
url
https://github.com/OpenRouterTeam/benchmark-harness
name
OpenRouter benchmark harness
direction
higherIsBetter
run_count
1
conditions
window
newest 90 days; successful visible nonzero runs
dataset
benchmark-leaderboard
benchmark
id
gpqa_diamond
name
GPQA Diamond
record_kind
aggregate
provider_name
Novita
attribution
Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
last_run_at
2026-08-22T17:45:06.645Z
license_url
https://creativecommons.org/licenses/by/4.0/
methodology
Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
metric_kind
GPQA Diamond · Accuracy
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
sample_size
198
source_file
benchmark-leaderboard.json
source_page
https://openrouter.ai/benchmarks/gpqa-diamond
measurements
accuracy
0.7323230000000001
cost_per_task_usd
0.0787700707070707
output_tokens_per_task
26175.146464646463
latency_seconds_per_task
1116.1990808080807
published_at
2026-11-16T01:46:02.653Z
snapshot_date
2026-11-16
source_revision
7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
transformations
TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
metric_population
59
benchmark_evaluation
true
openrouter_permaslug
qwen/qwen3-235b-a22b-thinking-2507
source model id
qwen/qwen3-235b-a22b-thinking-2507
Original result ↗

OpenRouter Benchmarks

58%

source id
openRouterBenchmarks
leaderboard name
tau2-bench Verified Airline · Accuracy · Novita
dimension
agentic
metric
tau2-bench Verified Airline · Accuracy
rank
42
sample size
50
aggregation scope
status
published
harness
url
https://github.com/OpenRouterTeam/benchmark-harness
name
OpenRouter benchmark harness
direction
higherIsBetter
run_count
1
conditions
window
newest 90 days; successful visible nonzero runs
dataset
benchmark-leaderboard
benchmark
id
tau_bench_verified_airline
name
tau2-bench Verified Airline
record_kind
aggregate
provider_name
Novita
attribution
Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
last_run_at
2026-08-22T16:20:42.106Z
license_url
https://creativecommons.org/licenses/by/4.0/
methodology
Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
metric_kind
tau2-bench Verified Airline · Accuracy
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
sample_size
50
source_file
benchmark-leaderboard.json
source_page
https://openrouter.ai/benchmarks/tau2-bench-airline
measurements
accuracy
0.58
cost_per_task_usd
0.14568854
output_tokens_per_task
23072.82
latency_seconds_per_task
545.4362199999999
published_at
2026-11-16T01:46:02.653Z
snapshot_date
2026-11-16
source_revision
7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
transformations
TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
metric_population
52
benchmark_evaluation
true
openrouter_permaslug
qwen/qwen3-235b-a22b-thinking-2507
source model id
qwen/qwen3-235b-a22b-thinking-2507
Original result ↗

LMArena

1,457.016

source id
lmarena
leaderboard name
LMArena Text Industry Medicine And Healthcare
dimension
general
metric
Arena rating
rank
58
sample size
507
evaluated at
2026-11-09T00:00:00.000Z
source model id
Qwen3-235B-A22B-Thinking (2507)
confidence lower
1431.0574002319586
confidence upper
1482.9744811187854
Original result ↗

LMArena

1,462.712

source id
lmarena
leaderboard name
LMArena Text Expert
dimension
general
metric
Arena rating
rank
71
sample size
423
evaluated at
2026-11-09T00:00:00.000Z
source model id
Qwen3-235B-A22B-Thinking (2507)
confidence lower
1434.5929494343623
confidence upper
1490.830548897392
Original result ↗

LMArena

1,481.685

source id
lmarena
leaderboard name
LMArena Text Chinese
dimension
general
metric
Arena rating
rank
75
sample size
535
evaluated at
2026-11-09T00:00:00.000Z
source model id
Qwen3-235B-A22B-Thinking (2507)
confidence lower
1455.6489429789663
confidence upper
1507.720736889068
Original result ↗

OpenRouter Benchmarks

78.62%

source id
openRouterBenchmarks
leaderboard name
GPQA Diamond · Accuracy · auto-routing
dimension
knowledge
metric
GPQA Diamond · Accuracy
rank
83
sample size
198
aggregation scope
status
published
harness
url
https://github.com/OpenRouterTeam/benchmark-harness
name
OpenRouter benchmark harness
direction
higherIsBetter
run_count
1
conditions
window
newest 90 days; successful visible nonzero runs
dataset
benchmark-leaderboard
benchmark
id
gpqa_diamond
name
GPQA Diamond
record_kind
aggregate
provider_name
auto-routing
attribution
Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
last_run_at
2026-08-22T17:09:41.940Z
license_url
https://creativecommons.org/licenses/by/4.0/
methodology
Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
metric_kind
GPQA Diamond · Accuracy
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
sample_size
198
source_file
benchmark-leaderboard.json
source_page
https://openrouter.ai/benchmarks/gpqa-diamond
measurements
accuracy
0.786195
cost_per_task_usd
0.057962348484848485
output_tokens_per_task
23055.60101010101
latency_seconds_per_task
638.7363484848485
published_at
2026-11-16T01:46:02.653Z
snapshot_date
2026-11-16
source_revision
7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
transformations
TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
metric_population
132
benchmark_evaluation
true
openrouter_permaslug
qwen/qwen3-235b-a22b-thinking-2507
source model id
qwen/qwen3-235b-a22b-thinking-2507
Original result ↗

OpenRouter Benchmarks

62.667%

source id
openRouterBenchmarks
leaderboard name
tau2-bench Verified Airline · Accuracy · auto-routing
dimension
agentic
metric
tau2-bench Verified Airline · Accuracy
rank
95
sample size
50
aggregation scope
status
published
harness
url
https://github.com/OpenRouterTeam/benchmark-harness
name
OpenRouter benchmark harness
direction
higherIsBetter
run_count
1
conditions
window
newest 90 days; successful visible nonzero runs
dataset
benchmark-leaderboard
benchmark
id
tau_bench_verified_airline
name
tau2-bench Verified Airline
record_kind
aggregate
provider_name
auto-routing
attribution
Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
last_run_at
2026-08-22T16:01:40.197Z
license_url
https://creativecommons.org/licenses/by/4.0/
methodology
Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
metric_kind
tau2-bench Verified Airline · Accuracy
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
sample_size
50
source_file
benchmark-leaderboard.json
source_page
https://openrouter.ai/benchmarks/tau2-bench-airline
measurements
accuracy
0.626667
cost_per_task_usd
0.12019484
output_tokens_per_task
26775.66
latency_seconds_per_task
635.85502
published_at
2026-11-16T01:46:02.653Z
snapshot_date
2026-11-16
source_revision
7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
transformations
TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
metric_population
128
benchmark_evaluation
true
openrouter_permaslug
qwen/qwen3-235b-a22b-thinking-2507
source model id
qwen/qwen3-235b-a22b-thinking-2507
Original result ↗

LMArena

1,467.964

source id
lmarena
leaderboard name
LMArena Text Style Control Expert
dimension
style
metric
Arena rating
rank
96
sample size
423
evaluated at
2026-11-09T00:00:00.000Z
source model id
Qwen3-235B-A22B-Thinking (2507)
confidence lower
1439.7561467544688
confidence upper
1496.1713650720333
Original result ↗

Published prices

1–4 / 4

Provider specifications & access

1–8 / 8

Qwen/Qwen3-235B-A22B-Thinking-2507 ↗

source model id
HuggingFace/qwen-qwen3-235b-a22b-thinking-2507@1
hosting
foundry-managed-compute
platforms
  • Microsoft Foundry
license
apache-2.0
architecture
input_modalities
  • text
output_modalities
  • text
capabilities

    Qwen/Qwen3-235B-A22B-Thinking-2507 ↗

    source model id
    HuggingFace/qwen-qwen3-235b-a22b-thinking-2507@2
    hosting
    foundry-managed-compute
    platforms
    • Microsoft Foundry
    license
    apache-2.0
    architecture
    input_modalities
    • text
    output_modalities
    • text
    capabilities

      Qwen/Qwen3-235B-A22B-Thinking-2507 ↗

      source model id
      HuggingFace/qwen-qwen3-235b-a22b-thinking-2507@3
      hosting
      foundry-managed-compute
      platforms
      • Microsoft Foundry
      license
      apache-2.0
      architecture
      input_modalities
      • text
      output_modalities
      • text
      capabilities

        Qwen/Qwen3-235B-A22B-Thinking-2507 ↗

        source model id
        HuggingFace/qwen-qwen3-235b-a22b-thinking-2507@4
        hosting
        foundry-managed-compute
        platforms
        • Microsoft Foundry
        license
        apache-2.0
        architecture
        input_modalities
        • text
        output_modalities
        • text
        capabilities

          Qwen/Qwen3-235B-A22B-Thinking-2507 ↗

          source model id
          HuggingFace/qwen-qwen3-235b-a22b-thinking-2507@5
          hosting
          foundry-managed-compute
          platforms
          • Microsoft Foundry
          license
          apache-2.0
          architecture
          input_modalities
          • text
          output_modalities
          • text
          capabilities

            Qwen/Qwen3-235B-A22B-Thinking-2507 ↗

            source model id
            publishers/hf-qwen/models/qwen3-235b-a22b-thinking-2507@001
            hosting
            vertex-self-deployed
            platforms
            • Google Cloud Vertex AI
            architecture
            input_modalities
            • text
            output_modalities
            • text
            capabilities

              Qwen3-235B-A22B-Thinking-2507 ↗

              source model id
              publishers/qwen/models/qwen3@qwen3-235b-a22b-thinking-2507
              hosting
              vertex-self-deployed
              platforms
              • Google Cloud Vertex AI
              architecture
              input_modalities
              • text
              output_modalities
              • text
              capabilities

                Qwen3-235B-A22B-Thinking-2507 ↗

                source model id
                Qwen/Qwen3-235B-A22B-Thinking-2507
                hosting
                siliconflow-hosted
                platforms
                • SiliconFlow
                max output tokens
                262144
                architecture
                input_modalities
                • text
                output_modalities
                • text
                capabilities
                • structuredOutputs
                • toolCalling

                Safety assessments

                1–1 / 1

                Foundation Model Transparency Index 2025 ↗

                26 / 100

                source id
                stanfordFMTI
                canonical model id
                qwen/qwen3-235b-a22b-thinking-2507
                taxonomy
                transparency
                hazard category
                transparency
                hazard name
                Organization transparency
                policy area
                Alibaba
                assessment type
                organization-level disclosure
                metric
                disclosure_score
                result label
                26 / 100
                sample size
                100
                updated at
                2026-11-09T18:08:05.227Z

                Source specification observations

                1–5 / 5

                openRouter ↗

                field
                context_length
                value
                128000
                normalized value
                128000
                scope
                source-model
                source id
                openRouter
                confidence
                1

                siliconFlow ↗

                field
                context_length
                value
                262K
                scope
                source-model
                source id
                siliconFlow
                confidence
                1

                openRouter ↗

                field
                knowledge_cutoff
                reported field
                knowledge_cutoff
                value
                2025-06-30
                normalized value
                2025-06-30
                precision
                day
                scope
                source-model
                source id
                openRouter
                confidence
                1

                openRouter ↗

                field
                max_output_tokens
                value
                16384
                normalized value
                16384
                scope
                source-model
                source id
                openRouter
                confidence
                1

                siliconFlow ↗

                field
                max_output_tokens
                value
                262K
                scope
                source-model
                source id
                siliconFlow
                confidence
                1

                Original sources

                1–20 / 25

                SiliconFlow · Model page ↗

                Hugging Face · Model repository ↗

                DeepInfra · Model page ↗

                Google Cloud · Model page ↗

                Google Cloud · Model page ↗

                Microsoft Foundry · Model page ↗

                Microsoft Foundry · Model page ↗

                Microsoft Foundry · Model page ↗

                Microsoft Foundry · Model page ↗

                Microsoft Foundry · Model page ↗

                OpenRouter · Model page ↗

                SiliconFlow · Playground ↗

                Google Cloud · Model documentation ↗

                Microsoft Foundry · Model documentation ↗

                SiliconFlow · API documentation ↗

                DeepInfra · Catalog ↗

                GitHub · Catalog ↗

                GitHub · Catalog ↗

                Google Cloud · Catalog ↗

                Hugging Face · Catalog ↗

                Source links

                1–12 / 12

                Explore more models

                Models

                Loading
                EVIDENCE / COMPARISON

                Benchmarks

                Explore every benchmark

                Loading

                These charts contain model-linked results. The charts above also include unlinked source profiles. Compare scores within each evaluation. Model quality, usage, transparency and system performance measure different things.

                Install TensorTrace