Qwen: Qwen3 235B A22B Instruct 2507

Qwen3-235B-A22B-Instruct-2507 is a multilingual, instruction-tuned mixture-of-experts language model based on the Qwen3-235B architecture, with 22B active parameters per forward pass. It is optimized…

qwen/qwen3-235b-a22b-07-25

Qwen3-235B-A22B-Instruct-2507 is a multilingual, instruction-tuned mixture-of-experts language model based on the Qwen3-235B architecture, with 22B active parameters per forward pass. It is optimized for general-purpose text generation, including instruction following, logical reasoning, math, code, and tool usage. The model supports a native 262K context length and does not implement "thinking mode" (<think> blocks).

Compared to its base variant, this version delivers significant gains in knowledge coverage, long-context reasoning, coding benchmarks, and alignment with open-ended tasks. It is particularly strong on multilingual understanding, math reasoning (e.g., AIME, HMMT), and alignment evaluations like Arena-Hard and WritingBench.

Specifications & capabilities

Input
text
Output
text
Context window · tokens
262144
Knowledge cutoff
2025-06-30
Published model ID
Qwen3-235B-A22B-Instruct-2507
Capabilities
promptCaching, structuredOutputs, toolCalling
Description source ↗

Evaluations

41–60 / 160

LLM Stats · GitHub archive

0.573

source id
llmStatsArchive
leaderboard name
Aider-Polyglot · Normalized (reported)
dimension
general
metric
Aider-Polyglot · Normalized (reported)
rank
3
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
Aider-Polyglot
language
en
modality
text
verified
false
max_score
1
categories
  • general
  • code
created_at
2025-09-05T00:00:00.000000+00:00
updated_at
2025-09-30T00:00:00.000000+00:00
description
A coding benchmark that evaluates LLMs on 225 challenging Exercism programming exercises across C++, Go, Java, JavaScript, Python, and Rust. Models receive two attempts to solve each problem, with test error feedback provided after the first attempt if it fails. The benchmark measures both initial problem-solving ability and capacity to edit code based on error feedback, providing an end-to-end evaluation of code generation and editing capabilities across multiple programming languages.
benchmark_id
aider-polyglot
multilingual
false
implementation_link
https://github.com/Aider-AI/polyglot-benchmark
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
Aider-Polyglot · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/qwen/models/qwen3-235b-a22b-instruct-2507/benchmarks.json
archive_kind
llmStats
evidence_url
https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
qwen/qwen3-235b-a22b-instruct-2507
metric_population
5
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
qwen3-235b-a22b-instruct-2507
Original result ↗

LLM Stats · GitHub archive

0.573

source id
llmStatsArchive
leaderboard name
Aider-Polyglot · Raw
dimension
general
metric
Aider-Polyglot · Raw
rank
3
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
Aider-Polyglot
language
en
modality
text
verified
false
max_score
1
categories
  • general
  • code
created_at
2025-09-05T00:00:00.000000+00:00
updated_at
2025-09-30T00:00:00.000000+00:00
description
A coding benchmark that evaluates LLMs on 225 challenging Exercism programming exercises across C++, Go, Java, JavaScript, Python, and Rust. Models receive two attempts to solve each problem, with test error feedback provided after the first attempt if it fails. The benchmark measures both initial problem-solving ability and capacity to edit code based on error feedback, providing an end-to-end evaluation of code generation and editing capabilities across multiple programming languages.
benchmark_id
aider-polyglot
multilingual
false
implementation_link
https://github.com/Aider-AI/polyglot-benchmark
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
Aider-Polyglot · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/qwen/models/qwen3-235b-a22b-instruct-2507/benchmarks.json
archive_kind
llmStats
evidence_url
https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
qwen/qwen3-235b-a22b-instruct-2507
metric_population
5
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
qwen3-235b-a22b-instruct-2507
Original result ↗

OpenRouter Benchmarks

40.667%

source id
openRouterBenchmarks
leaderboard name
tau2-bench Verified Airline · Accuracy · PrimeIntellect
dimension
agentic
metric
tau2-bench Verified Airline · Accuracy
rank
15
sample size
50
aggregation scope
status
published
harness
url
https://github.com/OpenRouterTeam/benchmark-harness
name
OpenRouter benchmark harness
direction
higherIsBetter
run_count
1
conditions
window
newest 90 days; successful visible nonzero runs
dataset
benchmark-leaderboard
benchmark
id
tau_bench_verified_airline
name
tau2-bench Verified Airline
record_kind
aggregate
provider_name
PrimeIntellect
attribution
Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
last_run_at
2026-10-04T03:00:16.402Z
license_url
https://creativecommons.org/licenses/by/4.0/
methodology
Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
metric_kind
tau2-bench Verified Airline · Accuracy
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
sample_size
50
source_file
benchmark-leaderboard.json
source_page
https://openrouter.ai/benchmarks/tau2-bench-airline
measurements
accuracy
0.406667
cost_per_task_usd
0.06754938
output_tokens_per_task
3789.7
latency_seconds_per_task
114.52978
published_at
2026-11-16T01:46:02.653Z
snapshot_date
2026-11-16
source_revision
7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
transformations
TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
metric_population
21
benchmark_evaluation
true
openrouter_permaslug
qwen/qwen3-235b-a22b-07-25
source model id
qwen/qwen3-235b-a22b-07-25
Original result ↗

OpenRouter Benchmarks

75.253%

source id
openRouterBenchmarks
leaderboard name
GPQA Diamond · Accuracy · PrimeIntellect
dimension
knowledge
metric
GPQA Diamond · Accuracy
rank
16
sample size
198
aggregation scope
status
published
harness
url
https://github.com/OpenRouterTeam/benchmark-harness
name
OpenRouter benchmark harness
direction
higherIsBetter
run_count
1
conditions
window
newest 90 days; successful visible nonzero runs
dataset
benchmark-leaderboard
benchmark
id
gpqa_diamond
name
GPQA Diamond
record_kind
aggregate
provider_name
PrimeIntellect
attribution
Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
last_run_at
2026-10-04T03:06:11.710Z
license_url
https://creativecommons.org/licenses/by/4.0/
methodology
Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
metric_kind
GPQA Diamond · Accuracy
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
sample_size
198
source_file
benchmark-leaderboard.json
source_page
https://openrouter.ai/benchmarks/gpqa-diamond
measurements
accuracy
0.7525250000000001
cost_per_task_usd
0.008515843434343435
output_tokens_per_task
8312.732323232323
latency_seconds_per_task
110.88425757575757
published_at
2026-11-16T01:46:02.653Z
snapshot_date
2026-11-16
source_revision
7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
transformations
TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
metric_population
22
benchmark_evaluation
true
openrouter_permaslug
qwen/qwen3-235b-a22b-07-25
source model id
qwen/qwen3-235b-a22b-07-25
Original result ↗

OpenRouter Benchmarks

74.141%

source id
openRouterBenchmarks
leaderboard name
GPQA Diamond · Accuracy · Nebius
dimension
knowledge
metric
GPQA Diamond · Accuracy
rank
20
sample size
990
aggregation scope
status
published
harness
url
https://github.com/OpenRouterTeam/benchmark-harness
name
OpenRouter benchmark harness
direction
higherIsBetter
run_count
5
conditions
window
newest 90 days; successful visible nonzero runs
dataset
benchmark-leaderboard
benchmark
id
gpqa_diamond
name
GPQA Diamond
record_kind
aggregate
provider_name
Nebius
attribution
Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
last_run_at
2026-10-17T16:06:15.944Z
license_url
https://creativecommons.org/licenses/by/4.0/
methodology
Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
metric_kind
GPQA Diamond · Accuracy
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
sample_size
990
source_file
benchmark-leaderboard.json
source_page
https://openrouter.ai/benchmarks/gpqa-diamond
measurements
accuracy
0.7414141999999999
accuracy_std_dev
0.0027351667883330276
cost_per_task_usd
0.00532230808080808
output_tokens_per_task
8599.7
latency_seconds_per_task
145.93918484848484
published_at
2026-11-16T01:46:02.653Z
snapshot_date
2026-11-16
source_revision
7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
transformations
TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
metric_population
25
benchmark_evaluation
true
openrouter_permaslug
qwen/qwen3-235b-a22b-07-25
source model id
qwen/qwen3-235b-a22b-07-25
Original result ↗

OpenRouter Benchmarks

43.4%

source id
openRouterBenchmarks
leaderboard name
tau2-bench Verified Airline · Accuracy · Nebius
dimension
agentic
metric
tau2-bench Verified Airline · Accuracy
rank
20
sample size
250
aggregation scope
status
published
harness
url
https://github.com/OpenRouterTeam/benchmark-harness
name
OpenRouter benchmark harness
direction
higherIsBetter
run_count
5
conditions
window
newest 90 days; successful visible nonzero runs
dataset
benchmark-leaderboard
benchmark
id
tau_bench_verified_airline
name
tau2-bench Verified Airline
record_kind
aggregate
provider_name
Nebius
attribution
Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
last_run_at
2026-10-17T19:11:29.194Z
license_url
https://creativecommons.org/licenses/by/4.0/
methodology
Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
metric_kind
tau2-bench Verified Airline · Accuracy
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
sample_size
250
source_file
benchmark-leaderboard.json
source_page
https://openrouter.ai/benchmarks/tau2-bench-airline
measurements
accuracy
0.434
accuracy_std_dev
0.03129784515266187
cost_per_task_usd
0.053281564000000003
output_tokens_per_task
3661.612
latency_seconds_per_task
160.868104
published_at
2026-11-16T01:46:02.653Z
snapshot_date
2026-11-16
source_revision
7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
transformations
TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
metric_population
23
benchmark_evaluation
true
openrouter_permaslug
qwen/qwen3-235b-a22b-07-25
source model id
qwen/qwen3-235b-a22b-07-25
Original result ↗

OpenRouter Benchmarks

74.074%

source id
openRouterBenchmarks
leaderboard name
GPQA Diamond · Accuracy · Alibaba
dimension
knowledge
metric
GPQA Diamond · Accuracy
rank
23
sample size
594
aggregation scope
status
published
harness
url
https://github.com/OpenRouterTeam/benchmark-harness
name
OpenRouter benchmark harness
direction
higherIsBetter
run_count
3
conditions
window
newest 90 days; successful visible nonzero runs
dataset
benchmark-leaderboard
benchmark
id
gpqa_diamond
name
GPQA Diamond
record_kind
aggregate
provider_name
Alibaba
attribution
Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
last_run_at
2026-10-03T16:59:36.480Z
license_url
https://creativecommons.org/licenses/by/4.0/
methodology
Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
metric_kind
GPQA Diamond · Accuracy
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
sample_size
594
source_file
benchmark-leaderboard.json
source_page
https://openrouter.ai/benchmarks/gpqa-diamond
measurements
accuracy
0.7407406666666667
accuracy_std_dev
0.008584264726165454
cost_per_task_usd
0.004939484848484848
output_tokens_per_task
8056.897306397306
latency_seconds_per_task
108.5982037037037
published_at
2026-11-16T01:46:02.653Z
snapshot_date
2026-11-16
source_revision
7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
transformations
TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
metric_population
29
benchmark_evaluation
true
openrouter_permaslug
qwen/qwen3-235b-a22b-07-25
source model id
qwen/qwen3-235b-a22b-07-25
Original result ↗

OpenRouter Benchmarks

41.5%

source id
openRouterBenchmarks
leaderboard name
tau2-bench Verified Airline · Accuracy · StreamLake
dimension
agentic
metric
tau2-bench Verified Airline · Accuracy
rank
23
sample size
200
aggregation scope
status
published
harness
url
https://github.com/OpenRouterTeam/benchmark-harness
name
OpenRouter benchmark harness
direction
higherIsBetter
run_count
4
conditions
window
newest 90 days; successful visible nonzero runs
dataset
benchmark-leaderboard
benchmark
id
tau_bench_verified_airline
name
tau2-bench Verified Airline
record_kind
aggregate
provider_name
StreamLake
attribution
Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
last_run_at
2026-10-03T16:19:54.632Z
license_url
https://creativecommons.org/licenses/by/4.0/
methodology
Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
metric_kind
tau2-bench Verified Airline · Accuracy
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
sample_size
200
source_file
benchmark-leaderboard.json
source_page
https://openrouter.ai/benchmarks/tau2-bench-airline
measurements
accuracy
0.415
accuracy_std_dev
0.02763845951749121
cost_per_task_usd
0.051504745
output_tokens_per_task
3278.695
latency_seconds_per_task
126.44671000000001
published_at
2026-11-16T01:46:02.653Z
snapshot_date
2026-11-16
source_revision
7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
transformations
TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
metric_population
25
benchmark_evaluation
true
openrouter_permaslug
qwen/qwen3-235b-a22b-07-25
source model id
qwen/qwen3-235b-a22b-07-25
Original result ↗

OpenRouter Benchmarks

36.833%

source id
openRouterBenchmarks
leaderboard name
tau2-bench Verified Airline · Accuracy · GMICloud
dimension
agentic
metric
tau2-bench Verified Airline · Accuracy
rank
23
sample size
200
aggregation scope
status
published
harness
url
https://github.com/OpenRouterTeam/benchmark-harness
name
OpenRouter benchmark harness
direction
higherIsBetter
run_count
4
conditions
window
newest 90 days; successful visible nonzero runs
dataset
benchmark-leaderboard
benchmark
id
tau_bench_verified_airline
name
tau2-bench Verified Airline
record_kind
aggregate
provider_name
GMICloud
attribution
Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
last_run_at
2026-10-03T16:12:29.879Z
license_url
https://creativecommons.org/licenses/by/4.0/
methodology
Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
metric_kind
tau2-bench Verified Airline · Accuracy
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
sample_size
200
source_file
benchmark-leaderboard.json
source_page
https://openrouter.ai/benchmarks/tau2-bench-airline
measurements
accuracy
0.36833325
accuracy_std_dev
0.17853565236861657
cost_per_task_usd
0.015193219999999999
output_tokens_per_task
2917.505
latency_seconds_per_task
127.68209
published_at
2026-11-16T01:46:02.653Z
snapshot_date
2026-11-16
source_revision
7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
transformations
TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
metric_population
23
benchmark_evaluation
true
openrouter_permaslug
qwen/qwen3-235b-a22b-07-25
source model id
qwen/qwen3-235b-a22b-07-25
Original result ↗

OpenRouter Benchmarks

44.588%

source id
openRouterBenchmarks
leaderboard name
tau2-bench Verified Airline · Accuracy · Parasail
dimension
agentic
metric
tau2-bench Verified Airline · Accuracy
rank
24
sample size
348
aggregation scope
status
published
harness
url
https://github.com/OpenRouterTeam/benchmark-harness
name
OpenRouter benchmark harness
direction
higherIsBetter
run_count
7
conditions
window
newest 90 days; successful visible nonzero runs
dataset
benchmark-leaderboard
benchmark
id
tau_bench_verified_airline
name
tau2-bench Verified Airline
record_kind
aggregate
provider_name
Parasail
attribution
Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
last_run_at
2026-10-31T18:58:15.016Z
license_url
https://creativecommons.org/licenses/by/4.0/
methodology
Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
metric_kind
tau2-bench Verified Airline · Accuracy
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
sample_size
348
source_file
benchmark-leaderboard.json
source_page
https://openrouter.ai/benchmarks/tau2-bench-airline
measurements
accuracy
0.4458811867816092
accuracy_std_dev
0.045214591965110434
cost_per_task_usd
0.015823612068965518
output_tokens_per_task
3370.8706896551726
latency_seconds_per_task
133.6663620689655
published_at
2026-11-16T01:46:02.653Z
snapshot_date
2026-11-16
source_revision
7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
transformations
TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
metric_population
28
benchmark_evaluation
true
openrouter_permaslug
qwen/qwen3-235b-a22b-07-25
source model id
qwen/qwen3-235b-a22b-07-25
Original result ↗

OpenRouter Benchmarks

70.286%

source id
openRouterBenchmarks
leaderboard name
GPQA Diamond · Accuracy · GMICloud
dimension
knowledge
metric
GPQA Diamond · Accuracy
rank
25
sample size
792
aggregation scope
status
published
harness
url
https://github.com/OpenRouterTeam/benchmark-harness
name
OpenRouter benchmark harness
direction
higherIsBetter
run_count
4
conditions
window
newest 90 days; successful visible nonzero runs
dataset
benchmark-leaderboard
benchmark
id
gpqa_diamond
name
GPQA Diamond
record_kind
aggregate
provider_name
GMICloud
attribution
Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
last_run_at
2026-10-03T16:56:36.098Z
license_url
https://creativecommons.org/licenses/by/4.0/
methodology
Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
metric_kind
GPQA Diamond · Accuracy
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
sample_size
792
source_file
benchmark-leaderboard.json
source_page
https://openrouter.ai/benchmarks/gpqa-diamond
measurements
accuracy
0.7028622499999999
accuracy_std_dev
0.08200476941122087
cost_per_task_usd
0.0070032777777777775
output_tokens_per_task
7212.60101010101
latency_seconds_per_task
172.46112373737373
published_at
2026-11-16T01:46:02.653Z
snapshot_date
2026-11-16
source_revision
7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
transformations
TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
metric_population
26
benchmark_evaluation
true
openrouter_permaslug
qwen/qwen3-235b-a22b-07-25
source model id
qwen/qwen3-235b-a22b-07-25
Original result ↗

OpenRouter Benchmarks

73.617%

source id
openRouterBenchmarks
leaderboard name
GPQA Diamond · Accuracy · Parasail
dimension
knowledge
metric
GPQA Diamond · Accuracy
rank
27
sample size
1386
aggregation scope
status
published
harness
url
https://github.com/OpenRouterTeam/benchmark-harness
name
OpenRouter benchmark harness
direction
higherIsBetter
run_count
7
conditions
window
newest 90 days; successful visible nonzero runs
dataset
benchmark-leaderboard
benchmark
id
gpqa_diamond
name
GPQA Diamond
record_kind
aggregate
provider_name
Parasail
attribution
Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
last_run_at
2026-10-31T16:02:14.362Z
license_url
https://creativecommons.org/licenses/by/4.0/
methodology
Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
metric_kind
GPQA Diamond · Accuracy
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
sample_size
1386
source_file
benchmark-leaderboard.json
source_page
https://openrouter.ai/benchmarks/gpqa-diamond
measurements
accuracy
0.7361712857142858
accuracy_std_dev
0.00933947044559173
cost_per_task_usd
0.0070149227994227995
output_tokens_per_task
8674.670995670996
latency_seconds_per_task
250.93870995670997
published_at
2026-11-16T01:46:02.653Z
snapshot_date
2026-11-16
source_revision
7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
transformations
TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
metric_population
36
benchmark_evaluation
true
openrouter_permaslug
qwen/qwen3-235b-a22b-07-25
source model id
qwen/qwen3-235b-a22b-07-25
Original result ↗

OpenRouter Benchmarks

38.667%

source id
openRouterBenchmarks
leaderboard name
tau2-bench Verified Airline · Accuracy · AtlasCloud
dimension
agentic
metric
tau2-bench Verified Airline · Accuracy
rank
27
sample size
100
aggregation scope
status
published
harness
url
https://github.com/OpenRouterTeam/benchmark-harness
name
OpenRouter benchmark harness
direction
higherIsBetter
run_count
2
conditions
window
newest 90 days; successful visible nonzero runs
dataset
benchmark-leaderboard
benchmark
id
tau_bench_verified_airline
name
tau2-bench Verified Airline
record_kind
aggregate
provider_name
AtlasCloud
attribution
Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
last_run_at
2026-08-29T16:12:10.467Z
license_url
https://creativecommons.org/licenses/by/4.0/
methodology
Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
metric_kind
tau2-bench Verified Airline · Accuracy
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
sample_size
100
source_file
benchmark-leaderboard.json
source_page
https://openrouter.ai/benchmarks/tau2-bench-airline
measurements
accuracy
0.386667
accuracy_std_dev
0.06
cost_per_task_usd
0.052157499999999996
output_tokens_per_task
2942.27
latency_seconds_per_task
111.01769999999999
published_at
2026-11-16T01:46:02.653Z
snapshot_date
2026-11-16
source_revision
7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
transformations
TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
metric_population
29
benchmark_evaluation
true
openrouter_permaslug
qwen/qwen3-235b-a22b-07-25
source model id
qwen/qwen3-235b-a22b-07-25
Original result ↗

OpenRouter Benchmarks

35.333%

source id
openRouterBenchmarks
leaderboard name
tau2-bench Verified Airline · Accuracy · Alibaba
dimension
agentic
metric
tau2-bench Verified Airline · Accuracy
rank
27
sample size
150
aggregation scope
status
published
harness
url
https://github.com/OpenRouterTeam/benchmark-harness
name
OpenRouter benchmark harness
direction
higherIsBetter
run_count
3
conditions
window
newest 90 days; successful visible nonzero runs
dataset
benchmark-leaderboard
benchmark
id
tau_bench_verified_airline
name
tau2-bench Verified Airline
record_kind
aggregate
provider_name
Alibaba
attribution
Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
last_run_at
2026-10-03T16:05:51.296Z
license_url
https://creativecommons.org/licenses/by/4.0/
methodology
Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
metric_kind
tau2-bench Verified Airline · Accuracy
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
sample_size
150
source_file
benchmark-leaderboard.json
source_page
https://openrouter.ai/benchmarks/tau2-bench-airline
measurements
accuracy
0.35333333333333333
accuracy_std_dev
0.04455559988997518
cost_per_task_usd
0.06266134666666666
output_tokens_per_task
3931.2533333333336
latency_seconds_per_task
100.24880666666667
published_at
2026-11-16T01:46:02.653Z
snapshot_date
2026-11-16
source_revision
7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
transformations
TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
metric_population
29
benchmark_evaluation
true
openrouter_permaslug
qwen/qwen3-235b-a22b-07-25
source model id
qwen/qwen3-235b-a22b-07-25
Original result ↗

OpenCompass LLM

12.3%

source id
openCompass
leaderboard name
HLE
dimension
general
metric
HLE
rank
27
aggregation scope
status
published
direction
higherIsBetter
conditions
suite
OverallTable
catalog
open
snapshot
month
REALTIME
fileName
assets/research-rank/research-data.REALTIME.20260309.json
updateTime
2026-03-09
columnFileName
assets/research-rank/research-column.REALTIME.20260309.json
metric_kind
HLE
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
0-data.json
source_revision
117c5effa3066abf39f3e8f61109707b09f0cd95223246c07acd6289f4f7fcd2
metric_population
48
benchmark_evaluation
true
source model id
Qwen3-235B-A22B-Instruct-2507
Original result ↗

OpenCompass LLM

75.5%

source id
openCompass
leaderboard name
GPQA-Diamond
dimension
general
metric
GPQA-Diamond
rank
28
aggregation scope
status
published
direction
higherIsBetter
conditions
suite
OverallTable
catalog
open
snapshot
month
REALTIME
fileName
assets/research-rank/research-data.REALTIME.20260309.json
updateTime
2026-03-09
columnFileName
assets/research-rank/research-column.REALTIME.20260309.json
metric_kind
GPQA-Diamond
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
0-data.json
source_revision
117c5effa3066abf39f3e8f61109707b09f0cd95223246c07acd6289f4f7fcd2
metric_population
48
benchmark_evaluation
true
source model id
Qwen3-235B-A22B-Instruct-2507
Original result ↗

OpenRouter Benchmarks

74.537%

source id
openRouterBenchmarks
leaderboard name
GPQA Diamond · Accuracy · StreamLake
dimension
knowledge
metric
GPQA Diamond · Accuracy
rank
28
sample size
792
aggregation scope
status
published
harness
url
https://github.com/OpenRouterTeam/benchmark-harness
name
OpenRouter benchmark harness
direction
higherIsBetter
run_count
4
conditions
window
newest 90 days; successful visible nonzero runs
dataset
benchmark-leaderboard
benchmark
id
gpqa_diamond
name
GPQA Diamond
record_kind
aggregate
provider_name
StreamLake
attribution
Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
last_run_at
2026-10-03T16:58:32.590Z
license_url
https://creativecommons.org/licenses/by/4.0/
methodology
Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
metric_kind
GPQA Diamond · Accuracy
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
sample_size
792
source_file
benchmark-leaderboard.json
source_page
https://openrouter.ai/benchmarks/gpqa-diamond
measurements
accuracy
0.7453702499999999
accuracy_std_dev
0.0095511571910162
cost_per_task_usd
0.006491083333333334
output_tokens_per_task
7540.814393939394
latency_seconds_per_task
157.74153661616162
published_at
2026-11-16T01:46:02.653Z
snapshot_date
2026-11-16
source_revision
7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
transformations
TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
metric_population
30
benchmark_evaluation
true
openrouter_permaslug
qwen/qwen3-235b-a22b-07-25
source model id
qwen/qwen3-235b-a22b-07-25
Original result ↗

LMArena

1,402.695

source id
lmarena
leaderboard name
LMArena Text Factuality Korean
dimension
factuality
metric
Arena rating
rank
29
sample size
484
evaluated at
2026-11-09T00:00:00.000Z
source model id
Qwen3-235B-A22B-Instruct (2507)
confidence lower
1373.76607135845
confidence upper
1431.6229887928068
Original result ↗

OpenRouter Benchmarks

67.34%

source id
openRouterBenchmarks
leaderboard name
GPQA Diamond · Accuracy · AtlasCloud
dimension
knowledge
metric
GPQA Diamond · Accuracy
rank
29
sample size
396
aggregation scope
status
published
harness
url
https://github.com/OpenRouterTeam/benchmark-harness
name
OpenRouter benchmark harness
direction
higherIsBetter
run_count
2
conditions
window
newest 90 days; successful visible nonzero runs
dataset
benchmark-leaderboard
benchmark
id
gpqa_diamond
name
GPQA Diamond
record_kind
aggregate
provider_name
AtlasCloud
attribution
Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
last_run_at
2026-08-29T18:01:46.940Z
license_url
https://creativecommons.org/licenses/by/4.0/
methodology
Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
metric_kind
GPQA Diamond · Accuracy
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
sample_size
396
source_file
benchmark-leaderboard.json
source_page
https://openrouter.ai/benchmarks/gpqa-diamond
measurements
accuracy
0.6734009999999999
accuracy_std_dev
0.063973
cost_per_task_usd
0.00633028787878788
output_tokens_per_task
7022.232323232323
latency_seconds_per_task
117.05746717171718
published_at
2026-11-16T01:46:02.653Z
snapshot_date
2026-11-16
source_revision
7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
transformations
TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
metric_population
30
benchmark_evaluation
true
openrouter_permaslug
qwen/qwen3-235b-a22b-07-25
source model id
qwen/qwen3-235b-a22b-07-25
Original result ↗

LMArena

1,415.494

source id
lmarena
leaderboard name
LMArena Text Factuality German
dimension
factuality
metric
Arena rating
rank
30
sample size
600
evaluated at
2026-11-09T00:00:00.000Z
source model id
Qwen3-235B-A22B-Instruct (2507)
confidence lower
1394.293589320292
confidence upper
1436.6947415729555
Original result ↗

Published prices

1–3 / 3

Provider specifications & access

1–8 / 8

Qwen/Qwen3-235B-A22B-Instruct-2507 ↗

source model id
azure-huggingface/qwen--qwen3-235b-a22b-instruct-2507@1
hosting
foundry-managed-compute
platforms
  • Microsoft Foundry
license
apache-2.0
context length
262144
architecture
input_modalities
  • text
output_modalities
  • text
capabilities

    Qwen/Qwen3-235B-A22B-Instruct-2507 ↗

    source model id
    HuggingFace/qwen-qwen3-235b-a22b-instruct-2507@1
    hosting
    foundry-managed-compute
    platforms
    • Microsoft Foundry
    license
    apache-2.0
    architecture
    input_modalities
    • text
    output_modalities
    • text
    capabilities

      Qwen/Qwen3-235B-A22B-Instruct-2507 ↗

      source model id
      HuggingFace/qwen-qwen3-235b-a22b-instruct-2507@2
      hosting
      foundry-managed-compute
      platforms
      • Microsoft Foundry
      license
      apache-2.0
      architecture
      input_modalities
      • text
      output_modalities
      • text
      capabilities

        Qwen/Qwen3-235B-A22B-Instruct-2507 ↗

        source model id
        HuggingFace/qwen-qwen3-235b-a22b-instruct-2507@3
        hosting
        foundry-managed-compute
        platforms
        • Microsoft Foundry
        license
        apache-2.0
        architecture
        input_modalities
        • text
        output_modalities
        • text
        capabilities

          Qwen/Qwen3-235B-A22B-Instruct-2507 ↗

          source model id
          HuggingFace/qwen-qwen3-235b-a22b-instruct-2507@4
          hosting
          foundry-managed-compute
          platforms
          • Microsoft Foundry
          license
          apache-2.0
          architecture
          input_modalities
          • text
          output_modalities
          • text
          capabilities

            Qwen/Qwen3-235B-A22B-Instruct-2507 ↗

            source model id
            HuggingFace/qwen-qwen3-235b-a22b-instruct-2507@5
            hosting
            foundry-managed-compute
            platforms
            • Microsoft Foundry
            license
            apache-2.0
            architecture
            input_modalities
            • text
            output_modalities
            • text
            capabilities

              Qwen3-235B-A22B-Instruct-2507 ↗

              source model id
              publishers/qwen/models/qwen3@qwen3-235b-a22b-instruct-2507
              hosting
              vertex-self-deployed
              platforms
              • Google Cloud Vertex AI
              architecture
              input_modalities
              • text
              output_modalities
              • text
              capabilities

                Qwen3-235B-A22B-Instruct-2507 ↗

                source model id
                Qwen/Qwen3-235B-A22B-Instruct-2507
                hosting
                siliconflow-hosted
                platforms
                • SiliconFlow
                max output tokens
                262144
                architecture
                input_modalities
                • text
                output_modalities
                • text
                capabilities
                • structuredOutputs
                • toolCalling

                Safety assessments

                1–1 / 1

                Foundation Model Transparency Index 2025 ↗

                26 / 100

                source id
                stanfordFMTI
                canonical model id
                qwen/qwen3-235b-a22b-07-25
                taxonomy
                transparency
                hazard category
                transparency
                hazard name
                Organization transparency
                policy area
                Alibaba
                assessment type
                organization-level disclosure
                metric
                disclosure_score
                result label
                26 / 100
                sample size
                100
                updated at
                2026-11-09T18:08:05.227Z

                Source specification observations

                1–6 / 6

                azureFoundry ↗

                field
                context_length
                value
                262144
                normalized value
                262144
                scope
                source-model
                source id
                azureFoundry
                confidence
                1

                openRouter ↗

                field
                context_length
                value
                262144
                normalized value
                262144
                scope
                source-model
                source id
                openRouter
                confidence
                1

                siliconFlow ↗

                field
                context_length
                value
                262K
                scope
                source-model
                source id
                siliconFlow
                confidence
                1

                openRouter ↗

                field
                knowledge_cutoff
                reported field
                knowledge_cutoff
                value
                2025-06-30
                normalized value
                2025-06-30
                precision
                day
                scope
                source-model
                source id
                openRouter
                confidence
                1

                openRouter ↗

                field
                max_output_tokens
                value
                117964
                normalized value
                117964
                scope
                source-model
                source id
                openRouter
                confidence
                1

                siliconFlow ↗

                field
                max_output_tokens
                value
                262K
                scope
                source-model
                source id
                siliconFlow
                confidence
                1

                Original sources

                1–20 / 24

                SiliconFlow · Model page ↗

                Hugging Face · Model repository ↗

                Google Cloud · Model page ↗

                Microsoft Foundry · Model page ↗

                Microsoft Foundry · Model page ↗

                Microsoft Foundry · Model page ↗

                Microsoft Foundry · Model page ↗

                Microsoft Foundry · Model page ↗

                Microsoft Foundry · Model page ↗

                OpenRouter · Model page ↗

                SiliconFlow · Playground ↗

                Google Cloud · Model documentation ↗

                Microsoft Foundry · Model documentation ↗

                SiliconFlow · API documentation ↗

                GitHub · Catalog ↗

                GitHub · Catalog ↗

                Google Cloud · Catalog ↗

                Hugging Face · Catalog ↗

                Hugging Face · Catalog ↗

                LMArena · Catalog ↗

                Source links

                1–12 / 12

                Explore more models

                Models

                Loading
                EVIDENCE / COMPARISON

                Benchmarks

                Explore every benchmark

                Loading

                These charts contain model-linked results. The charts above also include unlinked source profiles. Compare scores within each evaluation. Model quality, usage, transparency and system performance measure different things.

                Install TensorTrace