Qwen: Qwen3 235B A22B Instruct 2507

Qwen3-235B-A22B-Instruct-2507 is a multilingual, instruction-tuned mixture-of-experts language model based on the Qwen3-235B architecture, with 22B active parameters per forward pass. It is optimized…

qwen/qwen3-235b-a22b-07-25

Qwen3-235B-A22B-Instruct-2507 is a multilingual, instruction-tuned mixture-of-experts language model based on the Qwen3-235B architecture, with 22B active parameters per forward pass. It is optimized for general-purpose text generation, including instruction following, logical reasoning, math, code, and tool usage. The model supports a native 262K context length and does not implement "thinking mode" (<think> blocks).

Compared to its base variant, this version delivers significant gains in knowledge coverage, long-context reasoning, coding benchmarks, and alignment with open-ended tasks. It is particularly strong on multilingual understanding, math reasoning (e.g., AIME, HMMT), and alignment evaluations like Arena-Hard and WritingBench.

Specifications & capabilities

Input
text
Output
text
Context window · tokens
262144
Knowledge cutoff
2025-06-30
Published model ID
Qwen3-235B-A22B-Instruct-2507
Capabilities
promptCaching, structuredOutputs, toolCalling
Description source ↗

Evaluations

1–20 / 160

LLM Stats · GitHub archive

0.95

source id
llmStatsArchive
leaderboard name
ZebraLogic · Raw
dimension
general
metric
ZebraLogic · Raw
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
ZebraLogic
language
en
modality
text
verified
false
max_score
1
categories
  • reasoning
created_at
2025-09-05T00:00:00.000000+00:00
paper_link
https://arxiv.org/abs/2502.01100
updated_at
2025-09-05T00:00:00.000000+00:00
description
ZebraLogic is an evaluation framework for assessing large language models' logical reasoning capabilities through logic grid puzzles derived from constraint satisfaction problems (CSPs). The benchmark consists of 1,000 programmatically generated puzzles with controllable and quantifiable complexity, revealing a 'curse of complexity' where model accuracy declines significantly as problem complexity grows.
benchmark_id
zebralogic
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
ZebraLogic · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/qwen/models/qwen3-235b-a22b-instruct-2507/benchmarks.json
archive_kind
llmStats
evidence_url
https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
qwen/qwen3-235b-a22b-instruct-2507
metric_population
3
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
qwen3-235b-a22b-instruct-2507
Original result ↗

LLM Stats · GitHub archive

0.95

source id
llmStatsArchive
leaderboard name
ZebraLogic · Normalized (reported)
dimension
general
metric
ZebraLogic · Normalized (reported)
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
ZebraLogic
language
en
modality
text
verified
false
max_score
1
categories
  • reasoning
created_at
2025-09-05T00:00:00.000000+00:00
paper_link
https://arxiv.org/abs/2502.01100
updated_at
2025-09-05T00:00:00.000000+00:00
description
ZebraLogic is an evaluation framework for assessing large language models' logical reasoning capabilities through logic grid puzzles derived from constraint satisfaction problems (CSPs). The benchmark consists of 1,000 programmatically generated puzzles with controllable and quantifiable complexity, revealing a 'curse of complexity' where model accuracy declines significantly as problem complexity grows.
benchmark_id
zebralogic
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
ZebraLogic · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/qwen/models/qwen3-235b-a22b-instruct-2507/benchmarks.json
archive_kind
llmStats
evidence_url
https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
qwen/qwen3-235b-a22b-instruct-2507
metric_population
3
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
qwen3-235b-a22b-instruct-2507
Original result ↗

LLM Stats · GitHub archive

0.931

source id
llmStatsArchive
leaderboard name
MMLU-Redux · Normalized (reported)
dimension
general
metric
MMLU-Redux · Normalized (reported)
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
MMLU-Redux
language
en
modality
text
verified
false
max_score
1
categories
  • language
  • reasoning
  • math
  • general
created_at
2025-09-05T00:00:00.000000+00:00
paper_link
https://arxiv.org/abs/2406.04127
updated_at
2025-09-05T00:00:00.000000+00:00
description
An improved version of the MMLU benchmark featuring manually re-annotated questions to identify and correct errors in the original dataset. Provides more reliable evaluation metrics for language models by addressing dataset quality issues found in the original MMLU.
benchmark_id
mmlu-redux
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
MMLU-Redux · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/qwen/models/qwen3-235b-a22b-instruct-2507/benchmarks.json
archive_kind
llmStats
evidence_url
https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
qwen/qwen3-235b-a22b-instruct-2507
metric_population
2
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
qwen3-235b-a22b-instruct-2507
Original result ↗

LLM Stats · GitHub archive

0.931

source id
llmStatsArchive
leaderboard name
MMLU-Redux · Raw
dimension
general
metric
MMLU-Redux · Raw
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
MMLU-Redux
language
en
modality
text
verified
false
max_score
1
categories
  • language
  • reasoning
  • math
  • general
created_at
2025-09-05T00:00:00.000000+00:00
paper_link
https://arxiv.org/abs/2406.04127
updated_at
2025-09-05T00:00:00.000000+00:00
description
An improved version of the MMLU benchmark featuring manually re-annotated questions to identify and correct errors in the original dataset. Provides more reliable evaluation metrics for language models by addressing dataset quality issues found in the original MMLU.
benchmark_id
mmlu-redux
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
MMLU-Redux · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/qwen/models/qwen3-235b-a22b-instruct-2507/benchmarks.json
archive_kind
llmStats
evidence_url
https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
qwen/qwen3-235b-a22b-instruct-2507
metric_population
2
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
qwen3-235b-a22b-instruct-2507
Original result ↗

LLM Stats · GitHub archive

0.887

source id
llmStatsArchive
leaderboard name
IFEval · Normalized (reported)
dimension
general
metric
IFEval · Normalized (reported)
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
IFEval
language
en
modality
text
verified
false
max_score
1
categories
  • general
created_at
2025-07-19T19:56:12.241350+00:00
paper_link
https://arxiv.org/abs/2311.07911
updated_at
2025-07-19T19:56:12.241350+00:00
description
Instruction-Following Evaluation (IFEval) benchmark for large language models, focusing on verifiable instructions with 25 types of instructions and around 500 prompts containing one or more verifiable constraints
benchmark_id
ifeval
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
IFEval · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/qwen/models/qwen3-235b-a22b-instruct-2507/benchmarks.json
archive_kind
llmStats
evidence_url
https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
qwen/qwen3-235b-a22b-instruct-2507
metric_population
3
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
qwen3-235b-a22b-instruct-2507
Original result ↗

LLM Stats · GitHub archive

0.887

source id
llmStatsArchive
leaderboard name
IFEval · Raw
dimension
general
metric
IFEval · Raw
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
IFEval
language
en
modality
text
verified
false
max_score
1
categories
  • general
created_at
2025-07-19T19:56:12.241350+00:00
paper_link
https://arxiv.org/abs/2311.07911
updated_at
2025-07-19T19:56:12.241350+00:00
description
Instruction-Following Evaluation (IFEval) benchmark for large language models, focusing on verifiable instructions with 25 types of instructions and around 500 prompts containing one or more verifiable constraints
benchmark_id
ifeval
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
IFEval · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/qwen/models/qwen3-235b-a22b-instruct-2507/benchmarks.json
archive_kind
llmStats
evidence_url
https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
qwen/qwen3-235b-a22b-instruct-2507
metric_population
3
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
qwen3-235b-a22b-instruct-2507
Original result ↗

LLM Stats · GitHub archive

0.852

source id
llmStatsArchive
leaderboard name
WritingBench · Normalized (reported)
dimension
general
metric
WritingBench · Normalized (reported)
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
WritingBench
language
en
modality
text
verified
false
max_score
1
categories
  • writing
  • creativity
  • communication
created_at
2025-08-03T22:06:11.074130+00:00
paper_link
https://arxiv.org/abs/2503.05244
updated_at
2025-08-03T22:06:11.074130+00:00
description
A comprehensive benchmark for evaluating large language models' generative writing capabilities across 6 core writing domains (Academic & Engineering, Finance & Business, Politics & Law, Literature & Art, Education, Advertising & Marketing) and 100 subdomains. Contains 1,239 queries with a query-dependent evaluation framework that dynamically generates 5 instance-specific assessment criteria for each writing task, using a fine-tuned critic model to score responses on style, format, and length dimensions.
benchmark_id
writingbench
multilingual
true
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
WritingBench · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/qwen/models/qwen3-235b-a22b-instruct-2507/benchmarks.json
archive_kind
llmStats
evidence_url
https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
qwen/qwen3-235b-a22b-instruct-2507
metric_population
1
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
qwen3-235b-a22b-instruct-2507
Original result ↗

LLM Stats · GitHub archive

0.852

source id
llmStatsArchive
leaderboard name
WritingBench · Raw
dimension
general
metric
WritingBench · Raw
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
WritingBench
language
en
modality
text
verified
false
max_score
1
categories
  • writing
  • creativity
  • communication
created_at
2025-08-03T22:06:11.074130+00:00
paper_link
https://arxiv.org/abs/2503.05244
updated_at
2025-08-03T22:06:11.074130+00:00
description
A comprehensive benchmark for evaluating large language models' generative writing capabilities across 6 core writing domains (Academic & Engineering, Finance & Business, Politics & Law, Literature & Art, Education, Advertising & Marketing) and 100 subdomains. Contains 1,239 queries with a query-dependent evaluation framework that dynamically generates 5 instance-specific assessment criteria for each writing task, using a fine-tuned critic model to score responses on style, format, and length dimensions.
benchmark_id
writingbench
multilingual
true
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
WritingBench · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/qwen/models/qwen3-235b-a22b-instruct-2507/benchmarks.json
archive_kind
llmStats
evidence_url
https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
qwen/qwen3-235b-a22b-instruct-2507
metric_population
1
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
qwen3-235b-a22b-instruct-2507
Original result ↗

LLM Stats · GitHub archive

0.843

source id
llmStatsArchive
leaderboard name
CSimpleQA · Normalized (reported)
dimension
general
metric
CSimpleQA · Normalized (reported)
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
CSimpleQA
language
en
modality
text
verified
false
max_score
1
categories
  • general
  • language
created_at
2025-07-19T19:56:11.931358+00:00
paper_link
https://arxiv.org/abs/2411.07140
updated_at
2025-07-19T19:56:11.931358+00:00
description
Chinese SimpleQA is the first comprehensive Chinese benchmark to evaluate the factuality ability of language models to answer short questions. It contains 3,000 high-quality questions spanning 6 major topics with 99 diverse subtopics, designed to assess Chinese factual knowledge across humanities, science, engineering, culture, and society.
benchmark_id
csimpleqa
multilingual
true
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
CSimpleQA · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/qwen/models/qwen3-235b-a22b-instruct-2507/benchmarks.json
archive_kind
llmStats
evidence_url
https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
qwen/qwen3-235b-a22b-instruct-2507
metric_population
1
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
qwen3-235b-a22b-instruct-2507
Original result ↗

LLM Stats · GitHub archive

0.843

source id
llmStatsArchive
leaderboard name
CSimpleQA · Raw
dimension
general
metric
CSimpleQA · Raw
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
CSimpleQA
language
en
modality
text
verified
false
max_score
1
categories
  • general
  • language
created_at
2025-07-19T19:56:11.931358+00:00
paper_link
https://arxiv.org/abs/2411.07140
updated_at
2025-07-19T19:56:11.931358+00:00
description
Chinese SimpleQA is the first comprehensive Chinese benchmark to evaluate the factuality ability of language models to answer short questions. It contains 3,000 high-quality questions spanning 6 major topics with 99 diverse subtopics, designed to assess Chinese factual knowledge across humanities, science, engineering, culture, and society.
benchmark_id
csimpleqa
multilingual
true
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
CSimpleQA · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/qwen/models/qwen3-235b-a22b-instruct-2507/benchmarks.json
archive_kind
llmStats
evidence_url
https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
qwen/qwen3-235b-a22b-instruct-2507
metric_population
1
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
qwen3-235b-a22b-instruct-2507
Original result ↗

LLM Stats · GitHub archive

0.83

source id
llmStatsArchive
leaderboard name
MMLU-Pro · Raw
dimension
general
metric
MMLU-Pro · Raw
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
MMLU-Pro
language
en
modality
text
verified
false
max_score
1
categories
  • language
  • reasoning
  • math
  • general
created_at
2025-07-19T19:56:11.408351+00:00
paper_link
https://arxiv.org/abs/2406.01574
updated_at
2025-07-19T19:56:11.408351+00:00
description
A more robust and challenging multi-task language understanding benchmark that extends MMLU by expanding multiple-choice options from 4 to 10, eliminating trivial questions, and focusing on reasoning-intensive tasks. Features over 12,000 curated questions across 14 domains and causes a 16-33% accuracy drop compared to original MMLU.
benchmark_id
mmlu-pro
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
MMLU-Pro · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/qwen/models/qwen3-235b-a22b-instruct-2507/benchmarks.json
archive_kind
llmStats
evidence_url
https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
qwen/qwen3-235b-a22b-instruct-2507
metric_population
5
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
qwen3-235b-a22b-instruct-2507
Original result ↗

LLM Stats · GitHub archive

0.83

source id
llmStatsArchive
leaderboard name
MMLU-Pro · Normalized (reported)
dimension
general
metric
MMLU-Pro · Normalized (reported)
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
MMLU-Pro
language
en
modality
text
verified
false
max_score
1
categories
  • language
  • reasoning
  • math
  • general
created_at
2025-07-19T19:56:11.408351+00:00
paper_link
https://arxiv.org/abs/2406.01574
updated_at
2025-07-19T19:56:11.408351+00:00
description
A more robust and challenging multi-task language understanding benchmark that extends MMLU by expanding multiple-choice options from 4 to 10, eliminating trivial questions, and focusing on reasoning-intensive tasks. Features over 12,000 curated questions across 14 domains and causes a 16-33% accuracy drop compared to original MMLU.
benchmark_id
mmlu-pro
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
MMLU-Pro · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/qwen/models/qwen3-235b-a22b-instruct-2507/benchmarks.json
archive_kind
llmStats
evidence_url
https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
qwen/qwen3-235b-a22b-instruct-2507
metric_population
5
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
qwen3-235b-a22b-instruct-2507
Original result ↗

LLM Stats · GitHub archive

0.794

source id
llmStatsArchive
leaderboard name
MMLU-ProX · Normalized (reported)
dimension
general
metric
MMLU-ProX · Normalized (reported)
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
MMLU-ProX
language
en
modality
text
verified
false
max_score
1
categories
  • language
  • reasoning
  • math
  • general
created_at
2025-07-19T19:56:13.738623+00:00
paper_link
https://arxiv.org/abs/2406.01574
updated_at
2025-07-19T19:56:13.738623+00:00
description
Extended version of MMLU-Pro providing additional challenging multiple-choice questions for evaluating language models across diverse academic and professional domains. Built on the foundation of the Massive Multitask Language Understanding benchmark framework.
benchmark_id
mmlu-prox
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
MMLU-ProX · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/qwen/models/qwen3-235b-a22b-instruct-2507/benchmarks.json
archive_kind
llmStats
evidence_url
https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
qwen/qwen3-235b-a22b-instruct-2507
metric_population
1
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
qwen3-235b-a22b-instruct-2507
Original result ↗

LLM Stats · GitHub archive

0.794

source id
llmStatsArchive
leaderboard name
MMLU-ProX · Raw
dimension
general
metric
MMLU-ProX · Raw
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
MMLU-ProX
language
en
modality
text
verified
false
max_score
1
categories
  • language
  • reasoning
  • math
  • general
created_at
2025-07-19T19:56:13.738623+00:00
paper_link
https://arxiv.org/abs/2406.01574
updated_at
2025-07-19T19:56:13.738623+00:00
description
Extended version of MMLU-Pro providing additional challenging multiple-choice questions for evaluating language models across diverse academic and professional domains. Built on the foundation of the Massive Multitask Language Understanding benchmark framework.
benchmark_id
mmlu-prox
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
MMLU-ProX · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/qwen/models/qwen3-235b-a22b-instruct-2507/benchmarks.json
archive_kind
llmStats
evidence_url
https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
qwen/qwen3-235b-a22b-instruct-2507
metric_population
1
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
qwen3-235b-a22b-instruct-2507
Original result ↗

LLM Stats · GitHub archive

0.775

source id
llmStatsArchive
leaderboard name
Multi-IF · Normalized (reported)
dimension
general
metric
Multi-IF · Normalized (reported)
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
Multi-IF
language
en
modality
text
verified
false
max_score
1
categories
  • reasoning
  • communication
  • language
created_at
2025-07-19T19:56:14.638787+00:00
paper_link
https://arxiv.org/abs/2410.15553
updated_at
2025-07-19T19:56:14.638787+00:00
description
Multi-IF benchmarks LLMs on multi-turn and multilingual instruction following. It expands upon IFEval by incorporating multi-turn sequences and translating English prompts into 7 other languages, resulting in 4,501 multilingual conversations with three turns each. The benchmark reveals that current leading LLMs struggle with maintaining accuracy in multi-turn instructions and shows higher error rates for non-Latin script languages.
benchmark_id
multi-if
multilingual
true
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
Multi-IF · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/qwen/models/qwen3-235b-a22b-instruct-2507/benchmarks.json
archive_kind
llmStats
evidence_url
https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
qwen/qwen3-235b-a22b-instruct-2507
metric_population
4
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
qwen3-235b-a22b-instruct-2507
Original result ↗

LLM Stats · GitHub archive

0.775

source id
llmStatsArchive
leaderboard name
GPQA · Normalized (reported)
dimension
general
metric
GPQA · Normalized (reported)
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
GPQA
language
en
modality
text
verified
false
max_score
1
categories
  • reasoning
  • general
created_at
2025-07-19T19:56:11.588605+00:00
paper_link
https://arxiv.org/abs/2311.12022
updated_at
2025-07-19T19:56:11.588605+00:00
description
A challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry. Questions are Google-proof and extremely difficult, with PhD experts reaching 65% accuracy.
benchmark_id
gpqa
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
GPQA · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/qwen/models/qwen3-235b-a22b-instruct-2507/benchmarks.json
archive_kind
llmStats
evidence_url
https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
qwen/qwen3-235b-a22b-instruct-2507
metric_population
11
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
qwen3-235b-a22b-instruct-2507
Original result ↗

LLM Stats · GitHub archive

0.775

source id
llmStatsArchive
leaderboard name
Multi-IF · Raw
dimension
general
metric
Multi-IF · Raw
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
Multi-IF
language
en
modality
text
verified
false
max_score
1
categories
  • reasoning
  • communication
  • language
created_at
2025-07-19T19:56:14.638787+00:00
paper_link
https://arxiv.org/abs/2410.15553
updated_at
2025-07-19T19:56:14.638787+00:00
description
Multi-IF benchmarks LLMs on multi-turn and multilingual instruction following. It expands upon IFEval by incorporating multi-turn sequences and translating English prompts into 7 other languages, resulting in 4,501 multilingual conversations with three turns each. The benchmark reveals that current leading LLMs struggle with maintaining accuracy in multi-turn instructions and shows higher error rates for non-Latin script languages.
benchmark_id
multi-if
multilingual
true
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
Multi-IF · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/qwen/models/qwen3-235b-a22b-instruct-2507/benchmarks.json
archive_kind
llmStats
evidence_url
https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
qwen/qwen3-235b-a22b-instruct-2507
metric_population
4
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
qwen3-235b-a22b-instruct-2507
Original result ↗

LLM Stats · GitHub archive

0.775

source id
llmStatsArchive
leaderboard name
GPQA · Raw
dimension
general
metric
GPQA · Raw
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
GPQA
language
en
modality
text
verified
false
max_score
1
categories
  • reasoning
  • general
created_at
2025-07-19T19:56:11.588605+00:00
paper_link
https://arxiv.org/abs/2311.12022
updated_at
2025-07-19T19:56:11.588605+00:00
description
A challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry. Questions are Google-proof and extremely difficult, with PhD experts reaching 65% accuracy.
benchmark_id
gpqa
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
GPQA · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/qwen/models/qwen3-235b-a22b-instruct-2507/benchmarks.json
archive_kind
llmStats
evidence_url
https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
qwen/qwen3-235b-a22b-instruct-2507
metric_population
11
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
qwen3-235b-a22b-instruct-2507
Original result ↗

LLM Stats · GitHub archive

0.754

source id
llmStatsArchive
leaderboard name
LiveBench 20241125 · Raw
dimension
general
metric
LiveBench 20241125 · Raw
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
LiveBench 20241125
language
en
modality
text
verified
false
max_score
1
categories
  • math
  • reasoning
  • general
created_at
2025-08-03T22:06:11.046321+00:00
paper_link
https://arxiv.org/abs/2406.19314
updated_at
2025-08-03T22:06:11.046321+00:00
description
LiveBench is a challenging, contamination-limited LLM benchmark that addresses test set contamination by releasing new questions monthly based on recently-released datasets, arXiv papers, news articles, and IMDb movie synopses. It comprises tasks across math, coding, reasoning, language, instruction following, and data analysis with verifiable, objective ground-truth answers.
benchmark_id
livebench-20241125
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
LiveBench 20241125 · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/qwen/models/qwen3-235b-a22b-instruct-2507/benchmarks.json
archive_kind
llmStats
evidence_url
https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
qwen/qwen3-235b-a22b-instruct-2507
metric_population
1
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
qwen3-235b-a22b-instruct-2507
Original result ↗

LLM Stats · GitHub archive

0.754

source id
llmStatsArchive
leaderboard name
LiveBench 20241125 · Normalized (reported)
dimension
general
metric
LiveBench 20241125 · Normalized (reported)
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
LiveBench 20241125
language
en
modality
text
verified
false
max_score
1
categories
  • math
  • reasoning
  • general
created_at
2025-08-03T22:06:11.046321+00:00
paper_link
https://arxiv.org/abs/2406.19314
updated_at
2025-08-03T22:06:11.046321+00:00
description
LiveBench is a challenging, contamination-limited LLM benchmark that addresses test set contamination by releasing new questions monthly based on recently-released datasets, arXiv papers, news articles, and IMDb movie synopses. It comprises tasks across math, coding, reasoning, language, instruction following, and data analysis with verifiable, objective ground-truth answers.
benchmark_id
livebench-20241125
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
LiveBench 20241125 · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/qwen/models/qwen3-235b-a22b-instruct-2507/benchmarks.json
archive_kind
llmStats
evidence_url
https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
qwen/qwen3-235b-a22b-instruct-2507
metric_population
1
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
qwen3-235b-a22b-instruct-2507
Original result ↗

Published prices

1–3 / 3

Provider specifications & access

1–8 / 8

Qwen/Qwen3-235B-A22B-Instruct-2507 ↗

source model id
azure-huggingface/qwen--qwen3-235b-a22b-instruct-2507@1
hosting
foundry-managed-compute
platforms
  • Microsoft Foundry
license
apache-2.0
context length
262144
architecture
input_modalities
  • text
output_modalities
  • text
capabilities

    Qwen/Qwen3-235B-A22B-Instruct-2507 ↗

    source model id
    HuggingFace/qwen-qwen3-235b-a22b-instruct-2507@1
    hosting
    foundry-managed-compute
    platforms
    • Microsoft Foundry
    license
    apache-2.0
    architecture
    input_modalities
    • text
    output_modalities
    • text
    capabilities

      Qwen/Qwen3-235B-A22B-Instruct-2507 ↗

      source model id
      HuggingFace/qwen-qwen3-235b-a22b-instruct-2507@2
      hosting
      foundry-managed-compute
      platforms
      • Microsoft Foundry
      license
      apache-2.0
      architecture
      input_modalities
      • text
      output_modalities
      • text
      capabilities

        Qwen/Qwen3-235B-A22B-Instruct-2507 ↗

        source model id
        HuggingFace/qwen-qwen3-235b-a22b-instruct-2507@3
        hosting
        foundry-managed-compute
        platforms
        • Microsoft Foundry
        license
        apache-2.0
        architecture
        input_modalities
        • text
        output_modalities
        • text
        capabilities

          Qwen/Qwen3-235B-A22B-Instruct-2507 ↗

          source model id
          HuggingFace/qwen-qwen3-235b-a22b-instruct-2507@4
          hosting
          foundry-managed-compute
          platforms
          • Microsoft Foundry
          license
          apache-2.0
          architecture
          input_modalities
          • text
          output_modalities
          • text
          capabilities

            Qwen/Qwen3-235B-A22B-Instruct-2507 ↗

            source model id
            HuggingFace/qwen-qwen3-235b-a22b-instruct-2507@5
            hosting
            foundry-managed-compute
            platforms
            • Microsoft Foundry
            license
            apache-2.0
            architecture
            input_modalities
            • text
            output_modalities
            • text
            capabilities

              Qwen3-235B-A22B-Instruct-2507 ↗

              source model id
              publishers/qwen/models/qwen3@qwen3-235b-a22b-instruct-2507
              hosting
              vertex-self-deployed
              platforms
              • Google Cloud Vertex AI
              architecture
              input_modalities
              • text
              output_modalities
              • text
              capabilities

                Qwen3-235B-A22B-Instruct-2507 ↗

                source model id
                Qwen/Qwen3-235B-A22B-Instruct-2507
                hosting
                siliconflow-hosted
                platforms
                • SiliconFlow
                max output tokens
                262144
                architecture
                input_modalities
                • text
                output_modalities
                • text
                capabilities
                • structuredOutputs
                • toolCalling

                Safety assessments

                1–1 / 1

                Foundation Model Transparency Index 2025 ↗

                26 / 100

                source id
                stanfordFMTI
                canonical model id
                qwen/qwen3-235b-a22b-07-25
                taxonomy
                transparency
                hazard category
                transparency
                hazard name
                Organization transparency
                policy area
                Alibaba
                assessment type
                organization-level disclosure
                metric
                disclosure_score
                result label
                26 / 100
                sample size
                100
                updated at
                2026-11-09T18:08:05.227Z

                Source specification observations

                1–6 / 6

                azureFoundry ↗

                field
                context_length
                value
                262144
                normalized value
                262144
                scope
                source-model
                source id
                azureFoundry
                confidence
                1

                openRouter ↗

                field
                context_length
                value
                262144
                normalized value
                262144
                scope
                source-model
                source id
                openRouter
                confidence
                1

                siliconFlow ↗

                field
                context_length
                value
                262K
                scope
                source-model
                source id
                siliconFlow
                confidence
                1

                openRouter ↗

                field
                knowledge_cutoff
                reported field
                knowledge_cutoff
                value
                2025-06-30
                normalized value
                2025-06-30
                precision
                day
                scope
                source-model
                source id
                openRouter
                confidence
                1

                openRouter ↗

                field
                max_output_tokens
                value
                117964
                normalized value
                117964
                scope
                source-model
                source id
                openRouter
                confidence
                1

                siliconFlow ↗

                field
                max_output_tokens
                value
                262K
                scope
                source-model
                source id
                siliconFlow
                confidence
                1

                Original sources

                1–20 / 24

                SiliconFlow · Model page ↗

                Hugging Face · Model repository ↗

                Google Cloud · Model page ↗

                Microsoft Foundry · Model page ↗

                Microsoft Foundry · Model page ↗

                Microsoft Foundry · Model page ↗

                Microsoft Foundry · Model page ↗

                Microsoft Foundry · Model page ↗

                Microsoft Foundry · Model page ↗

                OpenRouter · Model page ↗

                SiliconFlow · Playground ↗

                Google Cloud · Model documentation ↗

                Microsoft Foundry · Model documentation ↗

                SiliconFlow · API documentation ↗

                GitHub · Catalog ↗

                GitHub · Catalog ↗

                Google Cloud · Catalog ↗

                Hugging Face · Catalog ↗

                Hugging Face · Catalog ↗

                LMArena · Catalog ↗

                Source links

                1–12 / 12

                Explore more models

                Models

                Loading
                EVIDENCE / COMPARISON

                Benchmarks

                Explore every benchmark

                Loading

                These charts contain model-linked results. The charts above also include unlinked source profiles. Compare scores within each evaluation. Model quality, usage, transparency and system performance measure different things.

                Install TensorTrace