Qwen: Qwen3 235B A22B Instruct 2507
Qwen3-235B-A22B-Instruct-2507 is a multilingual, instruction-tuned mixture-of-experts language model based on the Qwen3-235B architecture, with 22B active parameters per forward pass. It is optimized…
qwen/qwen3-235b-a22b-07-25
Qwen3-235B-A22B-Instruct-2507 is a multilingual, instruction-tuned mixture-of-experts language model based on the Qwen3-235B architecture, with 22B active parameters per forward pass. It is optimized for general-purpose text generation, including instruction following, logical reasoning, math, code, and tool usage. The model supports a native 262K context length and does not implement "thinking mode" (<think> blocks).
Compared to its base variant, this version delivers significant gains in knowledge coverage, long-context reasoning, coding benchmarks, and alignment with open-ended tasks. It is particularly strong on multilingual understanding, math reasoning (e.g., AIME, HMMT), and alignment evaluations like Arena-Hard and WritingBench.
Specifications & capabilities
- Input
- text
- Output
- text
- Context window · tokens
- 262144
- Knowledge cutoff
- 2025-06-30
- Published model ID
- Qwen3-235B-A22B-Instruct-2507
- Capabilities
- promptCaching, structuredOutputs, toolCalling
Description source ↗Evaluations
41–60 / 160
0.573
- source id
- llmStatsArchive
- leaderboard name
- Aider-Polyglot · Normalized (reported)
- dimension
- general
- metric
- Aider-Polyglot · Normalized (reported)
- rank
- 3
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- Aider-Polyglot
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-09-05T00:00:00.000000+00:00
- updated_at
- 2025-09-30T00:00:00.000000+00:00
- description
- A coding benchmark that evaluates LLMs on 225 challenging Exercism programming exercises across C++, Go, Java, JavaScript, Python, and Rust. Models receive two attempts to solve each problem, with test error feedback provided after the first attempt if it fails. The benchmark measures both initial problem-solving ability and capacity to edit code based on error feedback, providing an end-to-end evaluation of code generation and editing capabilities across multiple programming languages.
- benchmark_id
- aider-polyglot
- multilingual
- false
- implementation_link
- https://github.com/Aider-AI/polyglot-benchmark
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- Aider-Polyglot · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/qwen/models/qwen3-235b-a22b-instruct-2507/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- qwen/qwen3-235b-a22b-instruct-2507
- metric_population
- 5
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- qwen3-235b-a22b-instruct-2507
Original result ↗0.573
- source id
- llmStatsArchive
- leaderboard name
- Aider-Polyglot · Raw
- dimension
- general
- metric
- Aider-Polyglot · Raw
- rank
- 3
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- Aider-Polyglot
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-09-05T00:00:00.000000+00:00
- updated_at
- 2025-09-30T00:00:00.000000+00:00
- description
- A coding benchmark that evaluates LLMs on 225 challenging Exercism programming exercises across C++, Go, Java, JavaScript, Python, and Rust. Models receive two attempts to solve each problem, with test error feedback provided after the first attempt if it fails. The benchmark measures both initial problem-solving ability and capacity to edit code based on error feedback, providing an end-to-end evaluation of code generation and editing capabilities across multiple programming languages.
- benchmark_id
- aider-polyglot
- multilingual
- false
- implementation_link
- https://github.com/Aider-AI/polyglot-benchmark
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- Aider-Polyglot · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/qwen/models/qwen3-235b-a22b-instruct-2507/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- qwen/qwen3-235b-a22b-instruct-2507
- metric_population
- 5
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- qwen3-235b-a22b-instruct-2507
Original result ↗40.667%
- source id
- openRouterBenchmarks
- leaderboard name
- tau2-bench Verified Airline · Accuracy · PrimeIntellect
- dimension
- agentic
- metric
- tau2-bench Verified Airline · Accuracy
- rank
- 15
- sample size
- 50
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 1
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- tau_bench_verified_airline
- name
- tau2-bench Verified Airline
- record_kind
- aggregate
- provider_name
- PrimeIntellect
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-10-04T03:00:16.402Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- tau2-bench Verified Airline · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 50
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/tau2-bench-airline
- measurements
- accuracy
- 0.406667
- cost_per_task_usd
- 0.06754938
- output_tokens_per_task
- 3789.7
- latency_seconds_per_task
- 114.52978
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 21
- benchmark_evaluation
- true
- openrouter_permaslug
- qwen/qwen3-235b-a22b-07-25
- source model id
- qwen/qwen3-235b-a22b-07-25
Original result ↗75.253%
- source id
- openRouterBenchmarks
- leaderboard name
- GPQA Diamond · Accuracy · PrimeIntellect
- dimension
- knowledge
- metric
- GPQA Diamond · Accuracy
- rank
- 16
- sample size
- 198
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 1
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- gpqa_diamond
- name
- GPQA Diamond
- record_kind
- aggregate
- provider_name
- PrimeIntellect
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-10-04T03:06:11.710Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- GPQA Diamond · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 198
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/gpqa-diamond
- measurements
- accuracy
- 0.7525250000000001
- cost_per_task_usd
- 0.008515843434343435
- output_tokens_per_task
- 8312.732323232323
- latency_seconds_per_task
- 110.88425757575757
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 22
- benchmark_evaluation
- true
- openrouter_permaslug
- qwen/qwen3-235b-a22b-07-25
- source model id
- qwen/qwen3-235b-a22b-07-25
Original result ↗74.141%
- source id
- openRouterBenchmarks
- leaderboard name
- GPQA Diamond · Accuracy · Nebius
- dimension
- knowledge
- metric
- GPQA Diamond · Accuracy
- rank
- 20
- sample size
- 990
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 5
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- gpqa_diamond
- name
- GPQA Diamond
- record_kind
- aggregate
- provider_name
- Nebius
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-10-17T16:06:15.944Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- GPQA Diamond · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 990
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/gpqa-diamond
- measurements
- accuracy
- 0.7414141999999999
- accuracy_std_dev
- 0.0027351667883330276
- cost_per_task_usd
- 0.00532230808080808
- output_tokens_per_task
- 8599.7
- latency_seconds_per_task
- 145.93918484848484
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 25
- benchmark_evaluation
- true
- openrouter_permaslug
- qwen/qwen3-235b-a22b-07-25
- source model id
- qwen/qwen3-235b-a22b-07-25
Original result ↗43.4%
- source id
- openRouterBenchmarks
- leaderboard name
- tau2-bench Verified Airline · Accuracy · Nebius
- dimension
- agentic
- metric
- tau2-bench Verified Airline · Accuracy
- rank
- 20
- sample size
- 250
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 5
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- tau_bench_verified_airline
- name
- tau2-bench Verified Airline
- record_kind
- aggregate
- provider_name
- Nebius
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-10-17T19:11:29.194Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- tau2-bench Verified Airline · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 250
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/tau2-bench-airline
- measurements
- accuracy
- 0.434
- accuracy_std_dev
- 0.03129784515266187
- cost_per_task_usd
- 0.053281564000000003
- output_tokens_per_task
- 3661.612
- latency_seconds_per_task
- 160.868104
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 23
- benchmark_evaluation
- true
- openrouter_permaslug
- qwen/qwen3-235b-a22b-07-25
- source model id
- qwen/qwen3-235b-a22b-07-25
Original result ↗74.074%
- source id
- openRouterBenchmarks
- leaderboard name
- GPQA Diamond · Accuracy · Alibaba
- dimension
- knowledge
- metric
- GPQA Diamond · Accuracy
- rank
- 23
- sample size
- 594
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 3
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- gpqa_diamond
- name
- GPQA Diamond
- record_kind
- aggregate
- provider_name
- Alibaba
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-10-03T16:59:36.480Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- GPQA Diamond · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 594
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/gpqa-diamond
- measurements
- accuracy
- 0.7407406666666667
- accuracy_std_dev
- 0.008584264726165454
- cost_per_task_usd
- 0.004939484848484848
- output_tokens_per_task
- 8056.897306397306
- latency_seconds_per_task
- 108.5982037037037
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 29
- benchmark_evaluation
- true
- openrouter_permaslug
- qwen/qwen3-235b-a22b-07-25
- source model id
- qwen/qwen3-235b-a22b-07-25
Original result ↗41.5%
- source id
- openRouterBenchmarks
- leaderboard name
- tau2-bench Verified Airline · Accuracy · StreamLake
- dimension
- agentic
- metric
- tau2-bench Verified Airline · Accuracy
- rank
- 23
- sample size
- 200
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 4
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- tau_bench_verified_airline
- name
- tau2-bench Verified Airline
- record_kind
- aggregate
- provider_name
- StreamLake
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-10-03T16:19:54.632Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- tau2-bench Verified Airline · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 200
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/tau2-bench-airline
- measurements
- accuracy
- 0.415
- accuracy_std_dev
- 0.02763845951749121
- cost_per_task_usd
- 0.051504745
- output_tokens_per_task
- 3278.695
- latency_seconds_per_task
- 126.44671000000001
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 25
- benchmark_evaluation
- true
- openrouter_permaslug
- qwen/qwen3-235b-a22b-07-25
- source model id
- qwen/qwen3-235b-a22b-07-25
Original result ↗36.833%
- source id
- openRouterBenchmarks
- leaderboard name
- tau2-bench Verified Airline · Accuracy · GMICloud
- dimension
- agentic
- metric
- tau2-bench Verified Airline · Accuracy
- rank
- 23
- sample size
- 200
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 4
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- tau_bench_verified_airline
- name
- tau2-bench Verified Airline
- record_kind
- aggregate
- provider_name
- GMICloud
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-10-03T16:12:29.879Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- tau2-bench Verified Airline · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 200
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/tau2-bench-airline
- measurements
- accuracy
- 0.36833325
- accuracy_std_dev
- 0.17853565236861657
- cost_per_task_usd
- 0.015193219999999999
- output_tokens_per_task
- 2917.505
- latency_seconds_per_task
- 127.68209
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 23
- benchmark_evaluation
- true
- openrouter_permaslug
- qwen/qwen3-235b-a22b-07-25
- source model id
- qwen/qwen3-235b-a22b-07-25
Original result ↗44.588%
- source id
- openRouterBenchmarks
- leaderboard name
- tau2-bench Verified Airline · Accuracy · Parasail
- dimension
- agentic
- metric
- tau2-bench Verified Airline · Accuracy
- rank
- 24
- sample size
- 348
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 7
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- tau_bench_verified_airline
- name
- tau2-bench Verified Airline
- record_kind
- aggregate
- provider_name
- Parasail
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-10-31T18:58:15.016Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- tau2-bench Verified Airline · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 348
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/tau2-bench-airline
- measurements
- accuracy
- 0.4458811867816092
- accuracy_std_dev
- 0.045214591965110434
- cost_per_task_usd
- 0.015823612068965518
- output_tokens_per_task
- 3370.8706896551726
- latency_seconds_per_task
- 133.6663620689655
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 28
- benchmark_evaluation
- true
- openrouter_permaslug
- qwen/qwen3-235b-a22b-07-25
- source model id
- qwen/qwen3-235b-a22b-07-25
Original result ↗70.286%
- source id
- openRouterBenchmarks
- leaderboard name
- GPQA Diamond · Accuracy · GMICloud
- dimension
- knowledge
- metric
- GPQA Diamond · Accuracy
- rank
- 25
- sample size
- 792
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 4
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- gpqa_diamond
- name
- GPQA Diamond
- record_kind
- aggregate
- provider_name
- GMICloud
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-10-03T16:56:36.098Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- GPQA Diamond · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 792
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/gpqa-diamond
- measurements
- accuracy
- 0.7028622499999999
- accuracy_std_dev
- 0.08200476941122087
- cost_per_task_usd
- 0.0070032777777777775
- output_tokens_per_task
- 7212.60101010101
- latency_seconds_per_task
- 172.46112373737373
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 26
- benchmark_evaluation
- true
- openrouter_permaslug
- qwen/qwen3-235b-a22b-07-25
- source model id
- qwen/qwen3-235b-a22b-07-25
Original result ↗73.617%
- source id
- openRouterBenchmarks
- leaderboard name
- GPQA Diamond · Accuracy · Parasail
- dimension
- knowledge
- metric
- GPQA Diamond · Accuracy
- rank
- 27
- sample size
- 1386
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 7
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- gpqa_diamond
- name
- GPQA Diamond
- record_kind
- aggregate
- provider_name
- Parasail
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-10-31T16:02:14.362Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- GPQA Diamond · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 1386
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/gpqa-diamond
- measurements
- accuracy
- 0.7361712857142858
- accuracy_std_dev
- 0.00933947044559173
- cost_per_task_usd
- 0.0070149227994227995
- output_tokens_per_task
- 8674.670995670996
- latency_seconds_per_task
- 250.93870995670997
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 36
- benchmark_evaluation
- true
- openrouter_permaslug
- qwen/qwen3-235b-a22b-07-25
- source model id
- qwen/qwen3-235b-a22b-07-25
Original result ↗38.667%
- source id
- openRouterBenchmarks
- leaderboard name
- tau2-bench Verified Airline · Accuracy · AtlasCloud
- dimension
- agentic
- metric
- tau2-bench Verified Airline · Accuracy
- rank
- 27
- sample size
- 100
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 2
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- tau_bench_verified_airline
- name
- tau2-bench Verified Airline
- record_kind
- aggregate
- provider_name
- AtlasCloud
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-08-29T16:12:10.467Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- tau2-bench Verified Airline · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 100
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/tau2-bench-airline
- measurements
- accuracy
- 0.386667
- accuracy_std_dev
- 0.06
- cost_per_task_usd
- 0.052157499999999996
- output_tokens_per_task
- 2942.27
- latency_seconds_per_task
- 111.01769999999999
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 29
- benchmark_evaluation
- true
- openrouter_permaslug
- qwen/qwen3-235b-a22b-07-25
- source model id
- qwen/qwen3-235b-a22b-07-25
Original result ↗35.333%
- source id
- openRouterBenchmarks
- leaderboard name
- tau2-bench Verified Airline · Accuracy · Alibaba
- dimension
- agentic
- metric
- tau2-bench Verified Airline · Accuracy
- rank
- 27
- sample size
- 150
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 3
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- tau_bench_verified_airline
- name
- tau2-bench Verified Airline
- record_kind
- aggregate
- provider_name
- Alibaba
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-10-03T16:05:51.296Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- tau2-bench Verified Airline · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 150
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/tau2-bench-airline
- measurements
- accuracy
- 0.35333333333333333
- accuracy_std_dev
- 0.04455559988997518
- cost_per_task_usd
- 0.06266134666666666
- output_tokens_per_task
- 3931.2533333333336
- latency_seconds_per_task
- 100.24880666666667
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 29
- benchmark_evaluation
- true
- openrouter_permaslug
- qwen/qwen3-235b-a22b-07-25
- source model id
- qwen/qwen3-235b-a22b-07-25
Original result ↗12.3%
- source id
- openCompass
- leaderboard name
- HLE
- dimension
- general
- metric
- HLE
- rank
- 27
- aggregation scope
- status
- published
- direction
- higherIsBetter
- conditions
- suite
- OverallTable
- catalog
- open
- snapshot
- month
- REALTIME
- fileName
- assets/research-rank/research-data.REALTIME.20260309.json
- updateTime
- 2026-03-09
- columnFileName
- assets/research-rank/research-column.REALTIME.20260309.json
- metric_kind
- HLE
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- 0-data.json
- source_revision
- 117c5effa3066abf39f3e8f61109707b09f0cd95223246c07acd6289f4f7fcd2
- metric_population
- 48
- benchmark_evaluation
- true
- source model id
- Qwen3-235B-A22B-Instruct-2507
Original result ↗75.5%
- source id
- openCompass
- leaderboard name
- GPQA-Diamond
- dimension
- general
- metric
- GPQA-Diamond
- rank
- 28
- aggregation scope
- status
- published
- direction
- higherIsBetter
- conditions
- suite
- OverallTable
- catalog
- open
- snapshot
- month
- REALTIME
- fileName
- assets/research-rank/research-data.REALTIME.20260309.json
- updateTime
- 2026-03-09
- columnFileName
- assets/research-rank/research-column.REALTIME.20260309.json
- metric_kind
- GPQA-Diamond
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- 0-data.json
- source_revision
- 117c5effa3066abf39f3e8f61109707b09f0cd95223246c07acd6289f4f7fcd2
- metric_population
- 48
- benchmark_evaluation
- true
- source model id
- Qwen3-235B-A22B-Instruct-2507
Original result ↗74.537%
- source id
- openRouterBenchmarks
- leaderboard name
- GPQA Diamond · Accuracy · StreamLake
- dimension
- knowledge
- metric
- GPQA Diamond · Accuracy
- rank
- 28
- sample size
- 792
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 4
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- gpqa_diamond
- name
- GPQA Diamond
- record_kind
- aggregate
- provider_name
- StreamLake
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-10-03T16:58:32.590Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- GPQA Diamond · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 792
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/gpqa-diamond
- measurements
- accuracy
- 0.7453702499999999
- accuracy_std_dev
- 0.0095511571910162
- cost_per_task_usd
- 0.006491083333333334
- output_tokens_per_task
- 7540.814393939394
- latency_seconds_per_task
- 157.74153661616162
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 30
- benchmark_evaluation
- true
- openrouter_permaslug
- qwen/qwen3-235b-a22b-07-25
- source model id
- qwen/qwen3-235b-a22b-07-25
Original result ↗1,402.695
- source id
- lmarena
- leaderboard name
- LMArena Text Factuality Korean
- dimension
- factuality
- metric
- Arena rating
- rank
- 29
- sample size
- 484
- evaluated at
- 2026-11-09T00:00:00.000Z
- source model id
- Qwen3-235B-A22B-Instruct (2507)
- confidence lower
- 1373.76607135845
- confidence upper
- 1431.6229887928068
Original result ↗67.34%
- source id
- openRouterBenchmarks
- leaderboard name
- GPQA Diamond · Accuracy · AtlasCloud
- dimension
- knowledge
- metric
- GPQA Diamond · Accuracy
- rank
- 29
- sample size
- 396
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 2
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- gpqa_diamond
- name
- GPQA Diamond
- record_kind
- aggregate
- provider_name
- AtlasCloud
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-08-29T18:01:46.940Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- GPQA Diamond · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 396
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/gpqa-diamond
- measurements
- accuracy
- 0.6734009999999999
- accuracy_std_dev
- 0.063973
- cost_per_task_usd
- 0.00633028787878788
- output_tokens_per_task
- 7022.232323232323
- latency_seconds_per_task
- 117.05746717171718
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 30
- benchmark_evaluation
- true
- openrouter_permaslug
- qwen/qwen3-235b-a22b-07-25
- source model id
- qwen/qwen3-235b-a22b-07-25
Original result ↗1,415.494
- source id
- lmarena
- leaderboard name
- LMArena Text Factuality German
- dimension
- factuality
- metric
- Arena rating
- rank
- 30
- sample size
- 600
- evaluated at
- 2026-11-09T00:00:00.000Z
- source model id
- Qwen3-235B-A22B-Instruct (2507)
- confidence lower
- 1394.293589320292
- confidence upper
- 1436.6947415729555
Original result ↗Published prices
1–3 / 3
$0.05
- Inference provider
- openRouter
- Platform
- OpenRouter
- Unit
- cache_read_tokens_per_million
- Currency
- USD
$0.14
- Inference provider
- openRouter
- Platform
- OpenRouter
- Unit
- input_tokens_per_million
- Currency
- USD
$0.80
- Inference provider
- openRouter
- Platform
- OpenRouter
- Unit
- output_tokens_per_million
- Currency
- USD
Provider specifications & access
1–8 / 8
- source model id
- azure-huggingface/qwen--qwen3-235b-a22b-instruct-2507@1
- hosting
- foundry-managed-compute
- license
- apache-2.0
- context length
- 262144
- source model id
- HuggingFace/qwen-qwen3-235b-a22b-instruct-2507@1
- hosting
- foundry-managed-compute
- license
- apache-2.0
- source model id
- HuggingFace/qwen-qwen3-235b-a22b-instruct-2507@2
- hosting
- foundry-managed-compute
- license
- apache-2.0
- source model id
- HuggingFace/qwen-qwen3-235b-a22b-instruct-2507@3
- hosting
- foundry-managed-compute
- license
- apache-2.0
- source model id
- HuggingFace/qwen-qwen3-235b-a22b-instruct-2507@4
- hosting
- foundry-managed-compute
- license
- apache-2.0
- source model id
- HuggingFace/qwen-qwen3-235b-a22b-instruct-2507@5
- hosting
- foundry-managed-compute
- license
- apache-2.0
- source model id
- publishers/qwen/models/qwen3@qwen3-235b-a22b-instruct-2507
- hosting
- vertex-self-deployed
- source model id
- Qwen/Qwen3-235B-A22B-Instruct-2507
- hosting
- siliconflow-hosted
- max output tokens
- 262144
- capabilities
- structuredOutputs
- toolCalling
Safety assessments
1–1 / 1
26 / 100
- source id
- stanfordFMTI
- canonical model id
- qwen/qwen3-235b-a22b-07-25
- taxonomy
- transparency
- hazard category
- transparency
- hazard name
- Organization transparency
- policy area
- Alibaba
- assessment type
- organization-level disclosure
- metric
- disclosure_score
- result label
- 26 / 100
- sample size
- 100
- updated at
- 2026-11-09T18:08:05.227Z
Source specification observations
1–6 / 6
- field
- context_length
- value
- 262144
- normalized value
- 262144
- scope
- source-model
- source id
- azureFoundry
- confidence
- 1
- field
- context_length
- value
- 262144
- normalized value
- 262144
- scope
- source-model
- source id
- openRouter
- confidence
- 1
- field
- context_length
- value
- 262K
- scope
- source-model
- source id
- siliconFlow
- confidence
- 1
- field
- knowledge_cutoff
- reported field
- knowledge_cutoff
- value
- 2025-06-30
- normalized value
- 2025-06-30
- precision
- day
- scope
- source-model
- source id
- openRouter
- confidence
- 1
- field
- max_output_tokens
- value
- 117964
- normalized value
- 117964
- scope
- source-model
- source id
- openRouter
- confidence
- 1
- field
- max_output_tokens
- value
- 262K
- scope
- source-model
- source id
- siliconFlow
- confidence
- 1
Explore more models