Google: Gemma 4 26B A4B
Gemma 4 26B A4B IT is an instruction-tuned Mixture-of-Experts (MoE) model from Google DeepMind. Despite 25.2B total parameters, only 3.8B activate per token during inference — delivering near-31B qua…
google/gemma-4-26b-a4b-it-20260403
Gemma 4 26B A4B IT is an instruction-tuned Mixture-of-Experts (MoE) model from Google DeepMind. Despite 25.2B total parameters, only 3.8B activate per token during inference — delivering near-31B quality at a fraction of the compute cost. Supports multimodal input including text, images, and video (up to 60s at 1fps). Features a 256K token context window, native function calling, configurable thinking/reasoning mode, and structured output support. Released under Apache 2.0.
Specifications & capabilities
- Input
- image, text, video
- Output
- text
- Context window · tokens
- 262144
- Published model ID
- gemma4@gemma-4-26b-a4b-it
- Capabilities
- multimodal, promptCaching, reasoning, structuredOutputs, toolCalling, visionInput
Description source ↗Evaluations
1–20 / 32
65.209%
- source id
- openRouterBenchmarks
- leaderboard name
- tau2-bench Verified Airline · Accuracy · Darkbloom
- dimension
- agentic
- metric
- tau2-bench Verified Airline · Accuracy
- rank
- 7
- sample size
- 549
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 11
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- tau_bench_verified_airline
- name
- tau2-bench Verified Airline
- record_kind
- aggregate
- provider_name
- Darkbloom
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-11-14T19:42:05.757Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- tau2-bench Verified Airline · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 549
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/tau2-bench-airline
- measurements
- accuracy
- 0.6520946265938069
- accuracy_std_dev
- 0.02296991635103518
- cost_per_task_usd
- 0.011938347905282332
- output_tokens_per_task
- 25229.615664845172
- latency_seconds_per_task
- 701.3510036429873
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 8
- benchmark_evaluation
- true
- openrouter_permaslug
- google/gemma-4-26b-a4b-it-20260403
- source model id
- google/gemma-4-26b-a4b-it-20260403
Original result ↗75.253%
- source id
- openRouterBenchmarks
- leaderboard name
- GPQA Diamond · Accuracy · NextBit
- dimension
- knowledge
- metric
- GPQA Diamond · Accuracy
- rank
- 8
- sample size
- 1980
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 10
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- gpqa_diamond
- name
- GPQA Diamond
- record_kind
- aggregate
- provider_name
- NextBit
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-11-01T13:06:18.223Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- GPQA Diamond · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 1980
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/gpqa-diamond
- measurements
- accuracy
- 0.7525252
- accuracy_std_dev
- 0.014481875091299456
- cost_per_task_usd
- 0.015641921717171717
- output_tokens_per_task
- 56157.93787878788
- latency_seconds_per_task
- 1362.9882676767677
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 10
- benchmark_evaluation
- true
- openrouter_permaslug
- google/gemma-4-26b-a4b-it-20260403
- source model id
- google/gemma-4-26b-a4b-it-20260403
Original result ↗71.002%
- source id
- openRouterBenchmarks
- leaderboard name
- GPQA Diamond · Accuracy · AkashML
- dimension
- knowledge
- metric
- GPQA Diamond · Accuracy
- rank
- 8
- sample size
- 792
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 4
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- gpqa_diamond
- name
- GPQA Diamond
- record_kind
- aggregate
- provider_name
- AkashML
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-10-31T16:02:14.361Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- GPQA Diamond · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 792
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/gpqa-diamond
- measurements
- accuracy
- 0.71001675
- accuracy_std_dev
- 0.018948683824675006
- cost_per_task_usd
- 0.015486457070707072
- output_tokens_per_task
- 64321.568181818184
- latency_seconds_per_task
- 1105.7884797979798
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 10
- benchmark_evaluation
- true
- openrouter_permaslug
- google/gemma-4-26b-a4b-it-20260403
- source model id
- google/gemma-4-26b-a4b-it-20260403
Original result ↗67.933%
- source id
- openRouterBenchmarks
- leaderboard name
- tau2-bench Verified Airline · Accuracy · NextBit
- dimension
- agentic
- metric
- tau2-bench Verified Airline · Accuracy
- rank
- 8
- sample size
- 500
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 10
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- tau_bench_verified_airline
- name
- tau2-bench Verified Airline
- record_kind
- aggregate
- provider_name
- NextBit
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-11-01T13:16:57.769Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- tau2-bench Verified Airline · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 500
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/tau2-bench-airline
- measurements
- accuracy
- 0.6793334
- accuracy_std_dev
- 0.03508080956933576
- cost_per_task_usd
- 0.016317446
- output_tokens_per_task
- 19934.788
- latency_seconds_per_task
- 294.258966
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 8
- benchmark_evaluation
- true
- openrouter_permaslug
- google/gemma-4-26b-a4b-it-20260403
- source model id
- google/gemma-4-26b-a4b-it-20260403
Original result ↗65.167%
- source id
- openRouterBenchmarks
- leaderboard name
- tau2-bench Verified Airline · Accuracy · AkashML
- dimension
- agentic
- metric
- tau2-bench Verified Airline · Accuracy
- rank
- 8
- sample size
- 200
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 4
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- tau_bench_verified_airline
- name
- tau2-bench Verified Airline
- record_kind
- aggregate
- provider_name
- AkashML
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-10-31T19:20:36.815Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- tau2-bench Verified Airline · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 200
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/tau2-bench-airline
- measurements
- accuracy
- 0.65166675
- accuracy_std_dev
- 0.028819327042585528
- cost_per_task_usd
- 0.01676374
- output_tokens_per_task
- 27732.185
- latency_seconds_per_task
- 406.82529
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 10
- benchmark_evaluation
- true
- openrouter_permaslug
- google/gemma-4-26b-a4b-it-20260403
- source model id
- google/gemma-4-26b-a4b-it-20260403
Original result ↗68%
- source id
- openRouterBenchmarks
- leaderboard name
- tau2-bench Verified Airline · Accuracy · DekaLLM
- dimension
- agentic
- metric
- tau2-bench Verified Airline · Accuracy
- rank
- 9
- sample size
- 350
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 7
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- tau_bench_verified_airline
- name
- tau2-bench Verified Airline
- record_kind
- aggregate
- provider_name
- DekaLLM
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-10-31T20:09:24.664Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- tau2-bench Verified Airline · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 350
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/tau2-bench-airline
- measurements
- accuracy
- 0.6799998571428572
- accuracy_std_dev
- 0.0198405229080613
- cost_per_task_usd
- 0.015568199999999999
- output_tokens_per_task
- 18504.7
- latency_seconds_per_task
- 259.0458428571429
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 10
- benchmark_evaluation
- true
- openrouter_permaslug
- google/gemma-4-26b-a4b-it-20260403
- source model id
- google/gemma-4-26b-a4b-it-20260403
Original result ↗78.845%
- source id
- openRouterBenchmarks
- leaderboard name
- GPQA Diamond · Accuracy · Io Net
- dimension
- knowledge
- metric
- GPQA Diamond · Accuracy
- rank
- 10
- sample size
- 1556
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 8
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- gpqa_diamond
- name
- GPQA Diamond
- record_kind
- aggregate
- provider_name
- Io Net
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-11-11T08:00:18.448Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- GPQA Diamond · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 1556
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/gpqa-diamond
- measurements
- accuracy
- 0.7884532538560409
- accuracy_std_dev
- 0.014149184070446418
- cost_per_task_usd
- 0.011693410025706941
- output_tokens_per_task
- 38791.87210796915
- latency_seconds_per_task
- 858.0917879177377
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 15
- benchmark_evaluation
- true
- openrouter_permaslug
- google/gemma-4-26b-a4b-it-20260403
- source model id
- google/gemma-4-26b-a4b-it-20260403
Original result ↗76.698%
- source id
- openRouterBenchmarks
- leaderboard name
- GPQA Diamond · Accuracy · DekaLLM
- dimension
- knowledge
- metric
- GPQA Diamond · Accuracy
- rank
- 10
- sample size
- 1384
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 7
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- gpqa_diamond
- name
- GPQA Diamond
- record_kind
- aggregate
- provider_name
- DekaLLM
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-10-31T16:03:21.445Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- GPQA Diamond · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 1384
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/gpqa-diamond
- measurements
- accuracy
- 0.7669798106936416
- accuracy_std_dev
- 0.011686708686296891
- cost_per_task_usd
- 0.019905706647398845
- output_tokens_per_task
- 60181.387283236996
- latency_seconds_per_task
- 1154.4923439306358
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 12
- benchmark_evaluation
- true
- openrouter_permaslug
- google/gemma-4-26b-a4b-it-20260403
- source model id
- google/gemma-4-26b-a4b-it-20260403
Original result ↗70.611%
- source id
- openRouterBenchmarks
- leaderboard name
- tau2-bench Verified Airline · Accuracy · Makora
- dimension
- agentic
- metric
- tau2-bench Verified Airline · Accuracy
- rank
- 10
- sample size
- 300
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 6
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- tau_bench_verified_airline
- name
- tau2-bench Verified Airline
- record_kind
- aggregate
- provider_name
- Makora
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-10-31T19:52:48.234Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- tau2-bench Verified Airline · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 300
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/tau2-bench-airline
- measurements
- accuracy
- 0.7061111666666666
- accuracy_std_dev
- 0.023602995992152234
- cost_per_task_usd
- 0.023716863333333334
- output_tokens_per_task
- 29595.083333333332
- latency_seconds_per_task
- 111.46781333333334
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 11
- benchmark_evaluation
- true
- openrouter_permaslug
- google/gemma-4-26b-a4b-it-20260403
- source model id
- google/gemma-4-26b-a4b-it-20260403
Original result ↗68.333%
- source id
- openRouterBenchmarks
- leaderboard name
- tau2-bench Verified Airline · Accuracy · PrimeIntellect
- dimension
- agentic
- metric
- tau2-bench Verified Airline · Accuracy
- rank
- 10
- sample size
- 100
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 2
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- tau_bench_verified_airline
- name
- tau2-bench Verified Airline
- record_kind
- aggregate
- provider_name
- PrimeIntellect
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-10-31T19:36:36.917Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- tau2-bench Verified Airline · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 100
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/tau2-bench-airline
- measurements
- accuracy
- 0.6833334999999999
- accuracy_std_dev
- 0.023333500000000007
- cost_per_task_usd
- 0.018626550000000002
- output_tokens_per_task
- 15949.04
- latency_seconds_per_task
- 239.74789
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 21
- benchmark_evaluation
- true
- openrouter_permaslug
- google/gemma-4-26b-a4b-it-20260403
- source model id
- google/gemma-4-26b-a4b-it-20260403
Original result ↗62.055%
- source id
- openRouterBenchmarks
- leaderboard name
- GPQA Diamond · Accuracy · Darkbloom
- dimension
- knowledge
- metric
- GPQA Diamond · Accuracy
- rank
- 10
- sample size
- 2165
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 11
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- gpqa_diamond
- name
- GPQA Diamond
- record_kind
- aggregate
- provider_name
- Darkbloom
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-11-14T19:04:39.024Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- GPQA Diamond · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 2165
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/gpqa-diamond
- measurements
- accuracy
- 0.6205542540415704
- accuracy_std_dev
- 0.07903198055604814
- cost_per_task_usd
- 0.011543149191685911
- output_tokens_per_task
- 55736.75750577367
- latency_seconds_per_task
- 1185.2551769053118
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 10
- benchmark_evaluation
- true
- openrouter_permaslug
- google/gemma-4-26b-a4b-it-20260403
- source model id
- google/gemma-4-26b-a4b-it-20260403
Original result ↗75.14%
- source id
- openRouterBenchmarks
- leaderboard name
- GPQA Diamond · Accuracy · Makora
- dimension
- knowledge
- metric
- GPQA Diamond · Accuracy
- rank
- 11
- sample size
- 1188
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 6
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- gpqa_diamond
- name
- GPQA Diamond
- record_kind
- aggregate
- provider_name
- Makora
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-10-31T16:38:47.420Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- GPQA Diamond · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 1188
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/gpqa-diamond
- measurements
- accuracy
- 0.7514029999999999
- accuracy_std_dev
- 0.013279676389129373
- cost_per_task_usd
- 0.015289352693602695
- output_tokens_per_task
- 62534.38636363636
- latency_seconds_per_task
- 350.9457154882155
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 12
- benchmark_evaluation
- true
- openrouter_permaslug
- google/gemma-4-26b-a4b-it-20260403
- source model id
- google/gemma-4-26b-a4b-it-20260403
Original result ↗78.723%
- source id
- openRouterBenchmarks
- leaderboard name
- GPQA Diamond · Accuracy · PrimeIntellect
- dimension
- knowledge
- metric
- GPQA Diamond · Accuracy
- rank
- 12
- sample size
- 488
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 3
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- gpqa_diamond
- name
- GPQA Diamond
- record_kind
- aggregate
- provider_name
- PrimeIntellect
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-10-31T16:20:33.127Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- GPQA Diamond · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 488
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/gpqa-diamond
- measurements
- accuracy
- 0.7872267663934426
- accuracy_std_dev
- 0.013963506030602306
- cost_per_task_usd
- 0.008632290983606556
- output_tokens_per_task
- 37091.375
- latency_seconds_per_task
- 586.8324200819673
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 22
- benchmark_evaluation
- true
- openrouter_permaslug
- google/gemma-4-26b-a4b-it-20260403
- source model id
- google/gemma-4-26b-a4b-it-20260403
Original result ↗67.676%
- source id
- openRouterBenchmarks
- leaderboard name
- tau2-bench Verified Airline · Accuracy · Io Net
- dimension
- agentic
- metric
- tau2-bench Verified Airline · Accuracy
- rank
- 12
- sample size
- 446
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 9
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- tau_bench_verified_airline
- name
- tau2-bench Verified Airline
- record_kind
- aggregate
- provider_name
- Io Net
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-11-11T08:13:05.865Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- tau2-bench Verified Airline · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 446
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/tau2-bench-airline
- measurements
- accuracy
- 0.6767563901345292
- accuracy_std_dev
- 0.029269691852423077
- cost_per_task_usd
- 0.019482901345291483
- output_tokens_per_task
- 16851.491031390135
- latency_seconds_per_task
- 358.5500986547085
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 15
- benchmark_evaluation
- true
- openrouter_permaslug
- google/gemma-4-26b-a4b-it-20260403
- source model id
- google/gemma-4-26b-a4b-it-20260403
Original result ↗68.174%
- source id
- openRouterBenchmarks
- leaderboard name
- tau2-bench Verified Airline · Accuracy · Cloudflare
- dimension
- agentic
- metric
- tau2-bench Verified Airline · Accuracy
- rank
- 13
- sample size
- 398
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 8
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- tau_bench_verified_airline
- name
- tau2-bench Verified Airline
- record_kind
- aggregate
- provider_name
- Cloudflare
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-10-31T20:14:20.109Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- tau2-bench Verified Airline · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 398
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/tau2-bench-airline
- measurements
- accuracy
- 0.6817421155778893
- accuracy_std_dev
- 0.023817302869644456
- cost_per_task_usd
- 0.02058146231155779
- output_tokens_per_task
- 16771.46984924623
- latency_seconds_per_task
- 307.1874547738694
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 15
- benchmark_evaluation
- true
- openrouter_permaslug
- google/gemma-4-26b-a4b-it-20260403
- source model id
- google/gemma-4-26b-a4b-it-20260403
Original result ↗78.867%
- source id
- openRouterBenchmarks
- leaderboard name
- GPQA Diamond · Accuracy · Cloudflare
- dimension
- knowledge
- metric
- GPQA Diamond · Accuracy
- rank
- 14
- sample size
- 1571
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 8
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- gpqa_diamond
- name
- GPQA Diamond
- record_kind
- aggregate
- provider_name
- Cloudflare
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-10-31T16:03:21.448Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- GPQA Diamond · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 1571
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/gpqa-diamond
- measurements
- accuracy
- 0.7886696422660725
- accuracy_std_dev
- 0.030494803180214167
- cost_per_task_usd
- 0.011359753660089117
- output_tokens_per_task
- 37620.8707829408
- latency_seconds_per_task
- 640.8519363462763
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 18
- benchmark_evaluation
- true
- openrouter_permaslug
- google/gemma-4-26b-a4b-it-20260403
- source model id
- google/gemma-4-26b-a4b-it-20260403
Original result ↗77.149%
- source id
- openRouterBenchmarks
- leaderboard name
- GPQA Diamond · Accuracy · CoreWeave
- dimension
- knowledge
- metric
- GPQA Diamond · Accuracy
- rank
- 21
- sample size
- 830
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 6
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- gpqa_diamond
- name
- GPQA Diamond
- record_kind
- aggregate
- provider_name
- CoreWeave
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-10-31T16:24:40.574Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- GPQA Diamond · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 830
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/gpqa-diamond
- measurements
- accuracy
- 0.7714860192771085
- accuracy_std_dev
- 0.016470310033005864
- cost_per_task_usd
- 0.009284253012048193
- output_tokens_per_task
- 30820.519277108433
- latency_seconds_per_task
- 720.483769879518
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 27
- benchmark_evaluation
- true
- openrouter_permaslug
- google/gemma-4-26b-a4b-it-20260403
- source model id
- google/gemma-4-26b-a4b-it-20260403
Original result ↗76.936%
- source id
- openRouterBenchmarks
- leaderboard name
- GPQA Diamond · Accuracy · SiliconFlow
- dimension
- knowledge
- metric
- GPQA Diamond · Accuracy
- rank
- 25
- sample size
- 1188
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 6
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- gpqa_diamond
- name
- GPQA Diamond
- record_kind
- aggregate
- provider_name
- SiliconFlow
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-11-15T02:47:43.141Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- GPQA Diamond · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 1188
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/gpqa-diamond
- measurements
- accuracy
- 0.7693603333333333
- accuracy_std_dev
- 0.007653292965921387
- cost_per_task_usd
- 0.02112240656565657
- output_tokens_per_task
- 52594.96717171717
- latency_seconds_per_task
- 763.5281077441077
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 34
- benchmark_evaluation
- true
- openrouter_permaslug
- google/gemma-4-26b-a4b-it-20260403
- source model id
- google/gemma-4-26b-a4b-it-20260403
Original result ↗76.503%
- source id
- openRouterBenchmarks
- leaderboard name
- GPQA Diamond · Accuracy · Parasail
- dimension
- knowledge
- metric
- GPQA Diamond · Accuracy
- rank
- 25
- sample size
- 1386
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 7
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- gpqa_diamond
- name
- GPQA Diamond
- record_kind
- aggregate
- provider_name
- Parasail
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-10-31T16:51:21.148Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- GPQA Diamond · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 1386
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/gpqa-diamond
- measurements
- accuracy
- 0.7650311428571429
- accuracy_std_dev
- 0.012201722845900242
- cost_per_task_usd
- 0.021599636363636363
- output_tokens_per_task
- 53902.65151515151
- latency_seconds_per_task
- 1135.0537258297259
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 36
- benchmark_evaluation
- true
- openrouter_permaslug
- google/gemma-4-26b-a4b-it-20260403
- source model id
- google/gemma-4-26b-a4b-it-20260403
Original result ↗78.234%
- source id
- openRouterBenchmarks
- leaderboard name
- GPQA Diamond · Accuracy · Google
- dimension
- knowledge
- metric
- GPQA Diamond · Accuracy
- rank
- 27
- sample size
- 1180
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 6
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- gpqa_diamond
- name
- GPQA Diamond
- record_kind
- aggregate
- provider_name
- Google
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-10-24T16:37:21.981Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- GPQA Diamond · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 1180
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/gpqa-diamond
- measurements
- accuracy
- 0.7823447220338983
- accuracy_std_dev
- 0.020068261404654758
- cost_per_task_usd
- 0.022374091525423728
- output_tokens_per_task
- 37830.4686440678
- latency_seconds_per_task
- 880.6669830508474
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 40
- benchmark_evaluation
- true
- openrouter_permaslug
- google/gemma-4-26b-a4b-it-20260403
- source model id
- google/gemma-4-26b-a4b-it-20260403
Original result ↗Published prices
21–26 / 26
$0.02
- Inference provider
- vercelAIGatewayCatalog
- Platform
- Vercel AI Gateway
- Unit
- cache_read_tokens_per_million
- Currency
- USD
- Billing conditions
- model_type
- language
- quote_scope
- gateway_catalog
- billing_scope
- Vercel public catalog; endpoint prices may differ
- varies_by_provider
- true
- Billing label
- Cached input / 1M
- Pricing via
- Vercel AI Gateway
$0.15
- Inference provider
- vercelAIGatewayCatalog
- Platform
- Vercel AI Gateway
- Unit
- input_tokens_per_million
- Currency
- USD
- Billing conditions
- model_type
- language
- quote_scope
- gateway_catalog
- billing_scope
- Vercel public catalog; endpoint prices may differ
- varies_by_provider
- true
- Billing label
- Input tokens / 1M
- Pricing via
- Vercel AI Gateway
$0.60
- Inference provider
- vercelAIGatewayCatalog
- Platform
- Vercel AI Gateway
- Unit
- output_tokens_per_million
- Currency
- USD
- Billing conditions
- model_type
- language
- quote_scope
- gateway_catalog
- billing_scope
- Vercel public catalog; endpoint prices may differ
- varies_by_provider
- true
- Billing label
- Output tokens / 1M
- Pricing via
- Vercel AI Gateway
$0.02
- Inference provider
- vercelAIGatewayCatalog
- Platform
- vertex
- Unit
- cache_read_tokens_per_million
- Currency
- USD
- Billing conditions
- endpoint
- vertex | google/gemma-4-26b-a4b-it
- model_type
- language
- billing_scope
- via Vercel AI Gateway
- Billing label
- Cached input / 1M
- Pricing via
- Vercel AI Gateway
$0.15
- Inference provider
- vercelAIGatewayCatalog
- Platform
- vertex
- Unit
- input_tokens_per_million
- Currency
- USD
- Billing conditions
- endpoint
- vertex | google/gemma-4-26b-a4b-it
- model_type
- language
- billing_scope
- via Vercel AI Gateway
- Billing label
- Input tokens / 1M
- Pricing via
- Vercel AI Gateway
$0.60
- Inference provider
- vercelAIGatewayCatalog
- Platform
- vertex
- Unit
- output_tokens_per_million
- Currency
- USD
- Billing conditions
- endpoint
- vertex | google/gemma-4-26b-a4b-it
- model_type
- language
- billing_scope
- via Vercel AI Gateway
- Billing label
- Output tokens / 1M
- Pricing via
- Vercel AI Gateway
Provider specifications & access
1–11 / 11
- source model id
- azure-huggingface/google--gemma-4-26b-a4b-it@1
- hosting
- foundry-managed-compute
- license
- apache-2.0
- context length
- 262144
- source model id
- azure-huggingface/google--gemma-4-26b-a4b-it@2
- hosting
- foundry-managed-compute
- license
- apache-2.0
- context length
- 262144
- source model id
- azure-huggingface/google--gemma-4-26b-a4b-it@3
- hosting
- foundry-managed-compute
- license
- apache-2.0
- context length
- 262144
- source model id
- azure-huggingface/google--gemma-4-26b-a4b-it@4
- hosting
- foundry-managed-compute
- license
- apache-2.0
- context length
- 262144
- source model id
- @cf/google/gemma-4-26b-a4b-it
- hosting
- cloudflare-hosted
- context length
- 256000
- source model id
- gemma-4-26b-a4b-it
- hosting
- gateway
- context length
- 262144
- max output tokens
- 131072
- capabilities
- structuredOutputs
- toolCalling
- visionInput
- source model id
- google/gemma-4-26b-a4b-it
- hosting
- vercel-gateway
- context length
- 262144
- max output tokens
- 131072
- capabilities
- promptCaching
- reasoning
- structuredOutputs
- toolCalling
- visionInput
- source model id
- google/gemma-4-26B-A4B-it
- hosting
- siliconflow-hosted
- max output tokens
- 262144
- capabilities
- promptCaching
- reasoning
- structuredOutputs
- toolCalling
- source model id
- HuggingFace/google-gemma-4-26b-a4b-it@2
- hosting
- foundry-managed-compute
- license
- apache-2.0
- source model id
- publishers/google/models/gemma4@gemma-4-26b-a4b-it
- hosting
- vertex-self-deployed
- source model id
- publishers/hf-google/models/gemma-4-26b-a4b-it@001
- hosting
- vertex-self-deployed
Safety assessments
1–1 / 1
41 / 100
- source id
- stanfordFMTI
- canonical model id
- google/gemma-4-26b-a4b-it-20260403
- taxonomy
- transparency
- hazard category
- transparency
- hazard name
- Organization transparency
- policy area
- Google
- assessment type
- organization-level disclosure
- metric
- disclosure_score
- result label
- 41 / 100
- sample size
- 100
- updated at
- 2026-11-09T18:08:05.227Z
Source specification observations
1–12 / 12
- field
- context_length
- value
- 262144
- normalized value
- 262144
- scope
- source-model
- source id
- azureFoundry
- confidence
- 1
- field
- context_length
- value
- 262144
- normalized value
- 262144
- scope
- source-model
- source id
- azureFoundry
- confidence
- 1
- field
- context_length
- value
- 262144
- normalized value
- 262144
- scope
- source-model
- source id
- azureFoundry
- confidence
- 1
- field
- context_length
- value
- 262144
- normalized value
- 262144
- scope
- source-model
- source id
- azureFoundry
- confidence
- 1
- field
- context_length
- value
- 262144
- normalized value
- 262144
- scope
- source-model
- source id
- openRouter
- confidence
- 1
- field
- context_length
- value
- 262144
- normalized value
- 262144
- scope
- source-model
- source id
- openRouter
- confidence
- 1
- field
- context_length
- value
- 262K
- scope
- source-model
- source id
- siliconFlow
- confidence
- 1
- field
- context_length
- value
- 262144
- normalized value
- 262144
- scope
- source-model
- source id
- vercelAIGatewayCatalog
- confidence
- 1
- field
- max_output_tokens
- value
- 235929
- normalized value
- 235929
- scope
- source-model
- source id
- openRouter
- confidence
- 1
- field
- max_output_tokens
- value
- 32768
- normalized value
- 32768
- scope
- source-model
- source id
- openRouter
- confidence
- 1
- field
- max_output_tokens
- value
- 262K
- scope
- source-model
- source id
- siliconFlow
- confidence
- 1
- field
- max_output_tokens
- value
- 131072
- normalized value
- 131072
- scope
- source-model
- source id
- vercelAIGatewayCatalog
- confidence
- 1
Hugging Face model card
1–1 / 1
- downloads
- 5365784
- likes
- 1752
- pipeline tag
- image-text-to-text
- library name
- transformers
- License
- apache-2.0
Explore more models