Google: Gemini 2.5 Pro
Gemini 2.5 Pro is Google’s state-of-the-art AI model designed for advanced reasoning, coding, mathematics, and scientific tasks. It employs “thinking” capabilities, enabling it to reason through resp…
google/gemini-2.5-pro
Gemini 2.5 Pro is Google’s state-of-the-art AI model designed for advanced reasoning, coding, mathematics, and scientific tasks. It employs “thinking” capabilities, enabling it to reason through responses with enhanced accuracy and nuanced context handling. Gemini 2.5 Pro achieves top-tier performance on multiple benchmarks, including first-place positioning on the LMArena leaderboard, reflecting superior human-preference alignment and complex problem-solving abilities.
Specifications & capabilities
- Input
- text, image, file, audio, video
- Output
- text
- Context window · tokens
- 1048576
- Knowledge cutoff
- 2025-01
- Published model ID
- gemini-2.5-pro@default
- Capabilities
- audioInput, batch, fileInput, multimodal, promptCaching, reasoning, structuredOutputs, toolCalling, visionInput, webSearch
Description source ↗Evaluations
61–80 / 1,319
0.93
- source id
- kaggleBenchmarks
- leaderboard name
- Global MMLU Lite German · German
- dimension
- reasoning
- metric
- Global MMLU Lite German · German
- rank
- 14
- aggregation scope
- kaggle
- true
- status
- published
- license
- Apache-2.0
- direction
- higherIsBetter
- conditions
- task
- id
- 41738
- name
- German
- version
- 2
- sort_order
- DESCENDING
- display_type
- rawValues
- benchmark
- id
- 155
- name
- Global MMLU Lite German
- type
- INDIVIDUAL
- owner
- cohere-labs
- version
- 1
- categories
- multilingual and cross-lingual capabilities
- general knowledge and reasoning
- version_id
- 137
- maintainer_tier
- Kaggle-maintained
- maintenance_level
- KAGGLE_MAINTAINED
- candidate
- candidate_class
- modelVersion
- record_kind
- aggregate
- attribution
- Kaggle · cohere-labs · Global MMLU Lite German
- license_url
- https://www.apache.org/licenses/LICENSE-2.0
- metric_kind
- Global MMLU Lite German · German
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- 137/leaderboard.json
- source_page
- https://www.kaggle.com/benchmarks/cohere-labs/global-mmlu-lite-german/versions/1
- measurements
- name
- German
- unit
- score
- scale
- 1
- metric
- Global MMLU Lite German · German
- result
- resultCase
- numericResult
- booleanResult
- false
- numericResult
- value
- 0.93
- standardDeviation
- 0
- confidenceInterval
- 0.0250039481496016
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0250039481496016
- hasUnevenConfidenceInterval
- false
- taskVersionId
- 0
- evaluationDate
- 2025-10-30T00:00:00Z
- hasBooleanResult
- false
- hasNumericResult
- true
- hasTaskVersionId
- false
- hasEvaluationDate
- true
- numericResultNullable
- value
- 0.93
- standardDeviation
- 0
- confidenceInterval
- 0.0250039481496016
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0250039481496016
- hasUnevenConfidenceInterval
- false
- hasNumericResultPublic
- false
- hasNumericResultPrivate
- false
- task_id
- 41738
- task_url
- /benchmarks/tasks/benchmarkler/global-mmlu-lite-german
- sort_order
- DESCENDING
- source_name
- German
- display_type
- rawValues
- task_version
- 2
- task_display_type
- RAW_VALUES
- normalization_status
- normalized
- published_at
- 2025-10-17T15:30:45.767915600Z
- candidate_class
- modelVersion
- source_revision
- b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
- publisher_anchor
- google/gemini-2.5-pro
- metric_population
- 41
- benchmark_evaluation
- true
- task_metadata_source
- benchmarkVersion
- candidate_metadata_source
- benchmarkVersionMappings
- evaluated at
- 2025-10-30T00:00:00.000Z
- source model id
- Gemini 2.5 Pro
- confidence lower
- 0.9049960518503984
- confidence upper
- 0.9550039481496017
Original result ↗0.924
- source id
- kaggleBenchmarks
- leaderboard name
- Global MMLU Lite Culturally Sensitive · CS Score
- dimension
- reasoning
- metric
- Global MMLU Lite Culturally Sensitive · CS Score
- rank
- 14
- aggregation scope
- kaggle
- true
- status
- published
- license
- Apache-2.0
- direction
- higherIsBetter
- conditions
- task
- id
- 41747
- name
- CS Score
- version
- 2
- sort_order
- DESCENDING
- display_type
- rawValues
- benchmark
- id
- 208
- name
- Global MMLU Lite Culturally Sensitive
- type
- INDIVIDUAL
- owner
- cohere-labs
- version
- 1
- categories
- multilingual and cross-lingual capabilities
- general knowledge and reasoning
- version_id
- 187
- maintainer_tier
- Kaggle-maintained
- maintenance_level
- KAGGLE_MAINTAINED
- candidate
- candidate_class
- modelVersion
- record_kind
- aggregate
- attribution
- Kaggle · cohere-labs · Global MMLU Lite Culturally Sensitive
- license_url
- https://www.apache.org/licenses/LICENSE-2.0
- metric_kind
- Global MMLU Lite Culturally Sensitive · CS Score
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- 187/leaderboard.json
- source_page
- https://www.kaggle.com/benchmarks/cohere-labs/global-mmlu-lite-cs/versions/1
- measurements
- name
- CS Score
- unit
- score
- scale
- 1
- metric
- Global MMLU Lite Culturally Sensitive · CS Score
- result
- resultCase
- numericResult
- booleanResult
- false
- numericResult
- value
- 0.9240625
- standardDeviation
- 0
- confidenceInterval
- 0
- hasStandardDeviation
- false
- hasConfidenceInterval
- false
- hasUnevenConfidenceInterval
- false
- taskVersionId
- 0
- evaluationDate
- 2025-11-03T22:31:06.2566667Z
- hasBooleanResult
- false
- hasNumericResult
- true
- hasTaskVersionId
- false
- hasEvaluationDate
- true
- numericResultNullable
- value
- 0.9240625
- standardDeviation
- 0
- confidenceInterval
- 0
- hasStandardDeviation
- false
- hasConfidenceInterval
- false
- hasUnevenConfidenceInterval
- false
- hasNumericResultPublic
- false
- hasNumericResultPrivate
- false
- task_id
- 41747
- task_url
- /benchmarks/tasks/benchmarkler/global-mmlu-lite-cs-score
- sort_order
- DESCENDING
- source_name
- CS Score
- display_type
- rawValues
- task_version
- 2
- task_display_type
- RAW_VALUES
- normalization_status
- normalized
- published_at
- 2025-11-10T16:52:15.372518Z
- candidate_class
- modelVersion
- source_revision
- b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
- publisher_anchor
- google/gemini-2.5-pro
- metric_population
- 41
- benchmark_evaluation
- true
- task_metadata_source
- benchmarkVersion
- candidate_metadata_source
- benchmarkVersionMappings
- evaluated at
- 2025-11-03T22:31:06.256Z
- source model id
- Gemini 2.5 Pro
Original result ↗0.93
- source id
- kaggleBenchmarks
- leaderboard name
- Global MMLU Lite Portuguese · Portuguese
- dimension
- reasoning
- metric
- Global MMLU Lite Portuguese · Portuguese
- rank
- 15
- aggregation scope
- kaggle
- true
- status
- published
- license
- Apache-2.0
- direction
- higherIsBetter
- conditions
- task
- id
- 41746
- name
- Portuguese
- version
- 2
- sort_order
- DESCENDING
- display_type
- rawValues
- benchmark
- id
- 162
- name
- Global MMLU Lite Portuguese
- type
- INDIVIDUAL
- owner
- cohere-labs
- version
- 1
- categories
- multilingual and cross-lingual capabilities
- general knowledge and reasoning
- version_id
- 144
- maintainer_tier
- Kaggle-maintained
- maintenance_level
- KAGGLE_MAINTAINED
- candidate
- candidate_class
- modelVersion
- record_kind
- aggregate
- attribution
- Kaggle · cohere-labs · Global MMLU Lite Portuguese
- license_url
- https://www.apache.org/licenses/LICENSE-2.0
- metric_kind
- Global MMLU Lite Portuguese · Portuguese
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- 144/leaderboard.json
- source_page
- https://www.kaggle.com/benchmarks/cohere-labs/global-mmlu-lite-portuguese/versions/1
- measurements
- name
- Portuguese
- unit
- score
- scale
- 1
- metric
- Global MMLU Lite Portuguese · Portuguese
- result
- resultCase
- numericResult
- booleanResult
- false
- numericResult
- value
- 0.93
- standardDeviation
- 0
- confidenceInterval
- 0.0250039481496016
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0250039481496016
- hasUnevenConfidenceInterval
- false
- taskVersionId
- 0
- evaluationDate
- 2025-10-30T00:00:00Z
- hasBooleanResult
- false
- hasNumericResult
- true
- hasTaskVersionId
- false
- hasEvaluationDate
- true
- numericResultNullable
- value
- 0.93
- standardDeviation
- 0
- confidenceInterval
- 0.0250039481496016
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0250039481496016
- hasUnevenConfidenceInterval
- false
- hasNumericResultPublic
- false
- hasNumericResultPrivate
- false
- task_id
- 41746
- task_url
- /benchmarks/tasks/benchmarkler/global-mmlu-lite-portuguese
- sort_order
- DESCENDING
- source_name
- Portuguese
- display_type
- rawValues
- task_version
- 2
- task_display_type
- RAW_VALUES
- normalization_status
- normalized
- published_at
- 2025-10-17T15:30:30.662890500Z
- candidate_class
- modelVersion
- source_revision
- b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
- publisher_anchor
- google/gemini-2.5-pro
- metric_population
- 41
- benchmark_evaluation
- true
- task_metadata_source
- benchmarkVersion
- candidate_metadata_source
- benchmarkVersionMappings
- evaluated at
- 2025-10-30T00:00:00.000Z
- source model id
- Gemini 2.5 Pro
- confidence lower
- 0.9049960518503984
- confidence upper
- 0.9550039481496017
Original result ↗0.928
- source id
- kaggleBenchmarks
- leaderboard name
- Global MMLU Lite Chinese · Chinese
- dimension
- reasoning
- metric
- Global MMLU Lite Chinese · Chinese
- rank
- 15
- aggregation scope
- kaggle
- true
- status
- published
- license
- Apache-2.0
- direction
- higherIsBetter
- conditions
- task
- id
- 41735
- name
- Chinese
- version
- 2
- sort_order
- DESCENDING
- display_type
- rawValues
- benchmark
- id
- 166
- name
- Global MMLU Lite Chinese
- type
- INDIVIDUAL
- owner
- cohere-labs
- version
- 1
- categories
- multilingual and cross-lingual capabilities
- general knowledge and reasoning
- version_id
- 148
- maintainer_tier
- Kaggle-maintained
- maintenance_level
- KAGGLE_MAINTAINED
- candidate
- candidate_class
- modelVersion
- record_kind
- aggregate
- attribution
- Kaggle · cohere-labs · Global MMLU Lite Chinese
- license_url
- https://www.apache.org/licenses/LICENSE-2.0
- metric_kind
- Global MMLU Lite Chinese · Chinese
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- 148/leaderboard.json
- source_page
- https://www.kaggle.com/benchmarks/cohere-labs/global-mmlu-lite-chinese/versions/1
- measurements
- name
- Chinese
- unit
- score
- scale
- 1
- metric
- Global MMLU Lite Chinese · Chinese
- result
- resultCase
- numericResult
- booleanResult
- false
- numericResult
- value
- 0.9275
- standardDeviation
- 0
- confidenceInterval
- 0.0254123049217328
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0254123049217328
- hasUnevenConfidenceInterval
- false
- taskVersionId
- 0
- evaluationDate
- 2025-10-30T00:00:00Z
- hasBooleanResult
- false
- hasNumericResult
- true
- hasTaskVersionId
- false
- hasEvaluationDate
- true
- numericResultNullable
- value
- 0.9275
- standardDeviation
- 0
- confidenceInterval
- 0.0254123049217328
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0254123049217328
- hasUnevenConfidenceInterval
- false
- hasNumericResultPublic
- false
- hasNumericResultPrivate
- false
- task_id
- 41735
- task_url
- /benchmarks/tasks/benchmarkler/global-mmlu-lite-chinese
- sort_order
- DESCENDING
- source_name
- Chinese
- display_type
- rawValues
- task_version
- 2
- task_display_type
- RAW_VALUES
- normalization_status
- normalized
- published_at
- 2025-10-17T15:30:22.953186100Z
- candidate_class
- modelVersion
- source_revision
- b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
- publisher_anchor
- google/gemini-2.5-pro
- metric_population
- 41
- benchmark_evaluation
- true
- task_metadata_source
- benchmarkVersion
- candidate_metadata_source
- benchmarkVersionMappings
- evaluated at
- 2025-10-30T00:00:00.000Z
- source model id
- Gemini 2.5 Pro
- confidence lower
- 0.9020876950782671
- confidence upper
- 0.9529123049217328
Original result ↗0.928
- source id
- kaggleBenchmarks
- leaderboard name
- Global MMLU Lite English · English
- dimension
- reasoning
- metric
- Global MMLU Lite English · English
- rank
- 16
- aggregation scope
- kaggle
- true
- status
- published
- license
- Apache-2.0
- direction
- higherIsBetter
- conditions
- task
- id
- 41749
- name
- English
- version
- 2
- sort_order
- DESCENDING
- display_type
- rawValues
- benchmark
- id
- 153
- name
- Global MMLU Lite English
- type
- INDIVIDUAL
- owner
- cohere-labs
- version
- 1
- categories
- multilingual and cross-lingual capabilities
- general knowledge and reasoning
- version_id
- 135
- maintainer_tier
- Kaggle-maintained
- maintenance_level
- KAGGLE_MAINTAINED
- candidate
- candidate_class
- modelVersion
- record_kind
- aggregate
- attribution
- Kaggle · cohere-labs · Global MMLU Lite English
- license_url
- https://www.apache.org/licenses/LICENSE-2.0
- metric_kind
- Global MMLU Lite English · English
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- 135/leaderboard.json
- source_page
- https://www.kaggle.com/benchmarks/cohere-labs/global-mmlu-lite-english/versions/1
- measurements
- name
- English
- unit
- score
- scale
- 1
- metric
- Global MMLU Lite English · English
- result
- resultCase
- numericResult
- booleanResult
- false
- numericResult
- value
- 0.9275
- standardDeviation
- 0
- confidenceInterval
- 0.0254123049217328
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0254123049217328
- hasUnevenConfidenceInterval
- false
- taskVersionId
- 0
- evaluationDate
- 2025-10-30T00:00:00Z
- hasBooleanResult
- false
- hasNumericResult
- true
- hasTaskVersionId
- false
- hasEvaluationDate
- true
- numericResultNullable
- value
- 0.9275
- standardDeviation
- 0
- confidenceInterval
- 0.0254123049217328
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0254123049217328
- hasUnevenConfidenceInterval
- false
- hasNumericResultPublic
- false
- hasNumericResultPrivate
- false
- task_id
- 41749
- task_url
- /benchmarks/tasks/benchmarkler/global-mmlu-lite-english
- sort_order
- DESCENDING
- source_name
- English
- display_type
- rawValues
- task_version
- 2
- task_display_type
- RAW_VALUES
- normalization_status
- normalized
- published_at
- 2025-10-17T15:30:49.982103800Z
- candidate_class
- modelVersion
- source_revision
- b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
- publisher_anchor
- google/gemini-2.5-pro
- metric_population
- 41
- benchmark_evaluation
- true
- task_metadata_source
- benchmarkVersion
- candidate_metadata_source
- benchmarkVersionMappings
- evaluated at
- 2025-10-30T00:00:00.000Z
- source model id
- Gemini 2.5 Pro
- confidence lower
- 0.9020876950782671
- confidence upper
- 0.9529123049217328
Original result ↗1,471.047
- source id
- lmarena
- leaderboard name
- LMArena Text Factuality Spanish
- dimension
- factuality
- metric
- Arena rating
- rank
- 18
- sample size
- 1279
- evaluated at
- 2026-11-09T00:00:00.000Z
- source model id
- Gemini 2.5 Pro
- confidence lower
- 1454.4626937817754
- confidence upper
- 1487.631143136493
Original result ↗1,455.495
- source id
- lmarena
- leaderboard name
- LMArena Text Factuality Polish
- dimension
- factuality
- metric
- Arena rating
- rank
- 18
- sample size
- 1123
- evaluated at
- 2026-11-09T00:00:00.000Z
- source model id
- Gemini 2.5 Pro
- confidence lower
- 1437.8397014041427
- confidence upper
- 1473.1501457586287
Original result ↗1,415.175
- source id
- lmarena
- leaderboard name
- LMArena Text Factuality Korean
- dimension
- factuality
- metric
- Arena rating
- rank
- 20
- sample size
- 552
- evaluated at
- 2026-11-09T00:00:00.000Z
- source model id
- Gemini 2.5 Pro
- confidence lower
- 1391.6190581293686
- confidence upper
- 1438.7309466352826
Original result ↗1,486.765
- source id
- lmarena
- leaderboard name
- LMArena Text German
- dimension
- general
- metric
- Arena rating
- rank
- 21
- sample size
- 2646
- evaluated at
- 2026-11-09T00:00:00.000Z
- source model id
- Gemini 2.5 Pro
- confidence lower
- 1474.2396768591714
- confidence upper
- 1499.290357842165
Original result ↗1,473.747
- source id
- lmarena
- leaderboard name
- LMArena Text Spanish
- dimension
- general
- metric
- Arena rating
- rank
- 21
- sample size
- 3431
- evaluated at
- 2026-11-09T00:00:00.000Z
- source model id
- Gemini 2.5 Pro
- confidence lower
- 1462.7376056424157
- confidence upper
- 1484.7571369245982
Original result ↗1,501.137
- source id
- lmarena
- leaderboard name
- LMArena Text Factuality Industry Medicine And Healthcare
- dimension
- factuality
- metric
- Arena rating
- rank
- 23
- sample size
- 2609
- evaluated at
- 2026-11-09T00:00:00.000Z
- source model id
- Gemini 2.5 Pro
- confidence lower
- 1490.4214351672229
- confidence upper
- 1511.8519327381173
Original result ↗82.435%
- source id
- openRouterBenchmarks
- leaderboard name
- GPQA Diamond · Accuracy · Google
- dimension
- knowledge
- metric
- GPQA Diamond · Accuracy
- rank
- 23
- sample size
- 1188
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 6
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- gpqa_diamond
- name
- GPQA Diamond
- record_kind
- aggregate
- provider_name
- Google
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-10-01T19:17:25.275Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- GPQA Diamond · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 1188
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/gpqa-diamond
- measurements
- accuracy
- 0.8243546666666667
- accuracy_std_dev
- 0.03111308863734931
- cost_per_task_usd
- 0.25956171212121215
- output_tokens_per_task
- 25863.887205387204
- latency_seconds_per_task
- 233.6810909090909
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 40
- benchmark_evaluation
- true
- openrouter_permaslug
- google/gemini-2.5-pro
- source model id
- google/gemini-2.5-pro
Original result ↗1,478.912
- source id
- lmarena
- leaderboard name
- LMArena Text Polish
- dimension
- general
- metric
- Arena rating
- rank
- 26
- sample size
- 6489
- evaluated at
- 2026-11-09T00:00:00.000Z
- source model id
- Gemini 2.5 Pro
- confidence lower
- 1470.4848137125482
- confidence upper
- 1487.3401073737236
Original result ↗1,460.887
- source id
- lmarena
- leaderboard name
- LMArena Text Japanese
- dimension
- general
- metric
- Arena rating
- rank
- 29
- sample size
- 2059
- evaluated at
- 2026-11-09T00:00:00.000Z
- source model id
- Gemini 2.5 Pro
- confidence lower
- 1446.2306539856543
- confidence upper
- 1475.5425934003104
Original result ↗1,454.046
- source id
- lmarena
- leaderboard name
- LMArena Text Creative Writing
- dimension
- general
- metric
- Arena rating
- rank
- 30
- sample size
- 17581
- evaluated at
- 2026-11-09T00:00:00.000Z
- source model id
- Gemini 2.5 Pro
- confidence lower
- 1448.9692204098683
- confidence upper
- 1459.1220434268325
Original result ↗60%
- source id
- openRouterBenchmarks
- leaderboard name
- tau2-bench Verified Airline · Accuracy · Google
- dimension
- agentic
- metric
- tau2-bench Verified Airline · Accuracy
- rank
- 30
- sample size
- 300
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 6
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- tau_bench_verified_airline
- name
- tau2-bench Verified Airline
- record_kind
- aggregate
- provider_name
- Google
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-10-01T19:04:55.725Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- tau2-bench Verified Airline · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 300
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/tau2-bench-airline
- measurements
- accuracy
- 0.6000001666666667
- accuracy_std_dev
- 0.03771239112642186
- cost_per_task_usd
- 0.19008634000000002
- output_tokens_per_task
- 11452.75
- latency_seconds_per_task
- 186.97522
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 36
- benchmark_evaluation
- true
- openrouter_permaslug
- google/gemini-2.5-pro
- source model id
- google/gemini-2.5-pro
Original result ↗1,488.348
- source id
- lmarena
- leaderboard name
- LMArena Text Factuality French
- dimension
- factuality
- metric
- Arena rating
- rank
- 31
- sample size
- 1018
- evaluated at
- 2026-11-09T00:00:00.000Z
- source model id
- Gemini 2.5 Pro
- confidence lower
- 1469.8438505555614
- confidence upper
- 1506.851304834845
Original result ↗1,476.08
- source id
- lmarena
- leaderboard name
- LMArena Text Industry Medicine And Healthcare
- dimension
- general
- metric
- Arena rating
- rank
- 31
- sample size
- 7940
- evaluated at
- 2026-11-09T00:00:00.000Z
- source model id
- Gemini 2.5 Pro
- confidence lower
- 1468.930230943948
- confidence upper
- 1483.228997452458
Original result ↗1,480.504
- source id
- lmarena
- leaderboard name
- LMArena Text Industry Legal And Government
- dimension
- general
- metric
- Arena rating
- rank
- 32
- sample size
- 8892
- evaluated at
- 2026-11-09T00:00:00.000Z
- source model id
- Gemini 2.5 Pro
- confidence lower
- 1473.7883528981872
- confidence upper
- 1487.2197803416043
Original result ↗1,490.935
- source id
- lmarena
- leaderboard name
- LMArena Text Factuality Industry Life And Physical And Social Science
- dimension
- factuality
- metric
- Arena rating
- rank
- 33
- sample size
- 7127
- evaluated at
- 2026-11-09T00:00:00.000Z
- source model id
- Gemini 2.5 Pro
- confidence lower
- 1483.5581126778256
- confidence upper
- 1498.3108893259068
Original result ↗Published prices
1–20 / 60
$0.13
- Inference provider
- cloudflareWorkersAI
- Platform
- Google Vertex AI
- Unit
- cache_read_tokens_per_million
- Currency
- USD
- Billing label
- Cached input tokens (per 1M)
- Pricing via
- Cloudflare AI Gateway
$1.25
- Inference provider
- cloudflareWorkersAI
- Platform
- Google Vertex AI
- Unit
- input_tokens_per_million
- Currency
- USD
- Billing label
- Input tokens (per 1M)
- Pricing via
- Cloudflare AI Gateway
$10.00
- Inference provider
- cloudflareWorkersAI
- Platform
- Google Vertex AI
- Unit
- output_tokens_per_million
- Currency
- USD
- Billing label
- Output tokens (per 1M)
- Pricing via
- Cloudflare AI Gateway
$1.25
- Inference provider
- deepInfra
- Platform
- DeepInfra
- Unit
- input_tokens_per_million
- Currency
- USD
$10.00
- Inference provider
- deepInfra
- Platform
- DeepInfra
- Unit
- output_tokens_per_million
- Currency
- USD
$0.13
- Inference provider
- openRouter
- Platform
- OpenRouter
- Unit
- cache_read_tokens_per_million
- Currency
- USD
$0.38
- Inference provider
- openRouter
- Platform
- OpenRouter
- Unit
- cache_write_tokens_per_million
- Currency
- USD
$0.000001
- Inference provider
- openRouter
- Platform
- OpenRouter
- Unit
- image
- Currency
- USD
$1.25
- Inference provider
- openRouter
- Platform
- OpenRouter
- Unit
- input_tokens_per_million
- Currency
- USD
$10.00
- Inference provider
- openRouter
- Platform
- OpenRouter
- Unit
- output_tokens_per_million
- Currency
- USD
$0.01
- Inference provider
- openRouter
- Platform
- OpenRouter
- Unit
- web_search
- Currency
- USD
$0.25
- Inference provider
- vercelAIGatewayCatalog
- Platform
- Vercel AI Gateway
- Unit
- cache_read_tokens_per_million
- Currency
- USD
- Billing conditions
- model_type
- language
- quote_scope
- gateway_catalog
- billing_scope
- Vercel public catalog; endpoint prices may differ
- context_tokens
- min_inclusive
- 200001
- Billing label
- Cached input / 1M
- Pricing via
- Vercel AI Gateway
$0.13
- Inference provider
- vercelAIGatewayCatalog
- Platform
- Vercel AI Gateway
- Unit
- cache_read_tokens_per_million
- Currency
- USD
- Billing conditions
- model_type
- language
- quote_scope
- gateway_catalog
- billing_scope
- Vercel public catalog; endpoint prices may differ
- context_tokens
- max_exclusive
- 200001
- min_inclusive
- 0
- Billing label
- Cached input / 1M
- Pricing via
- Vercel AI Gateway
$0.22
- Inference provider
- vercelAIGatewayCatalog
- Platform
- Vercel AI Gateway
- Unit
- cache_read_tokens_per_million
- Currency
- USD
- Billing conditions
- model_type
- language
- quote_scope
- gateway_catalog
- service_tier
- priority
- billing_scope
- Vercel public catalog; endpoint prices may differ
- Billing label
- Cached input / 1M
- Pricing via
- Vercel AI Gateway
$0.25
- Inference provider
- vercelAIGatewayCatalog
- Platform
- Vercel AI Gateway
- Unit
- cache_read_tokens_per_million
- Currency
- USD
- Billing conditions
- region
- eu
- model_type
- language
- quote_scope
- gateway_catalog
- billing_scope
- Vercel public catalog; endpoint prices may differ
- context_tokens
- min_inclusive
- 200001
- Billing label
- Cached input / 1M
- Pricing via
- Vercel AI Gateway
$0.13
- Inference provider
- vercelAIGatewayCatalog
- Platform
- Vercel AI Gateway
- Unit
- cache_read_tokens_per_million
- Currency
- USD
- Billing conditions
- region
- eu
- model_type
- language
- quote_scope
- gateway_catalog
- billing_scope
- Vercel public catalog; endpoint prices may differ
- context_tokens
- max_exclusive
- 200001
- min_inclusive
- 0
- Billing label
- Cached input / 1M
- Pricing via
- Vercel AI Gateway
$0.25
- Inference provider
- vercelAIGatewayCatalog
- Platform
- Vercel AI Gateway
- Unit
- cache_read_tokens_per_million
- Currency
- USD
- Billing conditions
- region
- us
- model_type
- language
- quote_scope
- gateway_catalog
- billing_scope
- Vercel public catalog; endpoint prices may differ
- context_tokens
- min_inclusive
- 200001
- Billing label
- Cached input / 1M
- Pricing via
- Vercel AI Gateway
$0.13
- Inference provider
- vercelAIGatewayCatalog
- Platform
- Vercel AI Gateway
- Unit
- cache_read_tokens_per_million
- Currency
- USD
- Billing conditions
- region
- us
- model_type
- language
- quote_scope
- gateway_catalog
- billing_scope
- Vercel public catalog; endpoint prices may differ
- context_tokens
- max_exclusive
- 200001
- min_inclusive
- 0
- Billing label
- Cached input / 1M
- Pricing via
- Vercel AI Gateway
$2.50
- Inference provider
- vercelAIGatewayCatalog
- Platform
- Vercel AI Gateway
- Unit
- input_tokens_per_million
- Currency
- USD
- Billing conditions
- model_type
- language
- quote_scope
- gateway_catalog
- billing_scope
- Vercel public catalog; endpoint prices may differ
- context_tokens
- min_inclusive
- 200001
- Billing label
- Input tokens / 1M
- Pricing via
- Vercel AI Gateway
$1.25
- Inference provider
- vercelAIGatewayCatalog
- Platform
- Vercel AI Gateway
- Unit
- input_tokens_per_million
- Currency
- USD
- Billing conditions
- model_type
- language
- quote_scope
- gateway_catalog
- billing_scope
- Vercel public catalog; endpoint prices may differ
- context_tokens
- max_exclusive
- 200001
- min_inclusive
- 0
- Billing label
- Input tokens / 1M
- Pricing via
- Vercel AI Gateway
Provider specifications & access
1–4 / 4
- source model id
- gemini-2.5-pro
- hosting
- gateway
- context length
- 1048576
- max output tokens
- 65535
- capabilities
- reasoning
- structuredOutputs
- toolCalling
- visionInput
- webSearch
- source model id
- google/gemini-2.5-pro
- hosting
- third-party
- context length
- 1000000
- max output tokens
- 65536
- source model id
- google/gemini-2.5-pro
- hosting
- vercel-gateway
- context length
- 1048576
- max output tokens
- 65535
- capabilities
- promptCaching
- reasoning
- structuredOutputs
- toolCalling
- visionInput
- webSearch
- source model id
- publishers/google/models/gemini-2.5-pro@default
- hosting
- vertex-managed
- context length
- 1048576
- max output tokens
- 65536
- capabilities
- batch
- multimodal
- promptCaching
- reasoning
- structuredOutputs
- toolCalling
- visionInput
Safety assessments
1–1 / 1
41 / 100
- source id
- stanfordFMTI
- canonical model id
- google/gemini-2.5-pro
- taxonomy
- transparency
- hazard category
- transparency
- hazard name
- Organization transparency
- policy area
- Google
- assessment type
- organization-level disclosure
- metric
- disclosure_score
- result label
- 41 / 100
- sample size
- 100
- updated at
- 2026-11-09T18:08:05.227Z
Source specification observations
1–10 / 10
- field
- context_length
- value
- 1000000
- normalized value
- 1000000
- scope
- source-model
- source id
- cloudflareWorkersAI
- confidence
- 1
- field
- context_length
- value
- 1048576
- normalized value
- 1048576
- scope
- source-model
- source id
- openRouter
- confidence
- 1
- field
- context_length
- value
- 1048576
- normalized value
- 1048576
- scope
- source-model
- source id
- vercelAIGatewayCatalog
- confidence
- 1
- field
- context_length
- value
- 1048576
- normalized value
- 1048576
- scope
- source-model
- source id
- vertexAI
- confidence
- 1
- field
- knowledge_cutoff
- reported field
- knowledge_cutoff
- value
- 2025-01-31
- normalized value
- 2025-01-31
- precision
- day
- scope
- source-model
- source id
- openRouter
- confidence
- 1
- field
- knowledge_cutoff
- reported field
- knowledge_cutoff
- value
- 2025-01
- normalized value
- 2025-01
- precision
- month
- scope
- source-model
- source id
- vercelAIGatewayCatalog
- confidence
- 1
- field
- max_output_tokens
- value
- 65536
- normalized value
- 65536
- scope
- source-model
- source id
- cloudflareWorkersAI
- confidence
- 1
- field
- max_output_tokens
- value
- 65536
- normalized value
- 65536
- scope
- source-model
- source id
- openRouter
- confidence
- 1
- field
- max_output_tokens
- value
- 65535
- normalized value
- 65535
- scope
- source-model
- source id
- vercelAIGatewayCatalog
- confidence
- 1
- field
- max_output_tokens
- value
- 65536
- normalized value
- 65536
- scope
- source-model
- source id
- vertexAI
- confidence
- 1
Explore more models