Google: Gemini 2.5 Pro
Gemini 2.5 Pro is Google’s state-of-the-art AI model designed for advanced reasoning, coding, mathematics, and scientific tasks. It employs “thinking” capabilities, enabling it to reason through resp…
google/gemini-2.5-pro
Gemini 2.5 Pro is Google’s state-of-the-art AI model designed for advanced reasoning, coding, mathematics, and scientific tasks. It employs “thinking” capabilities, enabling it to reason through responses with enhanced accuracy and nuanced context handling. Gemini 2.5 Pro achieves top-tier performance on multiple benchmarks, including first-place positioning on the LMArena leaderboard, reflecting superior human-preference alignment and complex problem-solving abilities.
Specifications & capabilities
- Input
- text, image, file, audio, video
- Output
- text
- Context window · tokens
- 1048576
- Knowledge cutoff
- 2025-01
- Published model ID
- gemini-2.5-pro@default
- Capabilities
- audioInput, batch, fileInput, multimodal, promptCaching, reasoning, structuredOutputs, toolCalling, visionInput, webSearch
Description source ↗Evaluations
41–60 / 1,319
68.843%
- source id
- kaggleBenchmarks
- leaderboard name
- AssetOpsBench · Elimination Accuracy
- dimension
- reasoning
- metric
- AssetOpsBench · Elimination Accuracy
- rank
- 8
- aggregation scope
- kaggle
- true
- status
- published
- license
- Apache-2.0
- direction
- higherIsBetter
- conditions
- task
- id
- 1041
- name
- Elimination Accuracy
- version
- 1
- sort_order
- DESCENDING
- display_type
- percentages
- benchmark
- id
- 243
- name
- AssetOpsBench
- type
- INDIVIDUAL
- owner
- ibm-research
- version
- 1
- categories
- evaluation
- general knowledge and reasoning
- version_id
- 222
- maintainer_tier
- Kaggle-maintained
- maintenance_level
- KAGGLE_MAINTAINED
- candidate
- candidate_class
- modelVersion
- record_kind
- aggregate
- attribution
- Kaggle · ibm-research · AssetOpsBench
- license_url
- https://www.apache.org/licenses/LICENSE-2.0
- metric_kind
- AssetOpsBench · Elimination Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- 222/leaderboard.json
- source_page
- https://www.kaggle.com/benchmarks/ibm-research/asset-ops-bench/versions/1
- measurements
- name
- Elimination Accuracy
- unit
- percent
- scale
- 1
- metric
- AssetOpsBench · Elimination Accuracy
- result
- resultCase
- numericResult
- booleanResult
- false
- numericResult
- value
- 0.6884335503638452
- standardDeviation
- 0
- confidenceInterval
- 0
- hasStandardDeviation
- false
- hasConfidenceInterval
- false
- hasUnevenConfidenceInterval
- false
- taskVersionId
- 0
- evaluationDate
- 2025-11-22T14:59:16.384029Z
- hasBooleanResult
- false
- hasNumericResult
- true
- hasTaskVersionId
- false
- hasEvaluationDate
- true
- numericResultNullable
- value
- 0.6884335503638452
- standardDeviation
- 0
- confidenceInterval
- 0
- hasStandardDeviation
- false
- hasConfidenceInterval
- false
- hasUnevenConfidenceInterval
- false
- hasNumericResultPublic
- false
- hasNumericResultPrivate
- false
- task_id
- 1041
- task_url
- /benchmarks/tasks/benchmarkler/failuresensoriq-eliminationaccuracy-benchmark-task
- sort_order
- DESCENDING
- source_name
- Elimination Accuracy
- display_type
- percentages
- task_version
- 1
- task_display_type
- RAW_VALUES
- normalization_status
- normalized
- published_at
- 2025-12-01T22:15:26.261663300Z
- candidate_class
- modelVersion
- source_revision
- b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
- publisher_anchor
- google/gemini-2.5-pro
- metric_population
- 25
- benchmark_evaluation
- true
- task_metadata_source
- benchmarkVersion
- candidate_metadata_source
- benchmarkVersionMappings
- evaluated at
- 2025-11-22T14:59:16.384Z
- source model id
- Gemini 2.5 Pro
Original result ↗66.97%
- source id
- kaggleBenchmarks
- leaderboard name
- AssetOpsBench · F-Score
- dimension
- reasoning
- metric
- AssetOpsBench · F-Score
- rank
- 8
- aggregation scope
- kaggle
- true
- status
- published
- license
- Apache-2.0
- direction
- higherIsBetter
- conditions
- task
- id
- 1040
- name
- F-Score
- version
- 1
- sort_order
- DESCENDING
- display_type
- percentages
- benchmark
- id
- 243
- name
- AssetOpsBench
- type
- INDIVIDUAL
- owner
- ibm-research
- version
- 1
- categories
- evaluation
- general knowledge and reasoning
- version_id
- 222
- maintainer_tier
- Kaggle-maintained
- maintenance_level
- KAGGLE_MAINTAINED
- candidate
- candidate_class
- modelVersion
- record_kind
- aggregate
- attribution
- Kaggle · ibm-research · AssetOpsBench
- license_url
- https://www.apache.org/licenses/LICENSE-2.0
- metric_kind
- AssetOpsBench · F-Score
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- 222/leaderboard.json
- source_page
- https://www.kaggle.com/benchmarks/ibm-research/asset-ops-bench/versions/1
- measurements
- name
- F-Score
- unit
- percent
- scale
- 1
- metric
- AssetOpsBench · F-Score
- result
- resultCase
- numericResult
- booleanResult
- false
- numericResult
- value
- 0.6697034163433864
- standardDeviation
- 0
- confidenceInterval
- 0
- hasStandardDeviation
- false
- hasConfidenceInterval
- false
- hasUnevenConfidenceInterval
- false
- taskVersionId
- 0
- evaluationDate
- 2025-11-22T14:27:51.636277Z
- hasBooleanResult
- false
- hasNumericResult
- true
- hasTaskVersionId
- false
- hasEvaluationDate
- true
- numericResultNullable
- value
- 0.6697034163433864
- standardDeviation
- 0
- confidenceInterval
- 0
- hasStandardDeviation
- false
- hasConfidenceInterval
- false
- hasUnevenConfidenceInterval
- false
- hasNumericResultPublic
- false
- hasNumericResultPrivate
- false
- task_id
- 1040
- task_url
- /benchmarks/tasks/benchmarkler/failuresensoriq-fscore-benchmark-task
- sort_order
- DESCENDING
- source_name
- F-Score
- display_type
- percentages
- task_version
- 1
- task_display_type
- RAW_VALUES
- normalization_status
- normalized
- published_at
- 2025-12-01T22:15:26.261663300Z
- candidate_class
- modelVersion
- source_revision
- b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
- publisher_anchor
- google/gemini-2.5-pro
- metric_population
- 25
- benchmark_evaluation
- true
- task_metadata_source
- benchmarkVersion
- candidate_metadata_source
- benchmarkVersionMappings
- evaluated at
- 2025-11-22T14:27:51.636Z
- source model id
- Gemini 2.5 Pro
Original result ↗62.622%
- source id
- kaggleBenchmarks
- leaderboard name
- AssetOpsBench · Selection Accuracy
- dimension
- reasoning
- metric
- AssetOpsBench · Selection Accuracy
- rank
- 8
- aggregation scope
- kaggle
- true
- status
- published
- license
- Apache-2.0
- direction
- higherIsBetter
- conditions
- task
- id
- 1042
- name
- Selection Accuracy
- version
- 1
- sort_order
- DESCENDING
- display_type
- percentages
- benchmark
- id
- 243
- name
- AssetOpsBench
- type
- INDIVIDUAL
- owner
- ibm-research
- version
- 1
- categories
- evaluation
- general knowledge and reasoning
- version_id
- 222
- maintainer_tier
- Kaggle-maintained
- maintenance_level
- KAGGLE_MAINTAINED
- candidate
- candidate_class
- modelVersion
- record_kind
- aggregate
- attribution
- Kaggle · ibm-research · AssetOpsBench
- license_url
- https://www.apache.org/licenses/LICENSE-2.0
- metric_kind
- AssetOpsBench · Selection Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- 222/leaderboard.json
- source_page
- https://www.kaggle.com/benchmarks/ibm-research/asset-ops-bench/versions/1
- measurements
- name
- Selection Accuracy
- unit
- percent
- scale
- 1
- metric
- AssetOpsBench · Selection Accuracy
- result
- resultCase
- numericResult
- booleanResult
- false
- numericResult
- value
- 0.6262188515709642
- standardDeviation
- 0
- confidenceInterval
- 0
- hasStandardDeviation
- false
- hasConfidenceInterval
- false
- hasUnevenConfidenceInterval
- false
- taskVersionId
- 0
- evaluationDate
- 2025-11-22T14:59:16.297106Z
- hasBooleanResult
- false
- hasNumericResult
- true
- hasTaskVersionId
- false
- hasEvaluationDate
- true
- numericResultNullable
- value
- 0.6262188515709642
- standardDeviation
- 0
- confidenceInterval
- 0
- hasStandardDeviation
- false
- hasConfidenceInterval
- false
- hasUnevenConfidenceInterval
- false
- hasNumericResultPublic
- false
- hasNumericResultPrivate
- false
- task_id
- 1042
- task_url
- /benchmarks/tasks/benchmarkler/failuresensoriq-selectionaccuracy-benchmark-task
- sort_order
- DESCENDING
- source_name
- Selection Accuracy
- display_type
- percentages
- task_version
- 1
- task_display_type
- RAW_VALUES
- normalization_status
- normalized
- published_at
- 2025-12-01T22:15:26.261663300Z
- candidate_class
- modelVersion
- source_revision
- b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
- publisher_anchor
- google/gemini-2.5-pro
- metric_population
- 25
- benchmark_evaluation
- true
- task_metadata_source
- benchmarkVersion
- candidate_metadata_source
- benchmarkVersionMappings
- evaluated at
- 2025-11-22T14:59:16.297Z
- source model id
- Gemini 2.5 Pro
Original result ↗55%
- source id
- openRouterBenchmarks
- leaderboard name
- tau2-bench Verified Airline · Accuracy · Google AI Studio
- dimension
- agentic
- metric
- tau2-bench Verified Airline · Accuracy
- rank
- 8
- sample size
- 100
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 2
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- tau_bench_verified_airline
- name
- tau2-bench Verified Airline
- record_kind
- aggregate
- provider_name
- Google AI Studio
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-10-01T19:04:52.649Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- tau2-bench Verified Airline · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 100
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/tau2-bench-airline
- measurements
- accuracy
- 0.55
- accuracy_std_dev
- 0.036667000000000005
- cost_per_task_usd
- 0.2373302
- output_tokens_per_task
- 12508.34
- latency_seconds_per_task
- 164.4292
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 10
- benchmark_evaluation
- true
- openrouter_permaslug
- google/gemini-2.5-pro
- source model id
- google/gemini-2.5-pro
Original result ↗31.25%
- source id
- kaggleBenchmarks
- leaderboard name
- ITBench · SRE Score
- dimension
- reasoning
- metric
- ITBench · SRE Score
- rank
- 8
- aggregation scope
- kaggle
- true
- status
- published
- license
- Apache-2.0
- direction
- higherIsBetter
- conditions
- task
- id
- 1172
- name
- SRE Score
- version
- 8
- sort_order
- DESCENDING
- display_type
- rawPercentages
- benchmark
- id
- 246
- name
- ITBench
- type
- INDIVIDUAL
- owner
- ibm-research
- version
- 1
- categories
- evaluation
- general knowledge and reasoning
- version_id
- 225
- maintainer_tier
- Kaggle-maintained
- maintenance_level
- KAGGLE_MAINTAINED
- candidate
- candidate_class
- modelVersion
- record_kind
- aggregate
- attribution
- Kaggle · ibm-research · ITBench
- license_url
- https://www.apache.org/licenses/LICENSE-2.0
- metric_kind
- ITBench · SRE Score
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- 225/leaderboard.json
- source_page
- https://www.kaggle.com/benchmarks/ibm-research/itbench/versions/1
- measurements
- name
- SRE Score
- unit
- percent
- scale
- 100
- metric
- ITBench · SRE Score
- result
- resultCase
- numericResult
- booleanResult
- false
- numericResult
- value
- 31.25
- standardDeviation
- 0
- confidenceInterval
- 0
- hasStandardDeviation
- false
- hasConfidenceInterval
- false
- hasUnevenConfidenceInterval
- false
- taskVersionId
- 0
- evaluationDate
- 2025-12-01T22:27:17.354159Z
- hasBooleanResult
- false
- hasNumericResult
- true
- hasTaskVersionId
- false
- hasEvaluationDate
- true
- numericResultNullable
- value
- 31.25
- standardDeviation
- 0
- confidenceInterval
- 0
- hasStandardDeviation
- false
- hasConfidenceInterval
- false
- hasUnevenConfidenceInterval
- false
- hasNumericResultPublic
- false
- hasNumericResultPrivate
- false
- task_id
- 1172
- task_url
- /benchmarks/tasks/benchmarkler/itbench-sre-main
- sort_order
- DESCENDING
- source_name
- SRE Score
- display_type
- rawPercentages
- task_version
- 8
- task_display_type
- RAW_VALUES
- normalization_status
- normalized
- published_at
- 2025-12-02T12:36:17.407125500Z
- candidate_class
- modelVersion
- source_revision
- b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
- publisher_anchor
- google/gemini-2.5-pro
- metric_population
- 13
- benchmark_evaluation
- true
- task_metadata_source
- benchmarkVersion
- candidate_metadata_source
- benchmarkVersionMappings
- evaluated at
- 2025-12-01T22:27:17.354Z
- source model id
- Gemini 2.5 Pro
Original result ↗0.93
- source id
- kaggleBenchmarks
- leaderboard name
- Global MMLU Lite Burmese · Burmese
- dimension
- reasoning
- metric
- Global MMLU Lite Burmese · Burmese
- rank
- 9
- aggregation scope
- kaggle
- true
- status
- published
- license
- Apache-2.0
- direction
- higherIsBetter
- conditions
- task
- id
- 41736
- name
- Burmese
- version
- 2
- sort_order
- DESCENDING
- display_type
- rawValues
- benchmark
- id
- 175
- name
- Global MMLU Lite Burmese
- type
- INDIVIDUAL
- owner
- cohere-labs
- version
- 1
- categories
- multilingual and cross-lingual capabilities
- general knowledge and reasoning
- version_id
- 153
- maintainer_tier
- Kaggle-maintained
- maintenance_level
- KAGGLE_MAINTAINED
- candidate
- candidate_class
- modelVersion
- record_kind
- aggregate
- attribution
- Kaggle · cohere-labs · Global MMLU Lite Burmese
- license_url
- https://www.apache.org/licenses/LICENSE-2.0
- metric_kind
- Global MMLU Lite Burmese · Burmese
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- 153/leaderboard.json
- source_page
- https://www.kaggle.com/benchmarks/cohere-labs/global-mmlu-lite-burmese/versions/1
- measurements
- name
- Burmese
- unit
- score
- scale
- 1
- metric
- Global MMLU Lite Burmese · Burmese
- result
- resultCase
- numericResult
- booleanResult
- false
- numericResult
- value
- 0.93
- standardDeviation
- 0
- confidenceInterval
- 0.0250039481496016
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0250039481496016
- hasUnevenConfidenceInterval
- false
- taskVersionId
- 0
- evaluationDate
- 2025-10-30T00:00:00Z
- hasBooleanResult
- false
- hasNumericResult
- true
- hasTaskVersionId
- false
- hasEvaluationDate
- true
- numericResultNullable
- value
- 0.93
- standardDeviation
- 0
- confidenceInterval
- 0.0250039481496016
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0250039481496016
- hasUnevenConfidenceInterval
- false
- hasNumericResultPublic
- false
- hasNumericResultPrivate
- false
- task_id
- 41736
- task_url
- /benchmarks/tasks/benchmarkler/global-mmlu-lite-burmese
- sort_order
- DESCENDING
- source_name
- Burmese
- display_type
- rawValues
- task_version
- 2
- task_display_type
- RAW_VALUES
- normalization_status
- normalized
- published_at
- 2025-10-17T15:30:19.694332700Z
- candidate_class
- modelVersion
- source_revision
- b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
- publisher_anchor
- google/gemini-2.5-pro
- metric_population
- 41
- benchmark_evaluation
- true
- task_metadata_source
- benchmarkVersion
- candidate_metadata_source
- benchmarkVersionMappings
- evaluated at
- 2025-10-30T00:00:00.000Z
- source model id
- Gemini 2.5 Pro
- confidence lower
- 0.9049960518503984
- confidence upper
- 0.9550039481496017
Original result ↗63.217%
- source id
- kaggleBenchmarks
- leaderboard name
- AssetOpsBench · Pert. Complex Accuracy
- dimension
- reasoning
- metric
- AssetOpsBench · Pert. Complex Accuracy
- rank
- 9
- aggregation scope
- kaggle
- true
- status
- published
- license
- Apache-2.0
- direction
- higherIsBetter
- conditions
- task
- id
- 1044
- name
- Pert. Complex Accuracy
- version
- 1
- sort_order
- DESCENDING
- display_type
- percentages
- benchmark
- id
- 243
- name
- AssetOpsBench
- type
- INDIVIDUAL
- owner
- ibm-research
- version
- 1
- categories
- evaluation
- general knowledge and reasoning
- version_id
- 222
- maintainer_tier
- Kaggle-maintained
- maintenance_level
- KAGGLE_MAINTAINED
- candidate
- candidate_class
- modelVersion
- record_kind
- aggregate
- attribution
- Kaggle · ibm-research · AssetOpsBench
- license_url
- https://www.apache.org/licenses/LICENSE-2.0
- metric_kind
- AssetOpsBench · Pert. Complex Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- 222/leaderboard.json
- source_page
- https://www.kaggle.com/benchmarks/ibm-research/asset-ops-bench/versions/1
- measurements
- name
- Pert. Complex Accuracy
- unit
- percent
- scale
- 1
- metric
- AssetOpsBench · Pert. Complex Accuracy
- result
- resultCase
- numericResult
- booleanResult
- false
- numericResult
- value
- 0.6321709786276716
- standardDeviation
- 0
- confidenceInterval
- 0
- hasStandardDeviation
- false
- hasConfidenceInterval
- false
- hasUnevenConfidenceInterval
- false
- taskVersionId
- 0
- evaluationDate
- 2025-11-24T03:34:20.061647Z
- hasBooleanResult
- false
- hasNumericResult
- true
- hasTaskVersionId
- false
- hasEvaluationDate
- true
- numericResultNullable
- value
- 0.6321709786276716
- standardDeviation
- 0
- confidenceInterval
- 0
- hasStandardDeviation
- false
- hasConfidenceInterval
- false
- hasUnevenConfidenceInterval
- false
- hasNumericResultPublic
- false
- hasNumericResultPrivate
- false
- task_id
- 1044
- task_url
- /benchmarks/tasks/benchmarkler/failuresensoriq-perturbationcomplex-benchmark-task
- sort_order
- DESCENDING
- source_name
- Pert. Complex Accuracy
- display_type
- percentages
- task_version
- 1
- task_display_type
- RAW_VALUES
- normalization_status
- normalized
- published_at
- 2025-12-01T22:15:26.261663300Z
- candidate_class
- modelVersion
- source_revision
- b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
- publisher_anchor
- google/gemini-2.5-pro
- metric_population
- 25
- benchmark_evaluation
- true
- task_metadata_source
- benchmarkVersion
- candidate_metadata_source
- benchmarkVersionMappings
- evaluated at
- 2025-11-24T03:34:20.061Z
- source model id
- Gemini 2.5 Pro
Original result ↗42.292%
- source id
- kaggleBenchmarks
- leaderboard name
- ITBench · Average Score
- dimension
- reasoning
- metric
- ITBench · Average Score
- rank
- 9
- aggregation scope
- kaggle
- true
- status
- published
- license
- Apache-2.0
- direction
- higherIsBetter
- conditions
- task
- id
- 1064
- name
- Average Score
- version
- 1
- sort_order
- DESCENDING
- display_type
- rawPercentages
- benchmark
- id
- 246
- name
- ITBench
- type
- INDIVIDUAL
- owner
- ibm-research
- version
- 1
- categories
- evaluation
- general knowledge and reasoning
- version_id
- 225
- maintainer_tier
- Kaggle-maintained
- maintenance_level
- KAGGLE_MAINTAINED
- candidate
- candidate_class
- modelVersion
- record_kind
- aggregate
- attribution
- Kaggle · ibm-research · ITBench
- license_url
- https://www.apache.org/licenses/LICENSE-2.0
- metric_kind
- ITBench · Average Score
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- 225/leaderboard.json
- source_page
- https://www.kaggle.com/benchmarks/ibm-research/itbench/versions/1
- measurements
- name
- Average Score
- unit
- percent
- scale
- 100
- metric
- ITBench · Average Score
- result
- resultCase
- numericResult
- booleanResult
- false
- numericResult
- value
- 42.29166666666667
- standardDeviation
- 0
- confidenceInterval
- 0
- hasStandardDeviation
- false
- hasConfidenceInterval
- false
- hasUnevenConfidenceInterval
- false
- taskVersionId
- 0
- evaluationDate
- 2025-12-01T22:27:17.354159Z
- hasBooleanResult
- false
- hasNumericResult
- true
- hasTaskVersionId
- false
- hasEvaluationDate
- true
- numericResultNullable
- value
- 42.29166666666667
- standardDeviation
- 0
- confidenceInterval
- 0
- hasStandardDeviation
- false
- hasConfidenceInterval
- false
- hasUnevenConfidenceInterval
- false
- hasNumericResultPublic
- false
- hasNumericResultPrivate
- false
- task_id
- 1064
- sort_order
- DESCENDING
- source_name
- Average Score
- display_type
- rawPercentages
- task_version
- 1
- task_display_type
- RAW_PERCENTAGES
- normalization_status
- normalized
- published_at
- 2025-12-02T12:36:17.407125500Z
- candidate_class
- modelVersion
- source_revision
- b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
- publisher_anchor
- google/gemini-2.5-pro
- metric_population
- 13
- benchmark_evaluation
- true
- task_metadata_source
- benchmarkVersion
- candidate_metadata_source
- benchmarkVersionMappings
- evaluated at
- 2025-12-01T22:27:17.354Z
- source model id
- Gemini 2.5 Pro
Original result ↗0.943
- source id
- kaggleBenchmarks
- leaderboard name
- Global MMLU Lite French · French
- dimension
- reasoning
- metric
- Global MMLU Lite French · French
- rank
- 10
- aggregation scope
- kaggle
- true
- status
- published
- license
- Apache-2.0
- direction
- higherIsBetter
- conditions
- task
- id
- 41752
- name
- French
- version
- 2
- sort_order
- DESCENDING
- display_type
- rawValues
- benchmark
- id
- 156
- name
- Global MMLU Lite French
- type
- INDIVIDUAL
- owner
- cohere-labs
- version
- 1
- categories
- multilingual and cross-lingual capabilities
- general knowledge and reasoning
- version_id
- 138
- maintainer_tier
- Kaggle-maintained
- maintenance_level
- KAGGLE_MAINTAINED
- candidate
- candidate_class
- modelVersion
- record_kind
- aggregate
- attribution
- Kaggle · cohere-labs · Global MMLU Lite French
- license_url
- https://www.apache.org/licenses/LICENSE-2.0
- metric_kind
- Global MMLU Lite French · French
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- 138/leaderboard.json
- source_page
- https://www.kaggle.com/benchmarks/cohere-labs/global-mmlu-lite-french/versions/1
- measurements
- name
- French
- unit
- score
- scale
- 1
- metric
- Global MMLU Lite French · French
- result
- resultCase
- numericResult
- booleanResult
- false
- numericResult
- value
- 0.9425
- standardDeviation
- 0
- confidenceInterval
- 0.0228135408783901
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0228135408783901
- hasUnevenConfidenceInterval
- false
- taskVersionId
- 0
- evaluationDate
- 2025-10-30T00:00:00Z
- hasBooleanResult
- false
- hasNumericResult
- true
- hasTaskVersionId
- false
- hasEvaluationDate
- true
- numericResultNullable
- value
- 0.9425
- standardDeviation
- 0
- confidenceInterval
- 0.0228135408783901
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0228135408783901
- hasUnevenConfidenceInterval
- false
- hasNumericResultPublic
- false
- hasNumericResultPrivate
- false
- task_id
- 41752
- task_url
- /benchmarks/tasks/benchmarkler/global-mmlu-lite-french
- sort_order
- DESCENDING
- source_name
- French
- display_type
- rawValues
- task_version
- 2
- task_display_type
- RAW_VALUES
- normalization_status
- normalized
- published_at
- 2025-10-17T15:30:43.548947500Z
- candidate_class
- modelVersion
- source_revision
- b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
- publisher_anchor
- google/gemini-2.5-pro
- metric_population
- 41
- benchmark_evaluation
- true
- task_metadata_source
- benchmarkVersion
- candidate_metadata_source
- benchmarkVersionMappings
- evaluated at
- 2025-10-30T00:00:00.000Z
- source model id
- Gemini 2.5 Pro
- confidence lower
- 0.91968645912161
- confidence upper
- 0.96531354087839
Original result ↗0.941
- source id
- kaggleBenchmarks
- leaderboard name
- Global MMLU Lite Culturally Agnostic · CA Score
- dimension
- reasoning
- metric
- Global MMLU Lite Culturally Agnostic · CA Score
- rank
- 10
- aggregation scope
- kaggle
- true
- status
- published
- license
- Apache-2.0
- direction
- higherIsBetter
- conditions
- task
- id
- 41744
- name
- CA Score
- version
- 2
- sort_order
- DESCENDING
- display_type
- rawValues
- benchmark
- id
- 209
- name
- Global MMLU Lite Culturally Agnostic
- type
- INDIVIDUAL
- owner
- cohere-labs
- version
- 1
- categories
- multilingual and cross-lingual capabilities
- general knowledge and reasoning
- version_id
- 188
- maintainer_tier
- Kaggle-maintained
- maintenance_level
- KAGGLE_MAINTAINED
- candidate
- candidate_class
- modelVersion
- record_kind
- aggregate
- attribution
- Kaggle · cohere-labs · Global MMLU Lite Culturally Agnostic
- license_url
- https://www.apache.org/licenses/LICENSE-2.0
- metric_kind
- Global MMLU Lite Culturally Agnostic · CA Score
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- 188/leaderboard.json
- source_page
- https://www.kaggle.com/benchmarks/cohere-labs/global-mmlu-lite-ca/versions/1
- measurements
- name
- CA Score
- unit
- score
- scale
- 1
- metric
- Global MMLU Lite Culturally Agnostic · CA Score
- result
- resultCase
- numericResult
- booleanResult
- false
- numericResult
- value
- 0.9406249999999999
- standardDeviation
- 0
- confidenceInterval
- 0
- hasStandardDeviation
- false
- hasConfidenceInterval
- false
- hasUnevenConfidenceInterval
- false
- taskVersionId
- 0
- evaluationDate
- 2025-11-03T22:31:06.2566667Z
- hasBooleanResult
- false
- hasNumericResult
- true
- hasTaskVersionId
- false
- hasEvaluationDate
- true
- numericResultNullable
- value
- 0.9406249999999999
- standardDeviation
- 0
- confidenceInterval
- 0
- hasStandardDeviation
- false
- hasConfidenceInterval
- false
- hasUnevenConfidenceInterval
- false
- hasNumericResultPublic
- false
- hasNumericResultPrivate
- false
- task_id
- 41744
- task_url
- /benchmarks/tasks/benchmarkler/global-mmlu-lite-ca-score
- sort_order
- DESCENDING
- source_name
- CA Score
- display_type
- rawValues
- task_version
- 2
- task_display_type
- RAW_VALUES
- normalization_status
- normalized
- published_at
- 2025-11-10T16:52:12.590864700Z
- candidate_class
- modelVersion
- source_revision
- b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
- publisher_anchor
- google/gemini-2.5-pro
- metric_population
- 41
- benchmark_evaluation
- true
- task_metadata_source
- benchmarkVersion
- candidate_metadata_source
- benchmarkVersionMappings
- evaluated at
- 2025-11-03T22:31:06.256Z
- source model id
- Gemini 2.5 Pro
Original result ↗57.518%
- source id
- kaggleBenchmarks
- leaderboard name
- AssetOpsBench · Consistency
- dimension
- reasoning
- metric
- AssetOpsBench · Consistency
- rank
- 10
- aggregation scope
- kaggle
- true
- status
- published
- license
- Apache-2.0
- direction
- higherIsBetter
- conditions
- task
- id
- 1045
- name
- Consistency
- version
- 1
- sort_order
- DESCENDING
- display_type
- percentages
- benchmark
- id
- 243
- name
- AssetOpsBench
- type
- INDIVIDUAL
- owner
- ibm-research
- version
- 1
- categories
- evaluation
- general knowledge and reasoning
- version_id
- 222
- maintainer_tier
- Kaggle-maintained
- maintenance_level
- KAGGLE_MAINTAINED
- candidate
- candidate_class
- modelVersion
- record_kind
- aggregate
- attribution
- Kaggle · ibm-research · AssetOpsBench
- license_url
- https://www.apache.org/licenses/LICENSE-2.0
- metric_kind
- AssetOpsBench · Consistency
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- 222/leaderboard.json
- source_page
- https://www.kaggle.com/benchmarks/ibm-research/asset-ops-bench/versions/1
- measurements
- name
- Consistency
- unit
- percent
- scale
- 1
- metric
- AssetOpsBench · Consistency
- result
- resultCase
- numericResult
- booleanResult
- false
- numericResult
- value
- 0.5751781027371579
- standardDeviation
- 0
- confidenceInterval
- 0
- hasStandardDeviation
- false
- hasConfidenceInterval
- false
- hasUnevenConfidenceInterval
- false
- taskVersionId
- 0
- evaluationDate
- 2025-11-24T03:35:15.458771Z
- hasBooleanResult
- false
- hasNumericResult
- true
- hasTaskVersionId
- false
- hasEvaluationDate
- true
- numericResultNullable
- value
- 0.5751781027371579
- standardDeviation
- 0
- confidenceInterval
- 0
- hasStandardDeviation
- false
- hasConfidenceInterval
- false
- hasUnevenConfidenceInterval
- false
- hasNumericResultPublic
- false
- hasNumericResultPrivate
- false
- task_id
- 1045
- task_url
- /benchmarks/tasks/benchmarkler/failuresensoriq-consistency-benchmark-task
- sort_order
- DESCENDING
- source_name
- Consistency
- display_type
- percentages
- task_version
- 1
- task_display_type
- RAW_VALUES
- normalization_status
- normalized
- published_at
- 2025-12-01T22:15:26.261663300Z
- candidate_class
- modelVersion
- source_revision
- b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
- publisher_anchor
- google/gemini-2.5-pro
- metric_population
- 25
- benchmark_evaluation
- true
- task_metadata_source
- benchmarkVersion
- candidate_metadata_source
- benchmarkVersionMappings
- evaluated at
- 2025-11-24T03:35:15.458Z
- source model id
- Gemini 2.5 Pro
Original result ↗1,446.579
- source id
- lmarena
- leaderboard name
- LMArena Text Factuality German
- dimension
- factuality
- metric
- Arena rating
- rank
- 12
- sample size
- 583
- evaluated at
- 2026-11-09T00:00:00.000Z
- source model id
- Gemini 2.5 Pro
- confidence lower
- 1422.969491280298
- confidence upper
- 1470.188753874323
Original result ↗0.928
- source id
- kaggleBenchmarks
- leaderboard name
- Global MMLU Lite Korean · Korean
- dimension
- reasoning
- metric
- Global MMLU Lite Korean · Korean
- rank
- 12
- aggregation scope
- kaggle
- true
- status
- published
- license
- Apache-2.0
- direction
- higherIsBetter
- conditions
- task
- id
- 41748
- name
- Korean
- version
- 2
- sort_order
- DESCENDING
- display_type
- rawValues
- benchmark
- id
- 161
- name
- Global MMLU Lite Korean
- type
- INDIVIDUAL
- owner
- cohere-labs
- version
- 1
- categories
- multilingual and cross-lingual capabilities
- general knowledge and reasoning
- version_id
- 143
- maintainer_tier
- Kaggle-maintained
- maintenance_level
- KAGGLE_MAINTAINED
- candidate
- candidate_class
- modelVersion
- record_kind
- aggregate
- attribution
- Kaggle · cohere-labs · Global MMLU Lite Korean
- license_url
- https://www.apache.org/licenses/LICENSE-2.0
- metric_kind
- Global MMLU Lite Korean · Korean
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- 143/leaderboard.json
- source_page
- https://www.kaggle.com/benchmarks/cohere-labs/global-mmlu-lite-korean/versions/1
- measurements
- name
- Korean
- unit
- score
- scale
- 1
- metric
- Global MMLU Lite Korean · Korean
- result
- resultCase
- numericResult
- booleanResult
- false
- numericResult
- value
- 0.9275
- standardDeviation
- 0
- confidenceInterval
- 0.0254123049217328
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0254123049217328
- hasUnevenConfidenceInterval
- false
- taskVersionId
- 0
- evaluationDate
- 2025-10-30T00:00:00Z
- hasBooleanResult
- false
- hasNumericResult
- true
- hasTaskVersionId
- false
- hasEvaluationDate
- true
- numericResultNullable
- value
- 0.9275
- standardDeviation
- 0
- confidenceInterval
- 0.0254123049217328
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0254123049217328
- hasUnevenConfidenceInterval
- false
- hasNumericResultPublic
- false
- hasNumericResultPrivate
- false
- task_id
- 41748
- task_url
- /benchmarks/tasks/benchmarkler/global-mmlu-lite-korean
- sort_order
- DESCENDING
- source_name
- Korean
- display_type
- rawValues
- task_version
- 2
- task_display_type
- RAW_VALUES
- normalization_status
- normalized
- published_at
- 2025-10-17T15:30:33.197926100Z
- candidate_class
- modelVersion
- source_revision
- b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
- publisher_anchor
- google/gemini-2.5-pro
- metric_population
- 41
- benchmark_evaluation
- true
- task_metadata_source
- benchmarkVersion
- candidate_metadata_source
- benchmarkVersionMappings
- evaluated at
- 2025-10-30T00:00:00.000Z
- source model id
- Gemini 2.5 Pro
- confidence lower
- 0.9020876950782671
- confidence upper
- 0.9529123049217328
Original result ↗0.928
- source id
- kaggleBenchmarks
- leaderboard name
- Global MMLU Lite Bengali · Bengali
- dimension
- reasoning
- metric
- Global MMLU Lite Bengali · Bengali
- rank
- 12
- aggregation scope
- kaggle
- true
- status
- published
- license
- Apache-2.0
- direction
- higherIsBetter
- conditions
- task
- id
- 41751
- name
- Bengali
- version
- 2
- sort_order
- DESCENDING
- display_type
- rawValues
- benchmark
- id
- 154
- name
- Global MMLU Lite Bengali
- type
- INDIVIDUAL
- owner
- cohere-labs
- version
- 1
- categories
- multilingual and cross-lingual capabilities
- general knowledge and reasoning
- version_id
- 136
- maintainer_tier
- Kaggle-maintained
- maintenance_level
- KAGGLE_MAINTAINED
- candidate
- candidate_class
- modelVersion
- record_kind
- aggregate
- attribution
- Kaggle · cohere-labs · Global MMLU Lite Bengali
- license_url
- https://www.apache.org/licenses/LICENSE-2.0
- metric_kind
- Global MMLU Lite Bengali · Bengali
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- 136/leaderboard.json
- source_page
- https://www.kaggle.com/benchmarks/cohere-labs/global-mmlu-lite-bengali/versions/1
- measurements
- name
- Bengali
- unit
- score
- scale
- 1
- metric
- Global MMLU Lite Bengali · Bengali
- result
- resultCase
- numericResult
- booleanResult
- false
- numericResult
- value
- 0.9275
- standardDeviation
- 0
- confidenceInterval
- 0.0254123049217328
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0254123049217328
- hasUnevenConfidenceInterval
- false
- taskVersionId
- 0
- evaluationDate
- 2025-10-30T00:00:00Z
- hasBooleanResult
- false
- hasNumericResult
- true
- hasTaskVersionId
- false
- hasEvaluationDate
- true
- numericResultNullable
- value
- 0.9275
- standardDeviation
- 0
- confidenceInterval
- 0.0254123049217328
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0254123049217328
- hasUnevenConfidenceInterval
- false
- hasNumericResultPublic
- false
- hasNumericResultPrivate
- false
- task_id
- 41751
- task_url
- /benchmarks/tasks/benchmarkler/global-mmlu-lite-bengali
- sort_order
- DESCENDING
- source_name
- Bengali
- display_type
- rawValues
- task_version
- 2
- task_display_type
- RAW_VALUES
- normalization_status
- normalized
- published_at
- 2025-10-17T15:30:47.931282Z
- candidate_class
- modelVersion
- source_revision
- b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
- publisher_anchor
- google/gemini-2.5-pro
- metric_population
- 41
- benchmark_evaluation
- true
- task_metadata_source
- benchmarkVersion
- candidate_metadata_source
- benchmarkVersionMappings
- evaluated at
- 2025-10-30T00:00:00.000Z
- source model id
- Gemini 2.5 Pro
- confidence lower
- 0.9020876950782671
- confidence upper
- 0.9529123049217328
Original result ↗0.928
- source id
- kaggleBenchmarks
- leaderboard name
- Global MMLU Lite Hindi · Hindi
- dimension
- reasoning
- metric
- Global MMLU Lite Hindi · Hindi
- rank
- 12
- aggregation scope
- kaggle
- true
- status
- published
- license
- Apache-2.0
- direction
- higherIsBetter
- conditions
- task
- id
- 41742
- name
- Hindi
- version
- 2
- sort_order
- DESCENDING
- display_type
- rawValues
- benchmark
- id
- 157
- name
- Global MMLU Lite Hindi
- type
- INDIVIDUAL
- owner
- cohere-labs
- version
- 1
- categories
- multilingual and cross-lingual capabilities
- general knowledge and reasoning
- version_id
- 139
- maintainer_tier
- Kaggle-maintained
- maintenance_level
- KAGGLE_MAINTAINED
- candidate
- candidate_class
- modelVersion
- record_kind
- aggregate
- attribution
- Kaggle · cohere-labs · Global MMLU Lite Hindi
- license_url
- https://www.apache.org/licenses/LICENSE-2.0
- metric_kind
- Global MMLU Lite Hindi · Hindi
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- 139/leaderboard.json
- source_page
- https://www.kaggle.com/benchmarks/cohere-labs/global-mmlu-lite-hindi/versions/1
- measurements
- name
- Hindi
- unit
- score
- scale
- 1
- metric
- Global MMLU Lite Hindi · Hindi
- result
- resultCase
- numericResult
- booleanResult
- false
- numericResult
- value
- 0.9275
- standardDeviation
- 0
- confidenceInterval
- 0.0254123049217328
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0254123049217328
- hasUnevenConfidenceInterval
- false
- taskVersionId
- 0
- evaluationDate
- 2025-10-30T00:00:00Z
- hasBooleanResult
- false
- hasNumericResult
- true
- hasTaskVersionId
- false
- hasEvaluationDate
- true
- numericResultNullable
- value
- 0.9275
- standardDeviation
- 0
- confidenceInterval
- 0.0254123049217328
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0254123049217328
- hasUnevenConfidenceInterval
- false
- hasNumericResultPublic
- false
- hasNumericResultPrivate
- false
- task_id
- 41742
- task_url
- /benchmarks/tasks/benchmarkler/global-mmlu-lite-hindi
- sort_order
- DESCENDING
- source_name
- Hindi
- display_type
- rawValues
- task_version
- 2
- task_display_type
- RAW_VALUES
- normalization_status
- normalized
- published_at
- 2025-10-17T15:30:41.662768400Z
- candidate_class
- modelVersion
- source_revision
- b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
- publisher_anchor
- google/gemini-2.5-pro
- metric_population
- 41
- benchmark_evaluation
- true
- task_metadata_source
- benchmarkVersion
- candidate_metadata_source
- benchmarkVersionMappings
- evaluated at
- 2025-10-30T00:00:00.000Z
- source model id
- Gemini 2.5 Pro
- confidence lower
- 0.9020876950782671
- confidence upper
- 0.9529123049217328
Original result ↗0.925
- source id
- kaggleBenchmarks
- leaderboard name
- Global MMLU Lite Indonesian · Indonesian
- dimension
- reasoning
- metric
- Global MMLU Lite Indonesian · Indonesian
- rank
- 12
- aggregation scope
- kaggle
- true
- status
- published
- license
- Apache-2.0
- direction
- higherIsBetter
- conditions
- task
- id
- 41740
- name
- Indonesian
- version
- 2
- sort_order
- DESCENDING
- display_type
- rawValues
- benchmark
- id
- 158
- name
- Global MMLU Lite Indonesian
- type
- INDIVIDUAL
- owner
- cohere-labs
- version
- 1
- categories
- multilingual and cross-lingual capabilities
- general knowledge and reasoning
- version_id
- 140
- maintainer_tier
- Kaggle-maintained
- maintenance_level
- KAGGLE_MAINTAINED
- candidate
- candidate_class
- modelVersion
- record_kind
- aggregate
- attribution
- Kaggle · cohere-labs · Global MMLU Lite Indonesian
- license_url
- https://www.apache.org/licenses/LICENSE-2.0
- metric_kind
- Global MMLU Lite Indonesian · Indonesian
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- 140/leaderboard.json
- source_page
- https://www.kaggle.com/benchmarks/cohere-labs/global-mmlu-lite-indonesian/versions/1
- measurements
- name
- Indonesian
- unit
- score
- scale
- 1
- metric
- Global MMLU Lite Indonesian · Indonesian
- result
- resultCase
- numericResult
- booleanResult
- false
- numericResult
- value
- 0.925
- standardDeviation
- 0
- confidenceInterval
- 0.0258118773864695
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0258118773864695
- hasUnevenConfidenceInterval
- false
- taskVersionId
- 0
- evaluationDate
- 2025-10-30T00:00:00Z
- hasBooleanResult
- false
- hasNumericResult
- true
- hasTaskVersionId
- false
- hasEvaluationDate
- true
- numericResultNullable
- value
- 0.925
- standardDeviation
- 0
- confidenceInterval
- 0.0258118773864695
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0258118773864695
- hasUnevenConfidenceInterval
- false
- hasNumericResultPublic
- false
- hasNumericResultPrivate
- false
- task_id
- 41740
- task_url
- /benchmarks/tasks/benchmarkler/global-mmlu-lite-indonesian
- sort_order
- DESCENDING
- source_name
- Indonesian
- display_type
- rawValues
- task_version
- 2
- task_display_type
- RAW_VALUES
- normalization_status
- normalized
- published_at
- 2025-10-17T15:30:39.240452500Z
- candidate_class
- modelVersion
- source_revision
- b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
- publisher_anchor
- google/gemini-2.5-pro
- metric_population
- 41
- benchmark_evaluation
- true
- task_metadata_source
- benchmarkVersion
- candidate_metadata_source
- benchmarkVersionMappings
- evaluated at
- 2025-10-30T00:00:00.000Z
- source model id
- Gemini 2.5 Pro
- confidence lower
- 0.8991881226135305
- confidence upper
- 0.9508118773864696
Original result ↗62.992%
- source id
- kaggleBenchmarks
- leaderboard name
- AssetOpsBench · Pert. Simple Accuracy
- dimension
- reasoning
- metric
- AssetOpsBench · Pert. Simple Accuracy
- rank
- 12
- aggregation scope
- kaggle
- true
- status
- published
- license
- Apache-2.0
- direction
- higherIsBetter
- conditions
- task
- id
- 1043
- name
- Pert. Simple Accuracy
- version
- 1
- sort_order
- DESCENDING
- display_type
- percentages
- benchmark
- id
- 243
- name
- AssetOpsBench
- type
- INDIVIDUAL
- owner
- ibm-research
- version
- 1
- categories
- evaluation
- general knowledge and reasoning
- version_id
- 222
- maintainer_tier
- Kaggle-maintained
- maintenance_level
- KAGGLE_MAINTAINED
- candidate
- candidate_class
- modelVersion
- record_kind
- aggregate
- attribution
- Kaggle · ibm-research · AssetOpsBench
- license_url
- https://www.apache.org/licenses/LICENSE-2.0
- metric_kind
- AssetOpsBench · Pert. Simple Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- 222/leaderboard.json
- source_page
- https://www.kaggle.com/benchmarks/ibm-research/asset-ops-bench/versions/1
- measurements
- name
- Pert. Simple Accuracy
- unit
- percent
- scale
- 1
- metric
- AssetOpsBench · Pert. Simple Accuracy
- result
- resultCase
- numericResult
- booleanResult
- false
- numericResult
- value
- 0.6299212598425197
- standardDeviation
- 0
- confidenceInterval
- 0
- hasStandardDeviation
- false
- hasConfidenceInterval
- false
- hasUnevenConfidenceInterval
- false
- taskVersionId
- 0
- evaluationDate
- 2025-11-23T14:35:37.103515Z
- hasBooleanResult
- false
- hasNumericResult
- true
- hasTaskVersionId
- false
- hasEvaluationDate
- true
- numericResultNullable
- value
- 0.6299212598425197
- standardDeviation
- 0
- confidenceInterval
- 0
- hasStandardDeviation
- false
- hasConfidenceInterval
- false
- hasUnevenConfidenceInterval
- false
- hasNumericResultPublic
- false
- hasNumericResultPrivate
- false
- task_id
- 1043
- task_url
- /benchmarks/tasks/benchmarkler/failuresensoriq-perturbationsimple-benchmark-task
- sort_order
- DESCENDING
- source_name
- Pert. Simple Accuracy
- display_type
- percentages
- task_version
- 1
- task_display_type
- RAW_VALUES
- normalization_status
- normalized
- published_at
- 2025-12-01T22:15:26.261663300Z
- candidate_class
- modelVersion
- source_revision
- b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
- publisher_anchor
- google/gemini-2.5-pro
- metric_population
- 25
- benchmark_evaluation
- true
- task_metadata_source
- benchmarkVersion
- candidate_metadata_source
- benchmarkVersionMappings
- evaluated at
- 2025-11-23T14:35:37.103Z
- source model id
- Gemini 2.5 Pro
Original result ↗0.94
- source id
- kaggleBenchmarks
- leaderboard name
- Global MMLU Lite Spanish · Spanish
- dimension
- reasoning
- metric
- Global MMLU Lite Spanish · Spanish
- rank
- 13
- aggregation scope
- kaggle
- true
- status
- published
- license
- Apache-2.0
- direction
- higherIsBetter
- conditions
- task
- id
- 41741
- name
- Spanish
- version
- 2
- sort_order
- DESCENDING
- display_type
- rawValues
- benchmark
- id
- 163
- name
- Global MMLU Lite Spanish
- type
- INDIVIDUAL
- owner
- cohere-labs
- version
- 1
- categories
- multilingual and cross-lingual capabilities
- general knowledge and reasoning
- version_id
- 145
- maintainer_tier
- Kaggle-maintained
- maintenance_level
- KAGGLE_MAINTAINED
- candidate
- candidate_class
- modelVersion
- record_kind
- aggregate
- attribution
- Kaggle · cohere-labs · Global MMLU Lite Spanish
- license_url
- https://www.apache.org/licenses/LICENSE-2.0
- metric_kind
- Global MMLU Lite Spanish · Spanish
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- 145/leaderboard.json
- source_page
- https://www.kaggle.com/benchmarks/cohere-labs/global-mmlu-lite-spanish/versions/1
- measurements
- name
- Spanish
- unit
- score
- scale
- 1
- metric
- Global MMLU Lite Spanish · Spanish
- result
- resultCase
- numericResult
- booleanResult
- false
- numericResult
- value
- 0.94
- standardDeviation
- 0
- confidenceInterval
- 0.0232732828307025
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0232732828307025
- hasUnevenConfidenceInterval
- false
- taskVersionId
- 0
- evaluationDate
- 2025-10-30T00:00:00Z
- hasBooleanResult
- false
- hasNumericResult
- true
- hasTaskVersionId
- false
- hasEvaluationDate
- true
- numericResultNullable
- value
- 0.94
- standardDeviation
- 0
- confidenceInterval
- 0.0232732828307025
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0232732828307025
- hasUnevenConfidenceInterval
- false
- hasNumericResultPublic
- false
- hasNumericResultPrivate
- false
- task_id
- 41741
- task_url
- /benchmarks/tasks/benchmarkler/global-mmlu-lite-spanish
- sort_order
- DESCENDING
- source_name
- Spanish
- display_type
- rawValues
- task_version
- 2
- task_display_type
- RAW_VALUES
- normalization_status
- normalized
- published_at
- 2025-10-17T15:30:28.873808200Z
- candidate_class
- modelVersion
- source_revision
- b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
- publisher_anchor
- google/gemini-2.5-pro
- metric_population
- 41
- benchmark_evaluation
- true
- task_metadata_source
- benchmarkVersion
- candidate_metadata_source
- benchmarkVersionMappings
- evaluated at
- 2025-10-30T00:00:00.000Z
- source model id
- Gemini 2.5 Pro
- confidence lower
- 0.9167267171692974
- confidence upper
- 0.9632732828307025
Original result ↗0.938
- source id
- kaggleBenchmarks
- leaderboard name
- Global MMLU Lite Japanese · Japanese
- dimension
- reasoning
- metric
- Global MMLU Lite Japanese · Japanese
- rank
- 13
- aggregation scope
- kaggle
- true
- status
- published
- license
- Apache-2.0
- direction
- higherIsBetter
- conditions
- task
- id
- 41745
- name
- Japanese
- version
- 2
- sort_order
- DESCENDING
- display_type
- rawValues
- benchmark
- id
- 160
- name
- Global MMLU Lite Japanese
- type
- INDIVIDUAL
- owner
- cohere-labs
- version
- 1
- categories
- multilingual and cross-lingual capabilities
- general knowledge and reasoning
- version_id
- 142
- maintainer_tier
- Kaggle-maintained
- maintenance_level
- KAGGLE_MAINTAINED
- candidate
- candidate_class
- modelVersion
- record_kind
- aggregate
- attribution
- Kaggle · cohere-labs · Global MMLU Lite Japanese
- license_url
- https://www.apache.org/licenses/LICENSE-2.0
- metric_kind
- Global MMLU Lite Japanese · Japanese
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- 142/leaderboard.json
- source_page
- https://www.kaggle.com/benchmarks/cohere-labs/global-mmlu-lite-japanese/versions/1
- measurements
- name
- Japanese
- unit
- score
- scale
- 1
- metric
- Global MMLU Lite Japanese · Japanese
- result
- resultCase
- numericResult
- booleanResult
- false
- numericResult
- value
- 0.9375
- standardDeviation
- 0
- confidenceInterval
- 0.0237215870977811
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0237215870977811
- hasUnevenConfidenceInterval
- false
- taskVersionId
- 0
- evaluationDate
- 2025-10-30T00:00:00Z
- hasBooleanResult
- false
- hasNumericResult
- true
- hasTaskVersionId
- false
- hasEvaluationDate
- true
- numericResultNullable
- value
- 0.9375
- standardDeviation
- 0
- confidenceInterval
- 0.0237215870977811
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0237215870977811
- hasUnevenConfidenceInterval
- false
- hasNumericResultPublic
- false
- hasNumericResultPrivate
- false
- task_id
- 41745
- task_url
- /benchmarks/tasks/benchmarkler/global-mmlu-lite-japanese
- sort_order
- DESCENDING
- source_name
- Japanese
- display_type
- rawValues
- task_version
- 2
- task_display_type
- RAW_VALUES
- normalization_status
- normalized
- published_at
- 2025-10-17T15:30:34.896663Z
- candidate_class
- modelVersion
- source_revision
- b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
- publisher_anchor
- google/gemini-2.5-pro
- metric_population
- 41
- benchmark_evaluation
- true
- task_metadata_source
- benchmarkVersion
- candidate_metadata_source
- benchmarkVersionMappings
- evaluated at
- 2025-10-30T00:00:00.000Z
- source model id
- Gemini 2.5 Pro
- confidence lower
- 0.913778412902219
- confidence upper
- 0.961221587097781
Original result ↗0.935
- source id
- kaggleBenchmarks
- leaderboard name
- Global MMLU Lite Italian · Italian
- dimension
- reasoning
- metric
- Global MMLU Lite Italian · Italian
- rank
- 14
- aggregation scope
- kaggle
- true
- status
- published
- license
- Apache-2.0
- direction
- higherIsBetter
- conditions
- task
- id
- 41739
- name
- Italian
- version
- 2
- sort_order
- DESCENDING
- display_type
- rawValues
- benchmark
- id
- 159
- name
- Global MMLU Lite Italian
- type
- INDIVIDUAL
- owner
- cohere-labs
- version
- 1
- categories
- multilingual and cross-lingual capabilities
- general knowledge and reasoning
- version_id
- 141
- maintainer_tier
- Kaggle-maintained
- maintenance_level
- KAGGLE_MAINTAINED
- candidate
- candidate_class
- modelVersion
- record_kind
- aggregate
- attribution
- Kaggle · cohere-labs · Global MMLU Lite Italian
- license_url
- https://www.apache.org/licenses/LICENSE-2.0
- metric_kind
- Global MMLU Lite Italian · Italian
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- 141/leaderboard.json
- source_page
- https://www.kaggle.com/benchmarks/cohere-labs/global-mmlu-lite-italian/versions/1
- measurements
- name
- Italian
- unit
- score
- scale
- 1
- metric
- Global MMLU Lite Italian · Italian
- result
- resultCase
- numericResult
- booleanResult
- false
- numericResult
- value
- 0.935
- standardDeviation
- 0
- confidenceInterval
- 0.0241590904127041
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0241590904127041
- hasUnevenConfidenceInterval
- false
- taskVersionId
- 0
- evaluationDate
- 2025-10-30T00:00:00Z
- hasBooleanResult
- false
- hasNumericResult
- true
- hasTaskVersionId
- false
- hasEvaluationDate
- true
- numericResultNullable
- value
- 0.935
- standardDeviation
- 0
- confidenceInterval
- 0.0241590904127041
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0241590904127041
- hasUnevenConfidenceInterval
- false
- hasNumericResultPublic
- false
- hasNumericResultPrivate
- false
- task_id
- 41739
- task_url
- /benchmarks/tasks/benchmarkler/global-mmlu-lite-italian
- sort_order
- DESCENDING
- source_name
- Italian
- display_type
- rawValues
- task_version
- 2
- task_display_type
- RAW_VALUES
- normalization_status
- normalized
- published_at
- 2025-10-17T15:30:36.963753600Z
- candidate_class
- modelVersion
- source_revision
- b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
- publisher_anchor
- google/gemini-2.5-pro
- metric_population
- 41
- benchmark_evaluation
- true
- task_metadata_source
- benchmarkVersion
- candidate_metadata_source
- benchmarkVersionMappings
- evaluated at
- 2025-10-30T00:00:00.000Z
- source model id
- Gemini 2.5 Pro
- confidence lower
- 0.9108409095872959
- confidence upper
- 0.9591590904127042
Original result ↗Published prices
1–20 / 60
$0.13
- Inference provider
- cloudflareWorkersAI
- Platform
- Google Vertex AI
- Unit
- cache_read_tokens_per_million
- Currency
- USD
- Billing label
- Cached input tokens (per 1M)
- Pricing via
- Cloudflare AI Gateway
$1.25
- Inference provider
- cloudflareWorkersAI
- Platform
- Google Vertex AI
- Unit
- input_tokens_per_million
- Currency
- USD
- Billing label
- Input tokens (per 1M)
- Pricing via
- Cloudflare AI Gateway
$10.00
- Inference provider
- cloudflareWorkersAI
- Platform
- Google Vertex AI
- Unit
- output_tokens_per_million
- Currency
- USD
- Billing label
- Output tokens (per 1M)
- Pricing via
- Cloudflare AI Gateway
$1.25
- Inference provider
- deepInfra
- Platform
- DeepInfra
- Unit
- input_tokens_per_million
- Currency
- USD
$10.00
- Inference provider
- deepInfra
- Platform
- DeepInfra
- Unit
- output_tokens_per_million
- Currency
- USD
$0.13
- Inference provider
- openRouter
- Platform
- OpenRouter
- Unit
- cache_read_tokens_per_million
- Currency
- USD
$0.38
- Inference provider
- openRouter
- Platform
- OpenRouter
- Unit
- cache_write_tokens_per_million
- Currency
- USD
$0.000001
- Inference provider
- openRouter
- Platform
- OpenRouter
- Unit
- image
- Currency
- USD
$1.25
- Inference provider
- openRouter
- Platform
- OpenRouter
- Unit
- input_tokens_per_million
- Currency
- USD
$10.00
- Inference provider
- openRouter
- Platform
- OpenRouter
- Unit
- output_tokens_per_million
- Currency
- USD
$0.01
- Inference provider
- openRouter
- Platform
- OpenRouter
- Unit
- web_search
- Currency
- USD
$0.25
- Inference provider
- vercelAIGatewayCatalog
- Platform
- Vercel AI Gateway
- Unit
- cache_read_tokens_per_million
- Currency
- USD
- Billing conditions
- model_type
- language
- quote_scope
- gateway_catalog
- billing_scope
- Vercel public catalog; endpoint prices may differ
- context_tokens
- min_inclusive
- 200001
- Billing label
- Cached input / 1M
- Pricing via
- Vercel AI Gateway
$0.13
- Inference provider
- vercelAIGatewayCatalog
- Platform
- Vercel AI Gateway
- Unit
- cache_read_tokens_per_million
- Currency
- USD
- Billing conditions
- model_type
- language
- quote_scope
- gateway_catalog
- billing_scope
- Vercel public catalog; endpoint prices may differ
- context_tokens
- max_exclusive
- 200001
- min_inclusive
- 0
- Billing label
- Cached input / 1M
- Pricing via
- Vercel AI Gateway
$0.22
- Inference provider
- vercelAIGatewayCatalog
- Platform
- Vercel AI Gateway
- Unit
- cache_read_tokens_per_million
- Currency
- USD
- Billing conditions
- model_type
- language
- quote_scope
- gateway_catalog
- service_tier
- priority
- billing_scope
- Vercel public catalog; endpoint prices may differ
- Billing label
- Cached input / 1M
- Pricing via
- Vercel AI Gateway
$0.25
- Inference provider
- vercelAIGatewayCatalog
- Platform
- Vercel AI Gateway
- Unit
- cache_read_tokens_per_million
- Currency
- USD
- Billing conditions
- region
- eu
- model_type
- language
- quote_scope
- gateway_catalog
- billing_scope
- Vercel public catalog; endpoint prices may differ
- context_tokens
- min_inclusive
- 200001
- Billing label
- Cached input / 1M
- Pricing via
- Vercel AI Gateway
$0.13
- Inference provider
- vercelAIGatewayCatalog
- Platform
- Vercel AI Gateway
- Unit
- cache_read_tokens_per_million
- Currency
- USD
- Billing conditions
- region
- eu
- model_type
- language
- quote_scope
- gateway_catalog
- billing_scope
- Vercel public catalog; endpoint prices may differ
- context_tokens
- max_exclusive
- 200001
- min_inclusive
- 0
- Billing label
- Cached input / 1M
- Pricing via
- Vercel AI Gateway
$0.25
- Inference provider
- vercelAIGatewayCatalog
- Platform
- Vercel AI Gateway
- Unit
- cache_read_tokens_per_million
- Currency
- USD
- Billing conditions
- region
- us
- model_type
- language
- quote_scope
- gateway_catalog
- billing_scope
- Vercel public catalog; endpoint prices may differ
- context_tokens
- min_inclusive
- 200001
- Billing label
- Cached input / 1M
- Pricing via
- Vercel AI Gateway
$0.13
- Inference provider
- vercelAIGatewayCatalog
- Platform
- Vercel AI Gateway
- Unit
- cache_read_tokens_per_million
- Currency
- USD
- Billing conditions
- region
- us
- model_type
- language
- quote_scope
- gateway_catalog
- billing_scope
- Vercel public catalog; endpoint prices may differ
- context_tokens
- max_exclusive
- 200001
- min_inclusive
- 0
- Billing label
- Cached input / 1M
- Pricing via
- Vercel AI Gateway
$2.50
- Inference provider
- vercelAIGatewayCatalog
- Platform
- Vercel AI Gateway
- Unit
- input_tokens_per_million
- Currency
- USD
- Billing conditions
- model_type
- language
- quote_scope
- gateway_catalog
- billing_scope
- Vercel public catalog; endpoint prices may differ
- context_tokens
- min_inclusive
- 200001
- Billing label
- Input tokens / 1M
- Pricing via
- Vercel AI Gateway
$1.25
- Inference provider
- vercelAIGatewayCatalog
- Platform
- Vercel AI Gateway
- Unit
- input_tokens_per_million
- Currency
- USD
- Billing conditions
- model_type
- language
- quote_scope
- gateway_catalog
- billing_scope
- Vercel public catalog; endpoint prices may differ
- context_tokens
- max_exclusive
- 200001
- min_inclusive
- 0
- Billing label
- Input tokens / 1M
- Pricing via
- Vercel AI Gateway
Provider specifications & access
1–4 / 4
- source model id
- gemini-2.5-pro
- hosting
- gateway
- context length
- 1048576
- max output tokens
- 65535
- capabilities
- reasoning
- structuredOutputs
- toolCalling
- visionInput
- webSearch
- source model id
- google/gemini-2.5-pro
- hosting
- third-party
- context length
- 1000000
- max output tokens
- 65536
- source model id
- google/gemini-2.5-pro
- hosting
- vercel-gateway
- context length
- 1048576
- max output tokens
- 65535
- capabilities
- promptCaching
- reasoning
- structuredOutputs
- toolCalling
- visionInput
- webSearch
- source model id
- publishers/google/models/gemini-2.5-pro@default
- hosting
- vertex-managed
- context length
- 1048576
- max output tokens
- 65536
- capabilities
- batch
- multimodal
- promptCaching
- reasoning
- structuredOutputs
- toolCalling
- visionInput
Safety assessments
1–1 / 1
41 / 100
- source id
- stanfordFMTI
- canonical model id
- google/gemini-2.5-pro
- taxonomy
- transparency
- hazard category
- transparency
- hazard name
- Organization transparency
- policy area
- Google
- assessment type
- organization-level disclosure
- metric
- disclosure_score
- result label
- 41 / 100
- sample size
- 100
- updated at
- 2026-11-09T18:08:05.227Z
Source specification observations
1–10 / 10
- field
- context_length
- value
- 1000000
- normalized value
- 1000000
- scope
- source-model
- source id
- cloudflareWorkersAI
- confidence
- 1
- field
- context_length
- value
- 1048576
- normalized value
- 1048576
- scope
- source-model
- source id
- openRouter
- confidence
- 1
- field
- context_length
- value
- 1048576
- normalized value
- 1048576
- scope
- source-model
- source id
- vercelAIGatewayCatalog
- confidence
- 1
- field
- context_length
- value
- 1048576
- normalized value
- 1048576
- scope
- source-model
- source id
- vertexAI
- confidence
- 1
- field
- knowledge_cutoff
- reported field
- knowledge_cutoff
- value
- 2025-01-31
- normalized value
- 2025-01-31
- precision
- day
- scope
- source-model
- source id
- openRouter
- confidence
- 1
- field
- knowledge_cutoff
- reported field
- knowledge_cutoff
- value
- 2025-01
- normalized value
- 2025-01
- precision
- month
- scope
- source-model
- source id
- vercelAIGatewayCatalog
- confidence
- 1
- field
- max_output_tokens
- value
- 65536
- normalized value
- 65536
- scope
- source-model
- source id
- cloudflareWorkersAI
- confidence
- 1
- field
- max_output_tokens
- value
- 65536
- normalized value
- 65536
- scope
- source-model
- source id
- openRouter
- confidence
- 1
- field
- max_output_tokens
- value
- 65535
- normalized value
- 65535
- scope
- source-model
- source id
- vercelAIGatewayCatalog
- confidence
- 1
- field
- max_output_tokens
- value
- 65536
- normalized value
- 65536
- scope
- source-model
- source id
- vertexAI
- confidence
- 1
Explore more models