OpenAI: GPT-4o (2024-08-06)
The 2024-08-06 version of GPT-4o offers improved performance in structured outputs, with the ability to supply a JSON schema in the respone_format. Read more [here](https://openai.com/index/introduci…
openai/gpt-4o-2024-08-06
The 2024-08-06 version of GPT-4o offers improved performance in structured outputs, with the ability to supply a JSON schema in the respone_format. Read more [here](https://openai.com/index/introducing-structured-outputs-in-the-api/). GPT-4o ("o" for "omni") is...
Specifications & capabilities
- Input
- text, image, file
- Output
- text
- Context window · tokens
- 128000
- Knowledge cutoff
- 2023-10-31
- Capabilities
- fileInput, multimodal, promptCaching, structuredOutputs, toolCalling, visionInput
Evaluations
61–80 / 143
36.8%
- source id
- kaggleBenchmarks
- leaderboard name
- MultiLoKo Hindi · EM Hindi
- dimension
- reasoning
- metric
- MultiLoKo Hindi · EM Hindi
- rank
- 13
- aggregation scope
- kaggle
- true
- status
- published
- license
- Apache-2.0
- direction
- higherIsBetter
- conditions
- task
- id
- 39
- name
- EM Hindi
- version
- 1
- sort_order
- DESCENDING
- display_type
- percentages
- benchmark
- id
- 62
- name
- MultiLoKo Hindi
- type
- INDIVIDUAL
- owner
- metaresearch
- version
- 1
- categories
- multilingual and cross-lingual capabilities
- general knowledge and reasoning
- version_id
- 48
- maintainer_tier
- Kaggle-maintained
- maintenance_level
- KAGGLE_MAINTAINED
- candidate
- candidate_class
- modelVersion
- record_kind
- aggregate
- attribution
- Kaggle · metaresearch · MultiLoKo Hindi
- license_url
- https://www.apache.org/licenses/LICENSE-2.0
- metric_kind
- MultiLoKo Hindi · EM Hindi
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- 48/leaderboard.json
- source_page
- https://www.kaggle.com/benchmarks/metaresearch/multiloko-hindi/versions/1
- measurements
- name
- EM Hindi
- unit
- percent
- scale
- 1
- metric
- MultiLoKo Hindi · EM Hindi
- result
- resultCase
- numericResult
- booleanResult
- false
- numericResult
- value
- 0.368
- standardDeviation
- 0
- confidenceInterval
- 0.0597806365533776
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0597806365533776
- hasUnevenConfidenceInterval
- false
- taskVersionId
- 0
- evaluationDate
- 2025-06-16T00:00:00Z
- hasBooleanResult
- false
- hasNumericResult
- true
- hasTaskVersionId
- false
- hasEvaluationDate
- true
- numericResultNullable
- value
- 0.368
- standardDeviation
- 0
- confidenceInterval
- 0.0597806365533776
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0597806365533776
- hasUnevenConfidenceInterval
- false
- hasNumericResultPublic
- false
- hasNumericResultPrivate
- false
- task_id
- 39
- sort_order
- DESCENDING
- source_name
- EM Hindi
- display_type
- percentages
- task_version
- 1
- task_display_type
- PERCENTAGES
- normalization_status
- normalized
- published_at
- 2025-07-15T11:49:31.790Z
- candidate_class
- modelVersion
- source_revision
- b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 51
- benchmark_evaluation
- true
- task_metadata_source
- benchmarkVersion
- candidate_metadata_source
- benchmarkVersionMappings
- evaluated at
- 2025-06-16T00:00:00.000Z
- source model id
- GPT-4o
- confidence lower
- 0.3082193634466224
- confidence upper
- 0.4277806365533776
Original result ↗29.2%
- source id
- kaggleBenchmarks
- leaderboard name
- MultiLoKo Marathi · EM Marathi
- dimension
- reasoning
- metric
- MultiLoKo Marathi · EM Marathi
- rank
- 13
- aggregation scope
- kaggle
- true
- status
- published
- license
- Apache-2.0
- direction
- higherIsBetter
- conditions
- task
- id
- 46
- name
- EM Marathi
- version
- 1
- sort_order
- DESCENDING
- display_type
- percentages
- benchmark
- id
- 69
- name
- MultiLoKo Marathi
- type
- INDIVIDUAL
- owner
- metaresearch
- version
- 1
- categories
- multilingual and cross-lingual capabilities
- general knowledge and reasoning
- version_id
- 55
- maintainer_tier
- Kaggle-maintained
- maintenance_level
- KAGGLE_MAINTAINED
- candidate
- candidate_class
- modelVersion
- record_kind
- aggregate
- attribution
- Kaggle · metaresearch · MultiLoKo Marathi
- license_url
- https://www.apache.org/licenses/LICENSE-2.0
- metric_kind
- MultiLoKo Marathi · EM Marathi
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- 55/leaderboard.json
- source_page
- https://www.kaggle.com/benchmarks/metaresearch/multiloko-marathi/versions/1
- measurements
- name
- EM Marathi
- unit
- percent
- scale
- 1
- metric
- MultiLoKo Marathi · EM Marathi
- result
- resultCase
- numericResult
- booleanResult
- false
- numericResult
- value
- 0.292
- standardDeviation
- 0
- confidenceInterval
- 0.0563619669903391
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0563619669903391
- hasUnevenConfidenceInterval
- false
- taskVersionId
- 0
- evaluationDate
- 2025-06-16T00:00:00Z
- hasBooleanResult
- false
- hasNumericResult
- true
- hasTaskVersionId
- false
- hasEvaluationDate
- true
- numericResultNullable
- value
- 0.292
- standardDeviation
- 0
- confidenceInterval
- 0.0563619669903391
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0563619669903391
- hasUnevenConfidenceInterval
- false
- hasNumericResultPublic
- false
- hasNumericResultPrivate
- false
- task_id
- 46
- sort_order
- DESCENDING
- source_name
- EM Marathi
- display_type
- percentages
- task_version
- 1
- task_display_type
- PERCENTAGES
- normalization_status
- normalized
- published_at
- 2025-07-15T11:49:31.790Z
- candidate_class
- modelVersion
- source_revision
- b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 51
- benchmark_evaluation
- true
- task_metadata_source
- benchmarkVersion
- candidate_metadata_source
- benchmarkVersionMappings
- evaluated at
- 2025-06-16T00:00:00.000Z
- source model id
- GPT-4o
- confidence lower
- 0.23563803300966088
- confidence upper
- 0.34836196699033906
Original result ↗32.4%
- source id
- kaggleBenchmarks
- leaderboard name
- MultiLoKo German · EM German
- dimension
- reasoning
- metric
- MultiLoKo German · EM German
- rank
- 14
- aggregation scope
- kaggle
- true
- status
- published
- license
- Apache-2.0
- direction
- higherIsBetter
- conditions
- task
- id
- 37
- name
- EM German
- version
- 1
- sort_order
- DESCENDING
- display_type
- percentages
- benchmark
- id
- 60
- name
- MultiLoKo German
- type
- INDIVIDUAL
- owner
- metaresearch
- version
- 1
- categories
- multilingual and cross-lingual capabilities
- general knowledge and reasoning
- version_id
- 46
- maintainer_tier
- Kaggle-maintained
- maintenance_level
- KAGGLE_MAINTAINED
- candidate
- candidate_class
- modelVersion
- record_kind
- aggregate
- attribution
- Kaggle · metaresearch · MultiLoKo German
- license_url
- https://www.apache.org/licenses/LICENSE-2.0
- metric_kind
- MultiLoKo German · EM German
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- 46/leaderboard.json
- source_page
- https://www.kaggle.com/benchmarks/metaresearch/multiloko-german/versions/1
- measurements
- name
- EM German
- unit
- percent
- scale
- 1
- metric
- MultiLoKo German · EM German
- result
- resultCase
- numericResult
- booleanResult
- false
- numericResult
- value
- 0.324
- standardDeviation
- 0
- confidenceInterval
- 0.0580128150237069
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0580128150237069
- hasUnevenConfidenceInterval
- false
- taskVersionId
- 0
- evaluationDate
- 2025-06-16T00:00:00Z
- hasBooleanResult
- false
- hasNumericResult
- true
- hasTaskVersionId
- false
- hasEvaluationDate
- true
- numericResultNullable
- value
- 0.324
- standardDeviation
- 0
- confidenceInterval
- 0.0580128150237069
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0580128150237069
- hasUnevenConfidenceInterval
- false
- hasNumericResultPublic
- false
- hasNumericResultPrivate
- false
- task_id
- 37
- sort_order
- DESCENDING
- source_name
- EM German
- display_type
- percentages
- task_version
- 1
- task_display_type
- PERCENTAGES
- normalization_status
- normalized
- published_at
- 2025-07-15T11:49:31.790Z
- candidate_class
- modelVersion
- source_revision
- b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 51
- benchmark_evaluation
- true
- task_metadata_source
- benchmarkVersion
- candidate_metadata_source
- benchmarkVersionMappings
- evaluated at
- 2025-06-16T00:00:00.000Z
- source model id
- GPT-4o
- confidence lower
- 0.2659871849762931
- confidence upper
- 0.38201281502370693
Original result ↗26.8%
- source id
- kaggleBenchmarks
- leaderboard name
- MultiLoKo Japanese · EM Japanese
- dimension
- reasoning
- metric
- MultiLoKo Japanese · EM Japanese
- rank
- 14
- aggregation scope
- kaggle
- true
- status
- published
- license
- Apache-2.0
- direction
- higherIsBetter
- conditions
- task
- id
- 42
- name
- EM Japanese
- version
- 1
- sort_order
- DESCENDING
- display_type
- percentages
- benchmark
- id
- 65
- name
- MultiLoKo Japanese
- type
- INDIVIDUAL
- owner
- metaresearch
- version
- 1
- categories
- multilingual and cross-lingual capabilities
- general knowledge and reasoning
- version_id
- 51
- maintainer_tier
- Kaggle-maintained
- maintenance_level
- KAGGLE_MAINTAINED
- candidate
- candidate_class
- modelVersion
- record_kind
- aggregate
- attribution
- Kaggle · metaresearch · MultiLoKo Japanese
- license_url
- https://www.apache.org/licenses/LICENSE-2.0
- metric_kind
- MultiLoKo Japanese · EM Japanese
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- 51/leaderboard.json
- source_page
- https://www.kaggle.com/benchmarks/metaresearch/multiloko-japanese/versions/1
- measurements
- name
- EM Japanese
- unit
- percent
- scale
- 1
- metric
- MultiLoKo Japanese · EM Japanese
- result
- resultCase
- numericResult
- booleanResult
- false
- numericResult
- value
- 0.268
- standardDeviation
- 0
- confidenceInterval
- 0.0549036255855108
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0549036255855108
- hasUnevenConfidenceInterval
- false
- taskVersionId
- 0
- evaluationDate
- 2025-06-16T00:00:00Z
- hasBooleanResult
- false
- hasNumericResult
- true
- hasTaskVersionId
- false
- hasEvaluationDate
- true
- numericResultNullable
- value
- 0.268
- standardDeviation
- 0
- confidenceInterval
- 0.0549036255855108
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0549036255855108
- hasUnevenConfidenceInterval
- false
- hasNumericResultPublic
- false
- hasNumericResultPrivate
- false
- task_id
- 42
- sort_order
- DESCENDING
- source_name
- EM Japanese
- display_type
- percentages
- task_version
- 1
- task_display_type
- PERCENTAGES
- normalization_status
- normalized
- published_at
- 2025-07-15T11:49:31.790Z
- candidate_class
- modelVersion
- source_revision
- b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 51
- benchmark_evaluation
- true
- task_metadata_source
- benchmarkVersion
- candidate_metadata_source
- benchmarkVersionMappings
- evaluated at
- 2025-06-16T00:00:00.000Z
- source model id
- GPT-4o
- confidence lower
- 0.21309637441448923
- confidence upper
- 0.32290362558551083
Original result ↗38.4%
- source id
- kaggleBenchmarks
- leaderboard name
- MultiLoKo Traditional Mandarin · EM Traditional Mandarin
- dimension
- reasoning
- metric
- MultiLoKo Traditional Mandarin · EM Traditional Mandarin
- rank
- 15
- aggregation scope
- kaggle
- true
- status
- published
- license
- Apache-2.0
- direction
- higherIsBetter
- conditions
- task
- id
- 56
- name
- EM Traditional Mandarin
- version
- 1
- sort_order
- DESCENDING
- display_type
- percentages
- benchmark
- id
- 79
- name
- MultiLoKo Traditional Mandarin
- type
- INDIVIDUAL
- owner
- metaresearch
- version
- 1
- categories
- multilingual and cross-lingual capabilities
- general knowledge and reasoning
- version_id
- 65
- maintainer_tier
- Kaggle-maintained
- maintenance_level
- KAGGLE_MAINTAINED
- candidate
- candidate_class
- modelVersion
- record_kind
- aggregate
- attribution
- Kaggle · metaresearch · MultiLoKo Traditional Mandarin
- license_url
- https://www.apache.org/licenses/LICENSE-2.0
- metric_kind
- MultiLoKo Traditional Mandarin · EM Traditional Mandarin
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- 65/leaderboard.json
- source_page
- https://www.kaggle.com/benchmarks/metaresearch/multiloko-traditional-mandarin/versions/1
- measurements
- name
- EM Traditional Mandarin
- unit
- percent
- scale
- 1
- metric
- MultiLoKo Traditional Mandarin · EM Traditional Mandarin
- result
- resultCase
- numericResult
- booleanResult
- false
- numericResult
- value
- 0.384
- standardDeviation
- 0
- confidenceInterval
- 0.0602884411900746
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0602884411900746
- hasUnevenConfidenceInterval
- false
- taskVersionId
- 0
- evaluationDate
- 2025-06-16T00:00:00Z
- hasBooleanResult
- false
- hasNumericResult
- true
- hasTaskVersionId
- false
- hasEvaluationDate
- true
- numericResultNullable
- value
- 0.384
- standardDeviation
- 0
- confidenceInterval
- 0.0602884411900746
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0602884411900746
- hasUnevenConfidenceInterval
- false
- hasNumericResultPublic
- false
- hasNumericResultPrivate
- false
- task_id
- 56
- sort_order
- DESCENDING
- source_name
- EM Traditional Mandarin
- display_type
- percentages
- task_version
- 1
- task_display_type
- PERCENTAGES
- normalization_status
- normalized
- published_at
- 2025-07-15T11:49:31.790Z
- candidate_class
- modelVersion
- source_revision
- b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 51
- benchmark_evaluation
- true
- task_metadata_source
- benchmarkVersion
- candidate_metadata_source
- benchmarkVersionMappings
- evaluated at
- 2025-06-16T00:00:00.000Z
- source model id
- GPT-4o
- confidence lower
- 0.3237115588099254
- confidence upper
- 0.4442884411900746
Original result ↗35.6%
- source id
- kaggleBenchmarks
- leaderboard name
- MultiLoKo Simplified Mandarin · EM Simplified Mandarin
- dimension
- reasoning
- metric
- MultiLoKo Simplified Mandarin · EM Simplified Mandarin
- rank
- 15
- aggregation scope
- kaggle
- true
- status
- published
- license
- Apache-2.0
- direction
- higherIsBetter
- conditions
- task
- id
- 51
- name
- EM Simplified Mandarin
- version
- 1
- sort_order
- DESCENDING
- display_type
- percentages
- benchmark
- id
- 74
- name
- MultiLoKo Simplified Mandarin
- type
- INDIVIDUAL
- owner
- metaresearch
- version
- 1
- categories
- multilingual and cross-lingual capabilities
- general knowledge and reasoning
- version_id
- 60
- maintainer_tier
- Kaggle-maintained
- maintenance_level
- KAGGLE_MAINTAINED
- candidate
- candidate_class
- modelVersion
- record_kind
- aggregate
- attribution
- Kaggle · metaresearch · MultiLoKo Simplified Mandarin
- license_url
- https://www.apache.org/licenses/LICENSE-2.0
- metric_kind
- MultiLoKo Simplified Mandarin · EM Simplified Mandarin
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- 60/leaderboard.json
- source_page
- https://www.kaggle.com/benchmarks/metaresearch/multiloko-simplified-mandarin/versions/1
- measurements
- name
- EM Simplified Mandarin
- unit
- percent
- scale
- 1
- metric
- MultiLoKo Simplified Mandarin · EM Simplified Mandarin
- result
- resultCase
- numericResult
- booleanResult
- false
- numericResult
- value
- 0.356
- standardDeviation
- 0
- confidenceInterval
- 0.0593534570203831
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0593534570203831
- hasUnevenConfidenceInterval
- false
- taskVersionId
- 0
- evaluationDate
- 2025-06-16T00:00:00Z
- hasBooleanResult
- false
- hasNumericResult
- true
- hasTaskVersionId
- false
- hasEvaluationDate
- true
- numericResultNullable
- value
- 0.356
- standardDeviation
- 0
- confidenceInterval
- 0.0593534570203831
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0593534570203831
- hasUnevenConfidenceInterval
- false
- hasNumericResultPublic
- false
- hasNumericResultPrivate
- false
- task_id
- 51
- sort_order
- DESCENDING
- source_name
- EM Simplified Mandarin
- display_type
- percentages
- task_version
- 1
- task_display_type
- PERCENTAGES
- normalization_status
- normalized
- published_at
- 2025-07-15T11:49:31.790Z
- candidate_class
- modelVersion
- source_revision
- b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 51
- benchmark_evaluation
- true
- task_metadata_source
- benchmarkVersion
- candidate_metadata_source
- benchmarkVersionMappings
- evaluated at
- 2025-06-16T00:00:00.000Z
- source model id
- GPT-4o
- confidence lower
- 0.2966465429796169
- confidence upper
- 0.41535345702038307
Original result ↗31.6%
- source id
- kaggleBenchmarks
- leaderboard name
- MultiLoKo Russian · EM Russian
- dimension
- reasoning
- metric
- MultiLoKo Russian · EM Russian
- rank
- 19
- aggregation scope
- kaggle
- true
- status
- published
- license
- Apache-2.0
- direction
- higherIsBetter
- conditions
- task
- id
- 50
- name
- EM Russian
- version
- 1
- sort_order
- DESCENDING
- display_type
- percentages
- benchmark
- id
- 73
- name
- MultiLoKo Russian
- type
- INDIVIDUAL
- owner
- metaresearch
- version
- 1
- categories
- multilingual and cross-lingual capabilities
- general knowledge and reasoning
- version_id
- 59
- maintainer_tier
- Kaggle-maintained
- maintenance_level
- KAGGLE_MAINTAINED
- candidate
- candidate_class
- modelVersion
- record_kind
- aggregate
- attribution
- Kaggle · metaresearch · MultiLoKo Russian
- license_url
- https://www.apache.org/licenses/LICENSE-2.0
- metric_kind
- MultiLoKo Russian · EM Russian
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- 59/leaderboard.json
- source_page
- https://www.kaggle.com/benchmarks/metaresearch/multiloko-russian/versions/1
- measurements
- name
- EM Russian
- unit
- percent
- scale
- 1
- metric
- MultiLoKo Russian · EM Russian
- result
- resultCase
- numericResult
- booleanResult
- false
- numericResult
- value
- 0.316
- standardDeviation
- 0
- confidenceInterval
- 0.0576301405634277
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0576301405634277
- hasUnevenConfidenceInterval
- false
- taskVersionId
- 0
- evaluationDate
- 2025-06-16T00:00:00Z
- hasBooleanResult
- false
- hasNumericResult
- true
- hasTaskVersionId
- false
- hasEvaluationDate
- true
- numericResultNullable
- value
- 0.316
- standardDeviation
- 0
- confidenceInterval
- 0.0576301405634277
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0576301405634277
- hasUnevenConfidenceInterval
- false
- hasNumericResultPublic
- false
- hasNumericResultPrivate
- false
- task_id
- 50
- sort_order
- DESCENDING
- source_name
- EM Russian
- display_type
- percentages
- task_version
- 1
- task_display_type
- PERCENTAGES
- normalization_status
- normalized
- published_at
- 2025-07-15T11:49:31.790Z
- candidate_class
- modelVersion
- source_revision
- b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 51
- benchmark_evaluation
- true
- task_metadata_source
- benchmarkVersion
- candidate_metadata_source
- benchmarkVersionMappings
- evaluated at
- 2025-06-16T00:00:00.000Z
- source model id
- GPT-4o
- confidence lower
- 0.2583698594365723
- confidence upper
- 0.3736301405634277
Original result ↗11.6%
- source id
- kaggleBenchmarks
- leaderboard name
- MultiLoKo Khmer · EM Khmer
- dimension
- reasoning
- metric
- MultiLoKo Khmer · EM Khmer
- rank
- 19
- aggregation scope
- kaggle
- true
- status
- published
- license
- Apache-2.0
- direction
- higherIsBetter
- conditions
- task
- id
- 43
- name
- EM Khmer
- version
- 1
- sort_order
- DESCENDING
- display_type
- percentages
- benchmark
- id
- 66
- name
- MultiLoKo Khmer
- type
- INDIVIDUAL
- owner
- metaresearch
- version
- 1
- categories
- multilingual and cross-lingual capabilities
- general knowledge and reasoning
- version_id
- 52
- maintainer_tier
- Kaggle-maintained
- maintenance_level
- KAGGLE_MAINTAINED
- candidate
- candidate_class
- modelVersion
- record_kind
- aggregate
- attribution
- Kaggle · metaresearch · MultiLoKo Khmer
- license_url
- https://www.apache.org/licenses/LICENSE-2.0
- metric_kind
- MultiLoKo Khmer · EM Khmer
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- 52/leaderboard.json
- source_page
- https://www.kaggle.com/benchmarks/metaresearch/multiloko-khmer/versions/1
- measurements
- name
- EM Khmer
- unit
- percent
- scale
- 1
- metric
- MultiLoKo Khmer · EM Khmer
- result
- resultCase
- numericResult
- booleanResult
- false
- numericResult
- value
- 0.116
- standardDeviation
- 0
- confidenceInterval
- 0.0396947630454828
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0396947630454828
- hasUnevenConfidenceInterval
- false
- taskVersionId
- 0
- evaluationDate
- 2025-06-16T00:00:00Z
- hasBooleanResult
- false
- hasNumericResult
- true
- hasTaskVersionId
- false
- hasEvaluationDate
- true
- numericResultNullable
- value
- 0.116
- standardDeviation
- 0
- confidenceInterval
- 0.0396947630454828
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0396947630454828
- hasUnevenConfidenceInterval
- false
- hasNumericResultPublic
- false
- hasNumericResultPrivate
- false
- task_id
- 43
- sort_order
- DESCENDING
- source_name
- EM Khmer
- display_type
- percentages
- task_version
- 1
- task_display_type
- PERCENTAGES
- normalization_status
- normalized
- published_at
- 2025-07-15T11:49:31.790Z
- candidate_class
- modelVersion
- source_revision
- b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 51
- benchmark_evaluation
- true
- task_metadata_source
- benchmarkVersion
- candidate_metadata_source
- benchmarkVersionMappings
- evaluated at
- 2025-06-16T00:00:00.000Z
- source model id
- GPT-4o
- confidence lower
- 0.07630523695451721
- confidence upper
- 0.1556947630454828
Original result ↗18.4%
- source id
- kaggleBenchmarks
- leaderboard name
- MultiLoKo Cantonese · EM Cantonese
- dimension
- reasoning
- metric
- MultiLoKo Cantonese · EM Cantonese
- rank
- 20
- aggregation scope
- kaggle
- true
- status
- published
- license
- Apache-2.0
- direction
- higherIsBetter
- conditions
- task
- id
- 32
- name
- EM Cantonese
- version
- 1
- sort_order
- DESCENDING
- display_type
- percentages
- benchmark
- id
- 55
- name
- MultiLoKo Cantonese
- type
- INDIVIDUAL
- owner
- metaresearch
- version
- 1
- categories
- multilingual and cross-lingual capabilities
- general knowledge and reasoning
- version_id
- 41
- maintainer_tier
- Kaggle-maintained
- maintenance_level
- KAGGLE_MAINTAINED
- candidate
- candidate_class
- modelVersion
- record_kind
- aggregate
- attribution
- Kaggle · metaresearch · MultiLoKo Cantonese
- license_url
- https://www.apache.org/licenses/LICENSE-2.0
- metric_kind
- MultiLoKo Cantonese · EM Cantonese
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- 41/leaderboard.json
- source_page
- https://www.kaggle.com/benchmarks/metaresearch/multiloko-cantonese/versions/1
- measurements
- name
- EM Cantonese
- unit
- percent
- scale
- 1
- metric
- MultiLoKo Cantonese · EM Cantonese
- result
- resultCase
- numericResult
- booleanResult
- false
- numericResult
- value
- 0.184
- standardDeviation
- 0
- confidenceInterval
- 0.0480321556116024
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0480321556116024
- hasUnevenConfidenceInterval
- false
- taskVersionId
- 0
- evaluationDate
- 2025-06-16T00:00:00Z
- hasBooleanResult
- false
- hasNumericResult
- true
- hasTaskVersionId
- false
- hasEvaluationDate
- true
- numericResultNullable
- value
- 0.184
- standardDeviation
- 0
- confidenceInterval
- 0.0480321556116024
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0480321556116024
- hasUnevenConfidenceInterval
- false
- hasNumericResultPublic
- false
- hasNumericResultPrivate
- false
- task_id
- 32
- sort_order
- DESCENDING
- source_name
- EM Cantonese
- display_type
- percentages
- task_version
- 1
- task_display_type
- PERCENTAGES
- normalization_status
- normalized
- published_at
- 2025-07-15T11:49:31.790Z
- candidate_class
- modelVersion
- source_revision
- b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 51
- benchmark_evaluation
- true
- task_metadata_source
- benchmarkVersion
- candidate_metadata_source
- benchmarkVersionMappings
- evaluated at
- 2025-06-16T00:00:00.000Z
- source model id
- GPT-4o
- confidence lower
- 0.1359678443883976
- confidence upper
- 0.2320321556116024
Original result ↗52.806%
- source id
- openRouterBenchmarks
- leaderboard name
- GPQA Diamond · Accuracy · OpenAI
- dimension
- knowledge
- metric
- GPQA Diamond · Accuracy
- rank
- 23
- sample size
- 594
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 3
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- gpqa_diamond
- name
- GPQA Diamond
- record_kind
- aggregate
- provider_name
- OpenAI
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-10-02T23:16:52.382Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- GPQA Diamond · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 594
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/gpqa-diamond
- measurements
- accuracy
- 0.5280586666666666
- accuracy_std_dev
- 0.004827144451490511
- cost_per_task_usd
- 0.01799902861952862
- output_tokens_per_task
- 1607.080808080808
- latency_seconds_per_task
- 14.939786195286194
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 26
- benchmark_evaluation
- true
- openrouter_permaslug
- openai/gpt-4o-2024-08-06
- source model id
- openai/gpt-4o-2024-08-06
Original result ↗0.711%
- source id
- kaggleBenchmarks
- leaderboard name
- BrowseComp · Score
- dimension
- knowledge
- metric
- BrowseComp · Score
- rank
- 24
- aggregation scope
- kaggle
- true
- status
- published
- license
- Apache-2.0
- direction
- higherIsBetter
- conditions
- task
- id
- 26
- name
- Score
- version
- 1
- sort_order
- DESCENDING
- display_type
- percentages
- benchmark
- id
- 49
- name
- BrowseComp
- type
- INDIVIDUAL
- owner
- openai
- version
- 1
- categories
- general knowledge and reasoning
- factuality and grounding
- version_id
- 35
- maintainer_tier
- Kaggle-maintained
- maintenance_level
- KAGGLE_MAINTAINED
- candidate
- candidate_class
- modelVersion
- record_kind
- aggregate
- attribution
- Kaggle · openai · BrowseComp
- license_url
- https://www.apache.org/licenses/LICENSE-2.0
- metric_kind
- BrowseComp · Score
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- 35/leaderboard.json
- source_page
- https://www.kaggle.com/benchmarks/openai/browsecomp/versions/1
- measurements
- name
- Score
- unit
- percent
- scale
- 1
- metric
- BrowseComp · Score
- result
- resultCase
- numericResult
- booleanResult
- false
- numericResult
- value
- 0.0071090047393364
- standardDeviation
- 0
- confidenceInterval
- 0.004627926213592
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.004627926213592
- hasUnevenConfidenceInterval
- false
- taskVersionId
- 0
- evaluationDate
- 2025-06-13T00:00:00Z
- hasBooleanResult
- false
- hasNumericResult
- true
- hasTaskVersionId
- false
- hasEvaluationDate
- true
- numericResultNullable
- value
- 0.0071090047393364
- standardDeviation
- 0
- confidenceInterval
- 0.004627926213592
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.004627926213592
- hasUnevenConfidenceInterval
- false
- hasNumericResultPublic
- false
- hasNumericResultPrivate
- false
- task_id
- 26
- sort_order
- DESCENDING
- source_name
- Score
- display_type
- percentages
- task_version
- 1
- task_display_type
- PERCENTAGES
- normalization_status
- normalized
- published_at
- 2025-07-15T14:25:30.827974100Z
- candidate_class
- modelVersion
- source_revision
- b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 42
- benchmark_evaluation
- true
- task_metadata_source
- benchmarkVersion
- candidate_metadata_source
- benchmarkVersionMappings
- evaluated at
- 2025-06-13T00:00:00.000Z
- source model id
- GPT-4o
- confidence lower
- 0.0024810785257443997
- confidence upper
- 0.011736930952928399
Original result ↗43.111%
- source id
- openRouterBenchmarks
- leaderboard name
- tau2-bench Verified Airline · Accuracy · OpenAI
- dimension
- agentic
- metric
- tau2-bench Verified Airline · Accuracy
- rank
- 25
- sample size
- 150
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 3
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- tau_bench_verified_airline
- name
- tau2-bench Verified Airline
- record_kind
- aggregate
- provider_name
- OpenAI
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-10-02T23:09:02.629Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- tau2-bench Verified Airline · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 150
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/tau2-bench-airline
- measurements
- accuracy
- 0.431111
- accuracy_std_dev
- 0.00314238253559302
- cost_per_task_usd
- 0.24174756666666666
- output_tokens_per_task
- 2473.9733333333334
- latency_seconds_per_task
- 46.57624
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 27
- benchmark_evaluation
- true
- openrouter_permaslug
- openai/gpt-4o-2024-08-06
- source model id
- openai/gpt-4o-2024-08-06
Original result ↗0.077
- source id
- kaggleBenchmarks
- leaderboard name
- SciCode Main With Background · SciCode Main with Background
- dimension
- coding
- metric
- SciCode Main With Background · SciCode Main with Background
- rank
- 33
- aggregation scope
- kaggle
- true
- status
- published
- license
- Apache-2.0
- direction
- higherIsBetter
- conditions
- task
- id
- 40201
- name
- SciCode Main with Background
- version
- 3
- sort_order
- DESCENDING
- display_type
- rawValues
- benchmark
- id
- 13
- name
- SciCode Main With Background
- type
- INDIVIDUAL
- owner
- open-benchmarks
- version
- 1
- categories
- general knowledge and reasoning
- coding
- science
- version_id
- 13
- maintainer_tier
- Kaggle-maintained
- maintenance_level
- KAGGLE_MAINTAINED
- candidate
- candidate_class
- modelVersion
- record_kind
- aggregate
- attribution
- Kaggle · open-benchmarks · SciCode Main With Background
- license_url
- https://www.apache.org/licenses/LICENSE-2.0
- metric_kind
- SciCode Main With Background · SciCode Main with Background
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- 13/leaderboard.json
- source_page
- https://www.kaggle.com/benchmarks/open-benchmarks/scicode-main-with-background/versions/1
- measurements
- name
- SciCode Main with Background
- unit
- score
- scale
- 1
- metric
- SciCode Main With Background · SciCode Main with Background
- result
- resultCase
- numericResult
- booleanResult
- false
- numericResult
- value
- 0.0769230769230769
- standardDeviation
- 0
- confidenceInterval
- 0.0647796628574181
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0647796628574181
- hasUnevenConfidenceInterval
- false
- taskVersionId
- 0
- evaluationDate
- 2025-05-22T00:00:00Z
- hasBooleanResult
- false
- hasNumericResult
- true
- hasTaskVersionId
- false
- hasEvaluationDate
- true
- numericResultNullable
- value
- 0.0769230769230769
- standardDeviation
- 0
- confidenceInterval
- 0.0647796628574181
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0647796628574181
- hasUnevenConfidenceInterval
- false
- hasNumericResultPublic
- false
- hasNumericResultPrivate
- false
- task_id
- 40201
- task_url
- /benchmarks/tasks/benchmarkler/scicode-background-scicode-main-with-background
- sort_order
- DESCENDING
- source_name
- SciCode Main with Background
- display_type
- rawValues
- task_version
- 3
- task_display_type
- RAW_VALUES
- normalization_status
- normalized
- published_at
- 2025-07-15T11:49:31.790Z
- candidate_class
- modelVersion
- source_revision
- b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 57
- benchmark_evaluation
- true
- task_metadata_source
- benchmarkVersion
- candidate_metadata_source
- benchmarkVersionMappings
- evaluated at
- 2025-05-22T00:00:00.000Z
- source model id
- GPT-4o
- confidence lower
- 0.012143414065658803
- confidence upper
- 0.141702739780495
Original result ↗1.538%
- source id
- kaggleBenchmarks
- leaderboard name
- SciCode Main Standard · SciCode Main Standard
- dimension
- coding
- metric
- SciCode Main Standard · SciCode Main Standard
- rank
- 36
- aggregation scope
- kaggle
- true
- status
- published
- license
- Apache-2.0
- direction
- higherIsBetter
- conditions
- task
- id
- 40105
- name
- SciCode Main Standard
- version
- 3
- sort_order
- DESCENDING
- display_type
- percentages
- benchmark
- id
- 12
- name
- SciCode Main Standard
- type
- INDIVIDUAL
- owner
- open-benchmarks
- version
- 1
- categories
- general knowledge and reasoning
- coding
- science
- version_id
- 12
- maintainer_tier
- Kaggle-maintained
- maintenance_level
- KAGGLE_MAINTAINED
- candidate
- candidate_class
- modelVersion
- record_kind
- aggregate
- attribution
- Kaggle · open-benchmarks · SciCode Main Standard
- license_url
- https://www.apache.org/licenses/LICENSE-2.0
- metric_kind
- SciCode Main Standard · SciCode Main Standard
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- 12/leaderboard.json
- source_page
- https://www.kaggle.com/benchmarks/open-benchmarks/scicode-main-standard/versions/1
- measurements
- name
- SciCode Main Standard
- unit
- percent
- scale
- 1
- metric
- SciCode Main Standard · SciCode Main Standard
- result
- resultCase
- numericResult
- booleanResult
- false
- numericResult
- value
- 0.0153846153846153
- standardDeviation
- 0
- confidenceInterval
- 0.0299204446309948
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0299204446309948
- hasUnevenConfidenceInterval
- false
- taskVersionId
- 0
- evaluationDate
- 2025-05-22T00:00:00Z
- hasBooleanResult
- false
- hasNumericResult
- true
- hasTaskVersionId
- false
- hasEvaluationDate
- true
- numericResultNullable
- value
- 0.0153846153846153
- standardDeviation
- 0
- confidenceInterval
- 0.0299204446309948
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0299204446309948
- hasUnevenConfidenceInterval
- false
- hasNumericResultPublic
- false
- hasNumericResultPrivate
- false
- task_id
- 40105
- task_url
- /benchmarks/tasks/benchmarkler/scicode-standard-scicode-main-standard
- sort_order
- DESCENDING
- source_name
- SciCode Main Standard
- display_type
- percentages
- task_version
- 3
- task_display_type
- PERCENTAGES
- normalization_status
- normalized
- published_at
- 2025-07-15T11:49:31.790Z
- candidate_class
- modelVersion
- source_revision
- b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 57
- benchmark_evaluation
- true
- task_metadata_source
- benchmarkVersion
- candidate_metadata_source
- benchmarkVersionMappings
- evaluated at
- 2025-05-22T00:00:00.000Z
- source model id
- GPT-4o
- confidence lower
- -0.014535829246379499
- confidence upper
- 0.0453050600156101
Original result ↗36%
- source id
- sweBench
- leaderboard name
- SWE-bench Lite
- dimension
- coding
- metric
- Resolved tasks
- rank
- 37
- evaluated at
- 2024-11-13T00:00:00.000Z
Original result ↗7.692%
- source id
- kaggleBenchmarks
- leaderboard name
- SciCode · SciCode Main With Background
- dimension
- coding
- metric
- SciCode · SciCode Main With Background
- rank
- 37
- aggregation scope
- kaggle
- true
- status
- published
- license
- Apache-2.0
- direction
- higherIsBetter
- conditions
- task
- id
- 40201
- name
- SciCode Main With Background
- version
- 3
- sort_order
- DESCENDING
- display_type
- percentages
- benchmark
- id
- 16
- name
- SciCode
- type
- SUITE
- owner
- open-benchmarks
- version
- 1
- categories
- general knowledge and reasoning
- coding
- science
- version_id
- 16
- maintainer_tier
- Kaggle-maintained
- maintenance_level
- KAGGLE_MAINTAINED
- candidate
- candidate_class
- modelVersion
- record_kind
- aggregate
- attribution
- Kaggle · open-benchmarks · SciCode
- license_url
- https://www.apache.org/licenses/LICENSE-2.0
- metric_kind
- SciCode · SciCode Main With Background
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- 16/leaderboard.json
- source_page
- https://www.kaggle.com/benchmarks/open-benchmarks/scicode/versions/1
- measurements
- name
- SciCode Main With Background
- unit
- percent
- scale
- 1
- metric
- SciCode · SciCode Main With Background
- result
- resultCase
- numericResult
- booleanResult
- false
- numericResult
- value
- 0.0769230769230769
- standardDeviation
- 0
- confidenceInterval
- 0.0647796628574181
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0647796628574181
- hasUnevenConfidenceInterval
- false
- taskVersionId
- 0
- evaluationDate
- 2025-05-22T00:00:00Z
- hasBooleanResult
- false
- hasNumericResult
- true
- hasTaskVersionId
- false
- hasEvaluationDate
- true
- numericResultNullable
- value
- 0.0769230769230769
- standardDeviation
- 0
- confidenceInterval
- 0.0647796628574181
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0647796628574181
- hasUnevenConfidenceInterval
- false
- hasNumericResultPublic
- false
- hasNumericResultPrivate
- false
- task_id
- 40201
- task_url
- /benchmarks/open-benchmarks/scicode-main-with-background/versions/1
- sort_order
- DESCENDING
- source_name
- SciCode Main With Background
- display_type
- percentages
- task_version
- 3
- task_display_type
- RAW_VALUES
- normalization_status
- normalized
- published_at
- 2025-10-21T20:29:26.251536900Z
- candidate_class
- modelVersion
- source_revision
- b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 62
- benchmark_evaluation
- true
- task_metadata_source
- benchmarkVersion
- candidate_metadata_source
- benchmarkVersionMappings
- evaluated at
- 2025-05-22T00:00:00.000Z
- source model id
- GPT-4o
- confidence lower
- 0.012143414065658803
- confidence upper
- 0.141702739780495
Original result ↗88.086%
- source id
- kaggleBenchmarks
- leaderboard name
- MMLU · mmlu_task
- dimension
- reasoning
- metric
- MMLU · mmlu_task
- rank
- 39
- aggregation scope
- kaggle
- true
- status
- published
- license
- Apache-2.0
- direction
- higherIsBetter
- conditions
- task
- id
- 40139
- name
- mmlu_task
- version
- 7
- sort_order
- DESCENDING
- display_type
- percentages
- benchmark
- id
- 42
- name
- MMLU
- type
- INDIVIDUAL
- owner
- open-benchmarks
- version
- 1
- categories
- general knowledge and reasoning
- version_id
- 28
- maintainer_tier
- Kaggle-maintained
- maintenance_level
- KAGGLE_MAINTAINED
- candidate
- candidate_class
- modelVersion
- record_kind
- aggregate
- attribution
- Kaggle · open-benchmarks · MMLU
- license_url
- https://www.apache.org/licenses/LICENSE-2.0
- metric_kind
- MMLU · mmlu_task
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- 28/leaderboard.json
- source_page
- https://www.kaggle.com/benchmarks/open-benchmarks/mmlu/versions/1
- measurements
- name
- mmlu_task
- unit
- percent
- scale
- 1
- metric
- MMLU · mmlu_task
- result
- resultCase
- numericResult
- booleanResult
- false
- numericResult
- value
- 0.8808574277168495
- standardDeviation
- 0
- confidenceInterval
- 0.0053582121301719
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0053582121301719
- hasUnevenConfidenceInterval
- false
- taskVersionId
- 0
- evaluationDate
- 2025-05-29T00:00:00Z
- hasBooleanResult
- false
- hasNumericResult
- true
- hasTaskVersionId
- false
- hasEvaluationDate
- true
- numericResultNullable
- value
- 0.8808574277168495
- standardDeviation
- 0
- confidenceInterval
- 0.0053582121301719
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0053582121301719
- hasUnevenConfidenceInterval
- false
- hasNumericResultPublic
- false
- hasNumericResultPrivate
- false
- task_id
- 40139
- task_url
- /benchmarks/tasks/benchmarkler/mmlu-task
- sort_order
- DESCENDING
- source_name
- mmlu_task
- display_type
- percentages
- task_version
- 7
- task_display_type
- PERCENTAGES
- normalization_status
- normalized
- published_at
- 2025-07-15T11:49:31.790Z
- candidate_class
- modelVersion
- source_revision
- b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 61
- benchmark_evaluation
- true
- task_metadata_source
- benchmarkVersion
- candidate_metadata_source
- benchmarkVersionMappings
- evaluated at
- 2025-05-29T00:00:00.000Z
- source model id
- GPT-4o
- confidence lower
- 0.8754992155866776
- confidence upper
- 0.8862156398470213
Original result ↗0.337
- source id
- kaggleBenchmarks
- leaderboard name
- SciCode Subproblem With Background · SciCode Subproblem with Background
- dimension
- coding
- metric
- SciCode Subproblem With Background · SciCode Subproblem with Background
- rank
- 39
- aggregation scope
- kaggle
- true
- status
- published
- license
- Apache-2.0
- direction
- higherIsBetter
- conditions
- task
- id
- 40200
- name
- SciCode Subproblem with Background
- version
- 3
- sort_order
- DESCENDING
- display_type
- rawValues
- benchmark
- id
- 15
- name
- SciCode Subproblem With Background
- type
- INDIVIDUAL
- owner
- open-benchmarks
- version
- 1
- categories
- general knowledge and reasoning
- coding
- science
- version_id
- 15
- maintainer_tier
- Kaggle-maintained
- maintenance_level
- KAGGLE_MAINTAINED
- candidate
- candidate_class
- modelVersion
- record_kind
- aggregate
- attribution
- Kaggle · open-benchmarks · SciCode Subproblem With Background
- license_url
- https://www.apache.org/licenses/LICENSE-2.0
- metric_kind
- SciCode Subproblem With Background · SciCode Subproblem with Background
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- 15/leaderboard.json
- source_page
- https://www.kaggle.com/benchmarks/open-benchmarks/scicode-subproblem-with-background/versions/1
- measurements
- name
- SciCode Subproblem with Background
- unit
- score
- scale
- 1
- metric
- SciCode Subproblem With Background · SciCode Subproblem with Background
- result
- resultCase
- numericResult
- booleanResult
- false
- numericResult
- value
- 0.3367697594501718
- standardDeviation
- 0
- confidenceInterval
- 0.0543000588021355
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0543000588021355
- hasUnevenConfidenceInterval
- false
- taskVersionId
- 0
- evaluationDate
- 2025-05-22T00:00:00Z
- hasBooleanResult
- false
- hasNumericResult
- true
- hasTaskVersionId
- false
- hasEvaluationDate
- true
- numericResultNullable
- value
- 0.3367697594501718
- standardDeviation
- 0
- confidenceInterval
- 0.0543000588021355
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0543000588021355
- hasUnevenConfidenceInterval
- false
- hasNumericResultPublic
- false
- hasNumericResultPrivate
- false
- task_id
- 40200
- task_url
- /benchmarks/tasks/benchmarkler/scicode-background-scicode-subproblem-with-background
- sort_order
- DESCENDING
- source_name
- SciCode Subproblem with Background
- display_type
- rawValues
- task_version
- 3
- task_display_type
- RAW_VALUES
- normalization_status
- normalized
- published_at
- 2025-07-15T11:49:31.790Z
- candidate_class
- modelVersion
- source_revision
- b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 57
- benchmark_evaluation
- true
- task_metadata_source
- benchmarkVersion
- candidate_metadata_source
- benchmarkVersionMappings
- evaluated at
- 2025-05-22T00:00:00.000Z
- source model id
- GPT-4o
- confidence lower
- 0.2824697006480363
- confidence upper
- 0.3910698182523073
Original result ↗0.241
- source id
- kaggleBenchmarks
- leaderboard name
- SciCode Subproblem Standard · SciCode Subproblem Standard
- dimension
- coding
- metric
- SciCode Subproblem Standard · SciCode Subproblem Standard
- rank
- 39
- aggregation scope
- kaggle
- true
- status
- published
- license
- Apache-2.0
- direction
- higherIsBetter
- conditions
- task
- id
- 40106
- name
- SciCode Subproblem Standard
- version
- 3
- sort_order
- DESCENDING
- display_type
- rawValues
- benchmark
- id
- 14
- name
- SciCode Subproblem Standard
- type
- INDIVIDUAL
- owner
- open-benchmarks
- version
- 1
- categories
- general knowledge and reasoning
- coding
- science
- version_id
- 14
- maintainer_tier
- Kaggle-maintained
- maintenance_level
- KAGGLE_MAINTAINED
- candidate
- candidate_class
- modelVersion
- record_kind
- aggregate
- attribution
- Kaggle · open-benchmarks · SciCode Subproblem Standard
- license_url
- https://www.apache.org/licenses/LICENSE-2.0
- metric_kind
- SciCode Subproblem Standard · SciCode Subproblem Standard
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- 14/leaderboard.json
- source_page
- https://www.kaggle.com/benchmarks/open-benchmarks/scicode-subproblem-standard/versions/1
- measurements
- name
- SciCode Subproblem Standard
- unit
- score
- scale
- 1
- metric
- SciCode Subproblem Standard · SciCode Subproblem Standard
- result
- resultCase
- numericResult
- booleanResult
- false
- numericResult
- value
- 0.2405498281786941
- standardDeviation
- 0
- confidenceInterval
- 0.0491081715605116
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0491081715605116
- hasUnevenConfidenceInterval
- false
- taskVersionId
- 0
- evaluationDate
- 2025-05-22T00:00:00Z
- hasBooleanResult
- false
- hasNumericResult
- true
- hasTaskVersionId
- false
- hasEvaluationDate
- true
- numericResultNullable
- value
- 0.2405498281786941
- standardDeviation
- 0
- confidenceInterval
- 0.0491081715605116
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0491081715605116
- hasUnevenConfidenceInterval
- false
- hasNumericResultPublic
- false
- hasNumericResultPrivate
- false
- task_id
- 40106
- task_url
- /benchmarks/tasks/benchmarkler/scicode-standard-scicode-subproblem-standard
- sort_order
- DESCENDING
- source_name
- SciCode Subproblem Standard
- display_type
- rawValues
- task_version
- 3
- task_display_type
- RAW_VALUES
- normalization_status
- normalized
- published_at
- 2025-07-15T11:49:31.790Z
- candidate_class
- modelVersion
- source_revision
- b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 57
- benchmark_evaluation
- true
- task_metadata_source
- benchmarkVersion
- candidate_metadata_source
- benchmarkVersionMappings
- evaluated at
- 2025-05-22T00:00:00.000Z
- source model id
- GPT-4o
- confidence lower
- 0.19144165661818252
- confidence upper
- 0.2896579997392057
Original result ↗51.515%
- source id
- openRouterBenchmarks
- leaderboard name
- GPQA Diamond · Accuracy · Azure
- dimension
- knowledge
- metric
- GPQA Diamond · Accuracy
- rank
- 41
- sample size
- 396
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 2
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- gpqa_diamond
- name
- GPQA Diamond
- record_kind
- aggregate
- provider_name
- Azure
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-10-01T16:26:08.836Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- GPQA Diamond · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 396
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/gpqa-diamond
- measurements
- accuracy
- 0.5151519999999999
- accuracy_std_dev
- 0.006733999999999962
- cost_per_task_usd
- 0.018041959595959597
- output_tokens_per_task
- 1611.5353535353536
- latency_seconds_per_task
- 13.4984797979798
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 43
- benchmark_evaluation
- true
- openrouter_permaslug
- openai/gpt-4o-2024-08-06
- source model id
- openai/gpt-4o-2024-08-06
Original result ↗Published prices
1–3 / 3
$1.25
- Inference provider
- openRouter
- Platform
- OpenRouter
- Unit
- cache_read_tokens_per_million
- Currency
- USD
$2.50
- Inference provider
- openRouter
- Platform
- OpenRouter
- Unit
- input_tokens_per_million
- Currency
- USD
$10.00
- Inference provider
- openRouter
- Platform
- OpenRouter
- Unit
- output_tokens_per_million
- Currency
- USD
Safety assessments
1–1 / 1
35 / 100
- source id
- stanfordFMTI
- canonical model id
- openai/gpt-4o-2024-08-06
- taxonomy
- transparency
- hazard category
- transparency
- hazard name
- Organization transparency
- policy area
- OpenAI
- assessment type
- organization-level disclosure
- metric
- disclosure_score
- result label
- 35 / 100
- sample size
- 100
- updated at
- 2026-11-09T18:08:05.227Z
Source specification observations
1–3 / 3
- field
- context_length
- value
- 128000
- normalized value
- 128000
- scope
- source-model
- source id
- openRouter
- confidence
- 1
- field
- knowledge_cutoff
- reported field
- knowledge_cutoff
- value
- 2023-10-31
- normalized value
- 2023-10-31
- precision
- day
- scope
- source-model
- source id
- openRouter
- confidence
- 1
- field
- max_output_tokens
- value
- 16384
- normalized value
- 16384
- scope
- source-model
- source id
- openRouter
- confidence
- 1
Explore more models