OpenAI: GPT-4o (2024-08-06)
The 2024-08-06 version of GPT-4o offers improved performance in structured outputs, with the ability to supply a JSON schema in the respone_format. Read more [here](https://openai.com/index/introduci…
openai/gpt-4o-2024-08-06
The 2024-08-06 version of GPT-4o offers improved performance in structured outputs, with the ability to supply a JSON schema in the respone_format. Read more [here](https://openai.com/index/introducing-structured-outputs-in-the-api/). GPT-4o ("o" for "omni") is...
Specifications & capabilities
- Input
- text, image, file
- Output
- text
- Context window · tokens
- 128000
- Knowledge cutoff
- 2023-10-31
- Capabilities
- fileInput, multimodal, promptCaching, structuredOutputs, toolCalling, visionInput
Evaluations
21–40 / 143
0.814
- source id
- llmStatsArchive
- leaderboard name
- MMMLU · Normalized (reported)
- dimension
- general
- metric
- MMMLU · Normalized (reported)
- rank
- 3
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- MMMLU
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- categories
- language
- reasoning
- math
- general
- created_at
- 2025-07-19T19:56:14.144789+00:00
- paper_link
- https://arxiv.org/abs/2009.03300
- updated_at
- 2025-07-19T19:56:14.144789+00:00
- description
- Multilingual Massive Multitask Language Understanding dataset released by OpenAI, featuring professionally translated MMLU test questions across 14 languages including Arabic, Bengali, German, Spanish, French, Hindi, Indonesian, Italian, Japanese, Korean, Portuguese, Swahili, Yoruba, and Chinese. Contains approximately 15,908 multiple-choice questions per language covering 57 subjects.
- benchmark_id
- mmmlu
- multilingual
- true
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- MMMLU · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/gpt-4-1/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 3
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-2024-08-06
Original result ↗0.814
- source id
- llmStatsArchive
- leaderboard name
- MMMLU · Raw
- dimension
- general
- metric
- MMMLU · Raw
- rank
- 3
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- MMMLU
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- categories
- language
- reasoning
- math
- general
- created_at
- 2025-07-19T19:56:14.144789+00:00
- paper_link
- https://arxiv.org/abs/2009.03300
- updated_at
- 2025-07-19T19:56:14.144789+00:00
- description
- Multilingual Massive Multitask Language Understanding dataset released by OpenAI, featuring professionally translated MMLU test questions across 14 languages including Arabic, Bengali, German, Spanish, French, Hindi, Indonesian, Italian, Japanese, Korean, Portuguese, Swahili, Yoruba, and Chinese. Contains approximately 15,908 multiple-choice questions per language covering 57 subjects.
- benchmark_id
- mmmlu
- multilingual
- true
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- MMMLU · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/gpt-4-1/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 3
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-2024-08-06
Original result ↗0.81
- source id
- llmStatsArchive
- leaderboard name
- IFEval · Raw
- dimension
- general
- metric
- IFEval · Raw
- rank
- 3
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- IFEval
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:12.241350+00:00
- paper_link
- https://arxiv.org/abs/2311.07911
- updated_at
- 2025-07-19T19:56:12.241350+00:00
- description
- Instruction-Following Evaluation (IFEval) benchmark for large language models, focusing on verifiable instructions with 25 types of instructions and around 500 prompts containing one or more verifiable constraints
- benchmark_id
- ifeval
- multilingual
- false
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- IFEval · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/gpt-4-1/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 3
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-2024-08-06
Original result ↗0.81
- source id
- llmStatsArchive
- leaderboard name
- IFEval · Normalized (reported)
- dimension
- general
- metric
- IFEval · Normalized (reported)
- rank
- 3
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- IFEval
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:12.241350+00:00
- paper_link
- https://arxiv.org/abs/2311.07911
- updated_at
- 2025-07-19T19:56:12.241350+00:00
- description
- Instruction-Following Evaluation (IFEval) benchmark for large language models, focusing on verifiable instructions with 25 types of instructions and around 500 prompts containing one or more verifiable constraints
- benchmark_id
- ifeval
- multilingual
- false
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- IFEval · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/gpt-4-1/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 3
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-2024-08-06
Original result ↗0.332
- source id
- llmStatsArchive
- leaderboard name
- SWE-Bench Verified · Normalized (reported)
- dimension
- general
- metric
- SWE-Bench Verified · Normalized (reported)
- rank
- 3
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- SWE-Bench Verified
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- categories
- reasoning
- frontend_development
- code
- created_at
- 2025-07-19T19:56:13.812805+00:00
- paper_link
- https://arxiv.org/abs/2310.06770
- updated_at
- 2025-07-19T19:56:13.812805+00:00
- description
- A verified subset of 500 software engineering problems from real GitHub issues, validated by human annotators for evaluating language models' ability to resolve real-world coding issues by generating patches for Python codebases.
- benchmark_id
- swe-bench-verified
- multilingual
- false
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- SWE-Bench Verified · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/gpt-4-1/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 3
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-2024-08-06
Original result ↗0.332
- source id
- llmStatsArchive
- leaderboard name
- SWE-Bench Verified · Raw
- dimension
- general
- metric
- SWE-Bench Verified · Raw
- rank
- 3
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- SWE-Bench Verified
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- categories
- reasoning
- frontend_development
- code
- created_at
- 2025-07-19T19:56:13.812805+00:00
- paper_link
- https://arxiv.org/abs/2310.06770
- updated_at
- 2025-07-19T19:56:13.812805+00:00
- description
- A verified subset of 500 software engineering problems from real GitHub issues, validated by human annotators for evaluating language models' ability to resolve real-world coding issues by generating patches for Python codebases.
- benchmark_id
- swe-bench-verified
- multilingual
- false
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- SWE-Bench Verified · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/gpt-4-1/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 3
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-2024-08-06
Original result ↗0.182
- source id
- llmStatsArchive
- leaderboard name
- Aider-Polyglot Edit · Raw
- dimension
- general
- metric
- Aider-Polyglot Edit · Raw
- rank
- 3
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- Aider-Polyglot Edit
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:13.789839+00:00
- updated_at
- 2025-09-30T00:00:00.000000+00:00
- description
- A challenging multi-language coding benchmark that evaluates models' code editing abilities across C++, Go, Java, JavaScript, Python, and Rust. Contains 225 of Exercism's most difficult programming problems, selected as problems that were solved by 3 or fewer out of 7 top coding models. The benchmark focuses on code editing tasks and measures both correctness of solutions and proper edit format usage. Designed to re-calibrate evaluation scales so top models score between 5-50%.
- benchmark_id
- aider-polyglot-edit
- multilingual
- false
- implementation_link
- https://github.com/Aider-AI/polyglot-benchmark
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- Aider-Polyglot Edit · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/gpt-4-1/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 3
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-2024-08-06
Original result ↗0.182
- source id
- llmStatsArchive
- leaderboard name
- Aider-Polyglot Edit · Normalized (reported)
- dimension
- general
- metric
- Aider-Polyglot Edit · Normalized (reported)
- rank
- 3
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- Aider-Polyglot Edit
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:13.789839+00:00
- updated_at
- 2025-09-30T00:00:00.000000+00:00
- description
- A challenging multi-language coding benchmark that evaluates models' code editing abilities across C++, Go, Java, JavaScript, Python, and Rust. Contains 225 of Exercism's most difficult programming problems, selected as problems that were solved by 3 or fewer out of 7 top coding models. The benchmark focuses on code editing tasks and measures both correctness of solutions and proper edit format usage. Designed to re-calibrate evaluation scales so top models score between 5-50%.
- benchmark_id
- aider-polyglot-edit
- multilingual
- false
- implementation_link
- https://github.com/Aider-AI/polyglot-benchmark
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- Aider-Polyglot Edit · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/gpt-4-1/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 3
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-2024-08-06
Original result ↗0.131
- source id
- llmStatsArchive
- leaderboard name
- AIME 2024 · Raw
- dimension
- general
- metric
- AIME 2024 · Raw
- rank
- 3
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- AIME 2024
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:11.941652+00:00
- paper_link
- https://arxiv.org/html/2503.21380v2
- updated_at
- 2025-09-30T00:00:00.000000+00:00
- description
- American Invitational Mathematics Examination 2024, consisting of 30 challenging mathematical reasoning problems from AIME I and AIME II competitions. Each problem requires an integer answer between 0-999 and tests advanced mathematical reasoning across algebra, geometry, combinatorics, and number theory. Used as a benchmark for evaluating mathematical reasoning capabilities in large language models at Olympiad-level difficulty.
- benchmark_id
- aime-2024
- multilingual
- false
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- AIME 2024 · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/gpt-4-1/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 3
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-2024-08-06
Original result ↗0.131
- source id
- llmStatsArchive
- leaderboard name
- AIME 2024 · Normalized (reported)
- dimension
- general
- metric
- AIME 2024 · Normalized (reported)
- rank
- 3
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- AIME 2024
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:11.941652+00:00
- paper_link
- https://arxiv.org/html/2503.21380v2
- updated_at
- 2025-09-30T00:00:00.000000+00:00
- description
- American Invitational Mathematics Examination 2024, consisting of 30 challenging mathematical reasoning problems from AIME I and AIME II competitions. Each problem requires an integer answer between 0-999 and tests advanced mathematical reasoning across algebra, geometry, combinatorics, and number theory. Used as a benchmark for evaluating mathematical reasoning capabilities in large language models at Olympiad-level difficulty.
- benchmark_id
- aime-2024
- multilingual
- false
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- AIME 2024 · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/gpt-4-1/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 3
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-2024-08-06
Original result ↗0.609
- source id
- llmStatsArchive
- leaderboard name
- Multi-IF · Normalized (reported)
- dimension
- general
- metric
- Multi-IF · Normalized (reported)
- rank
- 4
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- Multi-IF
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- categories
- reasoning
- communication
- language
- created_at
- 2025-07-19T19:56:14.638787+00:00
- paper_link
- https://arxiv.org/abs/2410.15553
- updated_at
- 2025-07-19T19:56:14.638787+00:00
- description
- Multi-IF benchmarks LLMs on multi-turn and multilingual instruction following. It expands upon IFEval by incorporating multi-turn sequences and translating English prompts into 7 other languages, resulting in 4,501 multilingual conversations with three turns each. The benchmark reveals that current leading LLMs struggle with maintaining accuracy in multi-turn instructions and shows higher error rates for non-Latin script languages.
- benchmark_id
- multi-if
- multilingual
- true
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- Multi-IF · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/gpt-4-1/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 4
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-2024-08-06
Original result ↗0.609
- source id
- llmStatsArchive
- leaderboard name
- Multi-IF · Raw
- dimension
- general
- metric
- Multi-IF · Raw
- rank
- 4
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- Multi-IF
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- categories
- reasoning
- communication
- language
- created_at
- 2025-07-19T19:56:14.638787+00:00
- paper_link
- https://arxiv.org/abs/2410.15553
- updated_at
- 2025-07-19T19:56:14.638787+00:00
- description
- Multi-IF benchmarks LLMs on multi-turn and multilingual instruction following. It expands upon IFEval by incorporating multi-turn sequences and translating English prompts into 7 other languages, resulting in 4,501 multilingual conversations with three turns each. The benchmark reveals that current leading LLMs struggle with maintaining accuracy in multi-turn instructions and shows higher error rates for non-Latin script languages.
- benchmark_id
- multi-if
- multilingual
- true
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- Multi-IF · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/gpt-4-1/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 4
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-2024-08-06
Original result ↗0.382
- source id
- llmStatsArchive
- leaderboard name
- SimpleQA · Raw
- dimension
- general
- metric
- SimpleQA · Raw
- rank
- 4
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- SimpleQA
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-09-05T00:00:00.000000+00:00
- paper_link
- https://arxiv.org/abs/2411.04368
- updated_at
- 2025-09-05T00:00:00.000000+00:00
- description
- SimpleQA is a factuality benchmark developed by OpenAI that measures the short-form factual accuracy of large language models. The benchmark contains 4,326 short, fact-seeking questions that are adversarially collected and designed to have single, indisputable answers. Questions cover diverse topics from science and technology to entertainment, and the benchmark also measures model calibration by evaluating whether models know what they know.
- benchmark_id
- simpleqa
- multilingual
- false
- analysis_method
- accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- SimpleQA · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/introducing-gpt-4-5/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 5
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-2024-08-06
Original result ↗0.382
- source id
- llmStatsArchive
- leaderboard name
- SimpleQA · Normalized (reported)
- dimension
- general
- metric
- SimpleQA · Normalized (reported)
- rank
- 4
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- SimpleQA
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-09-05T00:00:00.000000+00:00
- paper_link
- https://arxiv.org/abs/2411.04368
- updated_at
- 2025-09-05T00:00:00.000000+00:00
- description
- SimpleQA is a factuality benchmark developed by OpenAI that measures the short-form factual accuracy of large language models. The benchmark contains 4,326 short, fact-seeking questions that are adversarially collected and designed to have single, indisputable answers. Questions cover diverse topics from science and technology to entertainment, and the benchmark also measures model calibration by evaluating whether models know what they know.
- benchmark_id
- simpleqa
- multilingual
- false
- analysis_method
- accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- SimpleQA · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/introducing-gpt-4-5/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 5
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-2024-08-06
Original result ↗13.2%
- source id
- kaggleBenchmarks
- leaderboard name
- MultiLoKo Urdu · EM Urdu
- dimension
- reasoning
- metric
- MultiLoKo Urdu · EM Urdu
- rank
- 4
- aggregation scope
- kaggle
- true
- status
- published
- license
- Apache-2.0
- direction
- higherIsBetter
- conditions
- task
- id
- 58
- name
- EM Urdu
- version
- 1
- sort_order
- DESCENDING
- display_type
- percentages
- benchmark
- id
- 81
- name
- MultiLoKo Urdu
- type
- INDIVIDUAL
- owner
- metaresearch
- version
- 1
- categories
- multilingual and cross-lingual capabilities
- general knowledge and reasoning
- version_id
- 67
- maintainer_tier
- Kaggle-maintained
- maintenance_level
- KAGGLE_MAINTAINED
- candidate
- candidate_class
- modelVersion
- record_kind
- aggregate
- attribution
- Kaggle · metaresearch · MultiLoKo Urdu
- license_url
- https://www.apache.org/licenses/LICENSE-2.0
- metric_kind
- MultiLoKo Urdu · EM Urdu
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- 67/leaderboard.json
- source_page
- https://www.kaggle.com/benchmarks/metaresearch/multiloko-urdu/versions/1
- measurements
- name
- EM Urdu
- unit
- percent
- scale
- 1
- metric
- MultiLoKo Urdu · EM Urdu
- result
- resultCase
- numericResult
- booleanResult
- false
- numericResult
- value
- 0.132
- standardDeviation
- 0
- confidenceInterval
- 0.0419589792935838
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0419589792935838
- hasUnevenConfidenceInterval
- false
- taskVersionId
- 0
- evaluationDate
- 2025-06-16T00:00:00Z
- hasBooleanResult
- false
- hasNumericResult
- true
- hasTaskVersionId
- false
- hasEvaluationDate
- true
- numericResultNullable
- value
- 0.132
- standardDeviation
- 0
- confidenceInterval
- 0.0419589792935838
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0419589792935838
- hasUnevenConfidenceInterval
- false
- hasNumericResultPublic
- false
- hasNumericResultPrivate
- false
- task_id
- 58
- sort_order
- DESCENDING
- source_name
- EM Urdu
- display_type
- percentages
- task_version
- 1
- task_display_type
- PERCENTAGES
- normalization_status
- normalized
- published_at
- 2025-07-15T11:49:31.790Z
- candidate_class
- modelVersion
- source_revision
- b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 51
- benchmark_evaluation
- true
- task_metadata_source
- benchmarkVersion
- candidate_metadata_source
- benchmarkVersionMappings
- evaluated at
- 2025-06-16T00:00:00.000Z
- source model id
- GPT-4o
- confidence lower
- 0.0900410207064162
- confidence upper
- 0.1739589792935838
Original result ↗0.614
- source id
- llmStatsArchive
- leaderboard name
- MathVista · Normalized (reported)
- dimension
- general
- metric
- MathVista · Normalized (reported)
- rank
- 5
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- MathVista
- language
- en
- modality
- multimodal
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:12.069611+00:00
- paper_link
- https://arxiv.org/abs/2310.02255
- updated_at
- 2025-07-19T19:56:12.069611+00:00
- description
- MathVista evaluates mathematical reasoning of foundation models in visual contexts. It consists of 6,141 examples derived from 28 existing multimodal datasets and 3 newly created datasets (IQTest, FunctionQA, and PaperQA), combining challenges from diverse mathematical and visual tasks to assess models' ability to understand complex figures and perform rigorous reasoning.
- benchmark_id
- mathvista
- multilingual
- false
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- MathVista · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/gpt-4-1/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 6
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-2024-08-06
Original result ↗0.614
- source id
- llmStatsArchive
- leaderboard name
- MathVista · Raw
- dimension
- general
- metric
- MathVista · Raw
- rank
- 5
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- MathVista
- language
- en
- modality
- multimodal
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:12.069611+00:00
- paper_link
- https://arxiv.org/abs/2310.02255
- updated_at
- 2025-07-19T19:56:12.069611+00:00
- description
- MathVista evaluates mathematical reasoning of foundation models in visual contexts. It consists of 6,141 examples derived from 28 existing multimodal datasets and 3 newly created datasets (IQTest, FunctionQA, and PaperQA), combining challenges from diverse mathematical and visual tasks to assess models' ability to understand complex figures and perform rigorous reasoning.
- benchmark_id
- mathvista
- multilingual
- false
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- MathVista · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/gpt-4-1/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 6
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-2024-08-06
Original result ↗0.307
- source id
- llmStatsArchive
- leaderboard name
- Aider-Polyglot · Normalized (reported)
- dimension
- general
- metric
- Aider-Polyglot · Normalized (reported)
- rank
- 5
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- Aider-Polyglot
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-09-05T00:00:00.000000+00:00
- updated_at
- 2025-09-30T00:00:00.000000+00:00
- description
- A coding benchmark that evaluates LLMs on 225 challenging Exercism programming exercises across C++, Go, Java, JavaScript, Python, and Rust. Models receive two attempts to solve each problem, with test error feedback provided after the first attempt if it fails. The benchmark measures both initial problem-solving ability and capacity to edit code based on error feedback, providing an end-to-end evaluation of code generation and editing capabilities across multiple programming languages.
- benchmark_id
- aider-polyglot
- multilingual
- false
- implementation_link
- https://github.com/Aider-AI/polyglot-benchmark
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- Aider-Polyglot · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/gpt-4-1/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 5
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-2024-08-06
Original result ↗0.307
- source id
- llmStatsArchive
- leaderboard name
- Aider-Polyglot · Raw
- dimension
- general
- metric
- Aider-Polyglot · Raw
- rank
- 5
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- Aider-Polyglot
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-09-05T00:00:00.000000+00:00
- updated_at
- 2025-09-30T00:00:00.000000+00:00
- description
- A coding benchmark that evaluates LLMs on 225 challenging Exercism programming exercises across C++, Go, Java, JavaScript, Python, and Rust. Models receive two attempts to solve each problem, with test error feedback provided after the first attempt if it fails. The benchmark measures both initial problem-solving ability and capacity to edit code based on error feedback, providing an end-to-end evaluation of code generation and editing capabilities across multiple programming languages.
- benchmark_id
- aider-polyglot
- multilingual
- false
- implementation_link
- https://github.com/Aider-AI/polyglot-benchmark
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- Aider-Polyglot · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/gpt-4-1/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 5
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-2024-08-06
Original result ↗32.4%
- source id
- kaggleBenchmarks
- leaderboard name
- MultiLoKo Bengali · EM Bengali
- dimension
- reasoning
- metric
- MultiLoKo Bengali · EM Bengali
- rank
- 6
- aggregation scope
- kaggle
- true
- status
- published
- license
- Apache-2.0
- direction
- higherIsBetter
- conditions
- task
- id
- 31
- name
- EM Bengali
- version
- 1
- sort_order
- DESCENDING
- display_type
- percentages
- benchmark
- id
- 54
- name
- MultiLoKo Bengali
- type
- INDIVIDUAL
- owner
- metaresearch
- version
- 1
- categories
- multilingual and cross-lingual capabilities
- general knowledge and reasoning
- version_id
- 40
- maintainer_tier
- Kaggle-maintained
- maintenance_level
- KAGGLE_MAINTAINED
- candidate
- candidate_class
- modelVersion
- record_kind
- aggregate
- attribution
- Kaggle · metaresearch · MultiLoKo Bengali
- license_url
- https://www.apache.org/licenses/LICENSE-2.0
- metric_kind
- MultiLoKo Bengali · EM Bengali
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- 40/leaderboard.json
- source_page
- https://www.kaggle.com/benchmarks/metaresearch/multiloko-bengali/versions/1
- measurements
- name
- EM Bengali
- unit
- percent
- scale
- 1
- metric
- MultiLoKo Bengali · EM Bengali
- result
- resultCase
- numericResult
- booleanResult
- false
- numericResult
- value
- 0.324
- standardDeviation
- 0
- confidenceInterval
- 0.0580128150237069
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0580128150237069
- hasUnevenConfidenceInterval
- false
- taskVersionId
- 0
- evaluationDate
- 2025-06-16T00:00:00Z
- hasBooleanResult
- false
- hasNumericResult
- true
- hasTaskVersionId
- false
- hasEvaluationDate
- true
- numericResultNullable
- value
- 0.324
- standardDeviation
- 0
- confidenceInterval
- 0.0580128150237069
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0580128150237069
- hasUnevenConfidenceInterval
- false
- hasNumericResultPublic
- false
- hasNumericResultPrivate
- false
- task_id
- 31
- sort_order
- DESCENDING
- source_name
- EM Bengali
- display_type
- percentages
- task_version
- 1
- task_display_type
- PERCENTAGES
- normalization_status
- normalized
- published_at
- 2025-07-15T11:49:31.790Z
- candidate_class
- modelVersion
- source_revision
- b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 51
- benchmark_evaluation
- true
- task_metadata_source
- benchmarkVersion
- candidate_metadata_source
- benchmarkVersionMappings
- evaluated at
- 2025-06-16T00:00:00.000Z
- source model id
- GPT-4o
- confidence lower
- 0.2659871849762931
- confidence upper
- 0.38201281502370693
Original result ↗Published prices
1–3 / 3
$1.25
- Inference provider
- openRouter
- Platform
- OpenRouter
- Unit
- cache_read_tokens_per_million
- Currency
- USD
$2.50
- Inference provider
- openRouter
- Platform
- OpenRouter
- Unit
- input_tokens_per_million
- Currency
- USD
$10.00
- Inference provider
- openRouter
- Platform
- OpenRouter
- Unit
- output_tokens_per_million
- Currency
- USD
Safety assessments
1–1 / 1
35 / 100
- source id
- stanfordFMTI
- canonical model id
- openai/gpt-4o-2024-08-06
- taxonomy
- transparency
- hazard category
- transparency
- hazard name
- Organization transparency
- policy area
- OpenAI
- assessment type
- organization-level disclosure
- metric
- disclosure_score
- result label
- 35 / 100
- sample size
- 100
- updated at
- 2026-11-09T18:08:05.227Z
Source specification observations
1–3 / 3
- field
- context_length
- value
- 128000
- normalized value
- 128000
- scope
- source-model
- source id
- openRouter
- confidence
- 1
- field
- knowledge_cutoff
- reported field
- knowledge_cutoff
- value
- 2023-10-31
- normalized value
- 2023-10-31
- precision
- day
- scope
- source-model
- source id
- openRouter
- confidence
- 1
- field
- max_output_tokens
- value
- 16384
- normalized value
- 16384
- scope
- source-model
- source id
- openRouter
- confidence
- 1
Explore more models