OpenAI: GPT-4o (2024-08-06)

The 2024-08-06 version of GPT-4o offers improved performance in structured outputs, with the ability to supply a JSON schema in the respone_format. Read more [here](https://openai.com/index/introduci…

openai/gpt-4o-2024-08-06

The 2024-08-06 version of GPT-4o offers improved performance in structured outputs, with the ability to supply a JSON schema in the respone_format. Read more [here](https://openai.com/index/introducing-structured-outputs-in-the-api/). GPT-4o ("o" for "omni") is...

Specifications & capabilities

Input
text, image, file
Output
text
Context window · tokens
128000
Knowledge cutoff
2023-10-31
Capabilities
fileInput, multimodal, promptCaching, structuredOutputs, toolCalling, visionInput

Evaluations

21–40 / 143

LLM Stats · GitHub archive

0.814

source id
llmStatsArchive
leaderboard name
MMMLU · Normalized (reported)
dimension
general
metric
MMMLU · Normalized (reported)
rank
3
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
MMMLU
language
en
modality
text
verified
false
max_score
1
categories
  • language
  • reasoning
  • math
  • general
created_at
2025-07-19T19:56:14.144789+00:00
paper_link
https://arxiv.org/abs/2009.03300
updated_at
2025-07-19T19:56:14.144789+00:00
description
Multilingual Massive Multitask Language Understanding dataset released by OpenAI, featuring professionally translated MMLU test questions across 14 languages including Arabic, Bengali, German, Spanish, French, Hindi, Indonesian, Italian, Japanese, Korean, Portuguese, Swahili, Yoruba, and Chinese. Contains approximately 15,908 multiple-choice questions per language covering 57 subjects.
benchmark_id
mmmlu
multilingual
true
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
MMMLU · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/gpt-4-1/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-2024-08-06
metric_population
3
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-2024-08-06
Original result ↗

LLM Stats · GitHub archive

0.814

source id
llmStatsArchive
leaderboard name
MMMLU · Raw
dimension
general
metric
MMMLU · Raw
rank
3
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
MMMLU
language
en
modality
text
verified
false
max_score
1
categories
  • language
  • reasoning
  • math
  • general
created_at
2025-07-19T19:56:14.144789+00:00
paper_link
https://arxiv.org/abs/2009.03300
updated_at
2025-07-19T19:56:14.144789+00:00
description
Multilingual Massive Multitask Language Understanding dataset released by OpenAI, featuring professionally translated MMLU test questions across 14 languages including Arabic, Bengali, German, Spanish, French, Hindi, Indonesian, Italian, Japanese, Korean, Portuguese, Swahili, Yoruba, and Chinese. Contains approximately 15,908 multiple-choice questions per language covering 57 subjects.
benchmark_id
mmmlu
multilingual
true
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
MMMLU · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/gpt-4-1/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-2024-08-06
metric_population
3
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-2024-08-06
Original result ↗

LLM Stats · GitHub archive

0.81

source id
llmStatsArchive
leaderboard name
IFEval · Raw
dimension
general
metric
IFEval · Raw
rank
3
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
IFEval
language
en
modality
text
verified
false
max_score
1
categories
  • general
created_at
2025-07-19T19:56:12.241350+00:00
paper_link
https://arxiv.org/abs/2311.07911
updated_at
2025-07-19T19:56:12.241350+00:00
description
Instruction-Following Evaluation (IFEval) benchmark for large language models, focusing on verifiable instructions with 25 types of instructions and around 500 prompts containing one or more verifiable constraints
benchmark_id
ifeval
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
IFEval · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/gpt-4-1/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-2024-08-06
metric_population
3
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-2024-08-06
Original result ↗

LLM Stats · GitHub archive

0.81

source id
llmStatsArchive
leaderboard name
IFEval · Normalized (reported)
dimension
general
metric
IFEval · Normalized (reported)
rank
3
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
IFEval
language
en
modality
text
verified
false
max_score
1
categories
  • general
created_at
2025-07-19T19:56:12.241350+00:00
paper_link
https://arxiv.org/abs/2311.07911
updated_at
2025-07-19T19:56:12.241350+00:00
description
Instruction-Following Evaluation (IFEval) benchmark for large language models, focusing on verifiable instructions with 25 types of instructions and around 500 prompts containing one or more verifiable constraints
benchmark_id
ifeval
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
IFEval · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/gpt-4-1/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-2024-08-06
metric_population
3
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-2024-08-06
Original result ↗

LLM Stats · GitHub archive

0.332

source id
llmStatsArchive
leaderboard name
SWE-Bench Verified · Normalized (reported)
dimension
general
metric
SWE-Bench Verified · Normalized (reported)
rank
3
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
SWE-Bench Verified
language
en
modality
text
verified
false
max_score
1
categories
  • reasoning
  • frontend_development
  • code
created_at
2025-07-19T19:56:13.812805+00:00
paper_link
https://arxiv.org/abs/2310.06770
updated_at
2025-07-19T19:56:13.812805+00:00
description
A verified subset of 500 software engineering problems from real GitHub issues, validated by human annotators for evaluating language models' ability to resolve real-world coding issues by generating patches for Python codebases.
benchmark_id
swe-bench-verified
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
SWE-Bench Verified · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/gpt-4-1/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-2024-08-06
metric_population
3
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-2024-08-06
Original result ↗

LLM Stats · GitHub archive

0.332

source id
llmStatsArchive
leaderboard name
SWE-Bench Verified · Raw
dimension
general
metric
SWE-Bench Verified · Raw
rank
3
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
SWE-Bench Verified
language
en
modality
text
verified
false
max_score
1
categories
  • reasoning
  • frontend_development
  • code
created_at
2025-07-19T19:56:13.812805+00:00
paper_link
https://arxiv.org/abs/2310.06770
updated_at
2025-07-19T19:56:13.812805+00:00
description
A verified subset of 500 software engineering problems from real GitHub issues, validated by human annotators for evaluating language models' ability to resolve real-world coding issues by generating patches for Python codebases.
benchmark_id
swe-bench-verified
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
SWE-Bench Verified · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/gpt-4-1/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-2024-08-06
metric_population
3
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-2024-08-06
Original result ↗

LLM Stats · GitHub archive

0.182

source id
llmStatsArchive
leaderboard name
Aider-Polyglot Edit · Raw
dimension
general
metric
Aider-Polyglot Edit · Raw
rank
3
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
Aider-Polyglot Edit
language
en
modality
text
verified
false
max_score
1
categories
  • general
  • code
created_at
2025-07-19T19:56:13.789839+00:00
updated_at
2025-09-30T00:00:00.000000+00:00
description
A challenging multi-language coding benchmark that evaluates models' code editing abilities across C++, Go, Java, JavaScript, Python, and Rust. Contains 225 of Exercism's most difficult programming problems, selected as problems that were solved by 3 or fewer out of 7 top coding models. The benchmark focuses on code editing tasks and measures both correctness of solutions and proper edit format usage. Designed to re-calibrate evaluation scales so top models score between 5-50%.
benchmark_id
aider-polyglot-edit
multilingual
false
implementation_link
https://github.com/Aider-AI/polyglot-benchmark
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
Aider-Polyglot Edit · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/gpt-4-1/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-2024-08-06
metric_population
3
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-2024-08-06
Original result ↗

LLM Stats · GitHub archive

0.182

source id
llmStatsArchive
leaderboard name
Aider-Polyglot Edit · Normalized (reported)
dimension
general
metric
Aider-Polyglot Edit · Normalized (reported)
rank
3
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
Aider-Polyglot Edit
language
en
modality
text
verified
false
max_score
1
categories
  • general
  • code
created_at
2025-07-19T19:56:13.789839+00:00
updated_at
2025-09-30T00:00:00.000000+00:00
description
A challenging multi-language coding benchmark that evaluates models' code editing abilities across C++, Go, Java, JavaScript, Python, and Rust. Contains 225 of Exercism's most difficult programming problems, selected as problems that were solved by 3 or fewer out of 7 top coding models. The benchmark focuses on code editing tasks and measures both correctness of solutions and proper edit format usage. Designed to re-calibrate evaluation scales so top models score between 5-50%.
benchmark_id
aider-polyglot-edit
multilingual
false
implementation_link
https://github.com/Aider-AI/polyglot-benchmark
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
Aider-Polyglot Edit · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/gpt-4-1/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-2024-08-06
metric_population
3
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-2024-08-06
Original result ↗

LLM Stats · GitHub archive

0.131

source id
llmStatsArchive
leaderboard name
AIME 2024 · Raw
dimension
general
metric
AIME 2024 · Raw
rank
3
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
AIME 2024
language
en
modality
text
verified
false
max_score
1
categories
  • math
  • reasoning
created_at
2025-07-19T19:56:11.941652+00:00
paper_link
https://arxiv.org/html/2503.21380v2
updated_at
2025-09-30T00:00:00.000000+00:00
description
American Invitational Mathematics Examination 2024, consisting of 30 challenging mathematical reasoning problems from AIME I and AIME II competitions. Each problem requires an integer answer between 0-999 and tests advanced mathematical reasoning across algebra, geometry, combinatorics, and number theory. Used as a benchmark for evaluating mathematical reasoning capabilities in large language models at Olympiad-level difficulty.
benchmark_id
aime-2024
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
AIME 2024 · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/gpt-4-1/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-2024-08-06
metric_population
3
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-2024-08-06
Original result ↗

LLM Stats · GitHub archive

0.131

source id
llmStatsArchive
leaderboard name
AIME 2024 · Normalized (reported)
dimension
general
metric
AIME 2024 · Normalized (reported)
rank
3
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
AIME 2024
language
en
modality
text
verified
false
max_score
1
categories
  • math
  • reasoning
created_at
2025-07-19T19:56:11.941652+00:00
paper_link
https://arxiv.org/html/2503.21380v2
updated_at
2025-09-30T00:00:00.000000+00:00
description
American Invitational Mathematics Examination 2024, consisting of 30 challenging mathematical reasoning problems from AIME I and AIME II competitions. Each problem requires an integer answer between 0-999 and tests advanced mathematical reasoning across algebra, geometry, combinatorics, and number theory. Used as a benchmark for evaluating mathematical reasoning capabilities in large language models at Olympiad-level difficulty.
benchmark_id
aime-2024
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
AIME 2024 · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/gpt-4-1/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-2024-08-06
metric_population
3
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-2024-08-06
Original result ↗

LLM Stats · GitHub archive

0.609

source id
llmStatsArchive
leaderboard name
Multi-IF · Normalized (reported)
dimension
general
metric
Multi-IF · Normalized (reported)
rank
4
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
Multi-IF
language
en
modality
text
verified
false
max_score
1
categories
  • reasoning
  • communication
  • language
created_at
2025-07-19T19:56:14.638787+00:00
paper_link
https://arxiv.org/abs/2410.15553
updated_at
2025-07-19T19:56:14.638787+00:00
description
Multi-IF benchmarks LLMs on multi-turn and multilingual instruction following. It expands upon IFEval by incorporating multi-turn sequences and translating English prompts into 7 other languages, resulting in 4,501 multilingual conversations with three turns each. The benchmark reveals that current leading LLMs struggle with maintaining accuracy in multi-turn instructions and shows higher error rates for non-Latin script languages.
benchmark_id
multi-if
multilingual
true
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
Multi-IF · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/gpt-4-1/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-2024-08-06
metric_population
4
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-2024-08-06
Original result ↗

LLM Stats · GitHub archive

0.609

source id
llmStatsArchive
leaderboard name
Multi-IF · Raw
dimension
general
metric
Multi-IF · Raw
rank
4
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
Multi-IF
language
en
modality
text
verified
false
max_score
1
categories
  • reasoning
  • communication
  • language
created_at
2025-07-19T19:56:14.638787+00:00
paper_link
https://arxiv.org/abs/2410.15553
updated_at
2025-07-19T19:56:14.638787+00:00
description
Multi-IF benchmarks LLMs on multi-turn and multilingual instruction following. It expands upon IFEval by incorporating multi-turn sequences and translating English prompts into 7 other languages, resulting in 4,501 multilingual conversations with three turns each. The benchmark reveals that current leading LLMs struggle with maintaining accuracy in multi-turn instructions and shows higher error rates for non-Latin script languages.
benchmark_id
multi-if
multilingual
true
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
Multi-IF · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/gpt-4-1/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-2024-08-06
metric_population
4
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-2024-08-06
Original result ↗

LLM Stats · GitHub archive

0.382

source id
llmStatsArchive
leaderboard name
SimpleQA · Raw
dimension
general
metric
SimpleQA · Raw
rank
4
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
SimpleQA
language
en
modality
text
verified
false
max_score
1
categories
  • general
  • reasoning
created_at
2025-09-05T00:00:00.000000+00:00
paper_link
https://arxiv.org/abs/2411.04368
updated_at
2025-09-05T00:00:00.000000+00:00
description
SimpleQA is a factuality benchmark developed by OpenAI that measures the short-form factual accuracy of large language models. The benchmark contains 4,326 short, fact-seeking questions that are adversarially collected and designed to have single, indisputable answers. Questions cover diverse topics from science and technology to entertainment, and the benchmark also measures model calibration by evaluating whether models know what they know.
benchmark_id
simpleqa
multilingual
false
analysis_method
accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
SimpleQA · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/introducing-gpt-4-5/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-2024-08-06
metric_population
5
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-2024-08-06
Original result ↗

LLM Stats · GitHub archive

0.382

source id
llmStatsArchive
leaderboard name
SimpleQA · Normalized (reported)
dimension
general
metric
SimpleQA · Normalized (reported)
rank
4
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
SimpleQA
language
en
modality
text
verified
false
max_score
1
categories
  • general
  • reasoning
created_at
2025-09-05T00:00:00.000000+00:00
paper_link
https://arxiv.org/abs/2411.04368
updated_at
2025-09-05T00:00:00.000000+00:00
description
SimpleQA is a factuality benchmark developed by OpenAI that measures the short-form factual accuracy of large language models. The benchmark contains 4,326 short, fact-seeking questions that are adversarially collected and designed to have single, indisputable answers. Questions cover diverse topics from science and technology to entertainment, and the benchmark also measures model calibration by evaluating whether models know what they know.
benchmark_id
simpleqa
multilingual
false
analysis_method
accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
SimpleQA · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/introducing-gpt-4-5/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-2024-08-06
metric_population
5
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-2024-08-06
Original result ↗

Kaggle Benchmarks

13.2%

source id
kaggleBenchmarks
leaderboard name
MultiLoKo Urdu · EM Urdu
dimension
reasoning
metric
MultiLoKo Urdu · EM Urdu
rank
4
aggregation scope
kaggle
true
status
published
license
Apache-2.0
direction
higherIsBetter
conditions
task
id
58
name
EM Urdu
version
1
sort_order
DESCENDING
display_type
percentages
benchmark
id
81
name
MultiLoKo Urdu
type
INDIVIDUAL
owner
metaresearch
version
1
categories
  • multilingual and cross-lingual capabilities
  • general knowledge and reasoning
version_id
67
maintainer_tier
Kaggle-maintained
maintenance_level
KAGGLE_MAINTAINED
candidate
candidate_class
modelVersion
record_kind
aggregate
attribution
Kaggle · metaresearch · MultiLoKo Urdu
license_url
https://www.apache.org/licenses/LICENSE-2.0
metric_kind
MultiLoKo Urdu · EM Urdu
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
67/leaderboard.json
source_page
https://www.kaggle.com/benchmarks/metaresearch/multiloko-urdu/versions/1
measurements
  • name
    EM Urdu
    unit
    percent
    scale
    1
    metric
    MultiLoKo Urdu · EM Urdu
    result
    resultCase
    numericResult
    booleanResult
    false
    numericResult
    value
    0.132
    standardDeviation
    0
    confidenceInterval
    0.0419589792935838
    hasStandardDeviation
    false
    hasConfidenceInterval
    true
    confidenceIntervalNullable
    0.0419589792935838
    hasUnevenConfidenceInterval
    false
    taskVersionId
    0
    evaluationDate
    2025-06-16T00:00:00Z
    hasBooleanResult
    false
    hasNumericResult
    true
    hasTaskVersionId
    false
    hasEvaluationDate
    true
    numericResultNullable
    value
    0.132
    standardDeviation
    0
    confidenceInterval
    0.0419589792935838
    hasStandardDeviation
    false
    hasConfidenceInterval
    true
    confidenceIntervalNullable
    0.0419589792935838
    hasUnevenConfidenceInterval
    false
    hasNumericResultPublic
    false
    customAdditionalResults
      hasNumericResultPrivate
      false
      task_id
      58
      sort_order
      DESCENDING
      source_name
      EM Urdu
      display_type
      percentages
      task_version
      1
      task_display_type
      PERCENTAGES
      normalization_status
      normalized
    published_at
    2025-07-15T11:49:31.790Z
    candidate_class
    modelVersion
    source_revision
    b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
    publisher_anchor
    openai/gpt-4o-2024-08-06
    metric_population
    51
    benchmark_evaluation
    true
    task_metadata_source
    benchmarkVersion
    candidate_metadata_source
    benchmarkVersionMappings
    evaluated at
    2025-06-16T00:00:00.000Z
    source model id
    GPT-4o
    confidence lower
    0.0900410207064162
    confidence upper
    0.1739589792935838
    Original result ↗

    LLM Stats · GitHub archive

    0.614

    source id
    llmStatsArchive
    leaderboard name
    MathVista · Normalized (reported)
    dimension
    general
    metric
    MathVista · Normalized (reported)
    rank
    5
    aggregation scope
    status
    archived
    license
    CC-BY-4.0
    direction
    higherIsBetter
    conditions
    language
    en
    benchmark
    name
    MathVista
    language
    en
    modality
    multimodal
    verified
    false
    max_score
    1
    categories
    • math
    • vision
    • multimodal
    created_at
    2025-07-19T19:56:12.069611+00:00
    paper_link
    https://arxiv.org/abs/2310.02255
    updated_at
    2025-07-19T19:56:12.069611+00:00
    description
    MathVista evaluates mathematical reasoning of foundation models in visual contexts. It consists of 6,141 examples derived from 28 existing multimodal datasets and 3 newly created datasets (IQTest, FunctionQA, and PaperQA), combining challenges from diverse mathematical and visual tasks to assess models' ability to understand complex figures and perform rigorous reasoning.
    benchmark_id
    mathvista
    multilingual
    false
    analysis_method
    Accuracy
    is_self_reported
    true
    verified_by_llmstats
    false
    attribution
    Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
    license_url
    https://creativecommons.org/licenses/by/4.0/
    metric_kind
    MathVista · Normalized (reported)
    rank_method
    competition rank among reported evaluations in identical conditions, including unlinked observations
    source_file
    data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
    archive_kind
    llmStats
    evidence_url
    https://openai.com/index/gpt-4-1/
    git_revision
    872b75f63b8d853c46ed17856424029a0c47b1d2
    published_at
    2025-10-24T17:47:59Z
    snapshot_date
    2025-10-24T17:47:59Z
    source_revision
    6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
    publisher_anchor
    openai/gpt-4o-2024-08-06
    metric_population
    6
    current_catalog_url
    https://llm-stats.com/
    benchmark_evaluation
    true
    source model id
    gpt-4o-2024-08-06
    Original result ↗

    LLM Stats · GitHub archive

    0.614

    source id
    llmStatsArchive
    leaderboard name
    MathVista · Raw
    dimension
    general
    metric
    MathVista · Raw
    rank
    5
    aggregation scope
    status
    archived
    license
    CC-BY-4.0
    direction
    higherIsBetter
    conditions
    language
    en
    benchmark
    name
    MathVista
    language
    en
    modality
    multimodal
    verified
    false
    max_score
    1
    categories
    • math
    • vision
    • multimodal
    created_at
    2025-07-19T19:56:12.069611+00:00
    paper_link
    https://arxiv.org/abs/2310.02255
    updated_at
    2025-07-19T19:56:12.069611+00:00
    description
    MathVista evaluates mathematical reasoning of foundation models in visual contexts. It consists of 6,141 examples derived from 28 existing multimodal datasets and 3 newly created datasets (IQTest, FunctionQA, and PaperQA), combining challenges from diverse mathematical and visual tasks to assess models' ability to understand complex figures and perform rigorous reasoning.
    benchmark_id
    mathvista
    multilingual
    false
    analysis_method
    Accuracy
    is_self_reported
    true
    verified_by_llmstats
    false
    attribution
    Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
    license_url
    https://creativecommons.org/licenses/by/4.0/
    metric_kind
    MathVista · Raw
    rank_method
    competition rank among reported evaluations in identical conditions, including unlinked observations
    source_file
    data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
    archive_kind
    llmStats
    evidence_url
    https://openai.com/index/gpt-4-1/
    git_revision
    872b75f63b8d853c46ed17856424029a0c47b1d2
    published_at
    2025-10-24T17:47:59Z
    snapshot_date
    2025-10-24T17:47:59Z
    source_revision
    6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
    publisher_anchor
    openai/gpt-4o-2024-08-06
    metric_population
    6
    current_catalog_url
    https://llm-stats.com/
    benchmark_evaluation
    true
    source model id
    gpt-4o-2024-08-06
    Original result ↗

    LLM Stats · GitHub archive

    0.307

    source id
    llmStatsArchive
    leaderboard name
    Aider-Polyglot · Normalized (reported)
    dimension
    general
    metric
    Aider-Polyglot · Normalized (reported)
    rank
    5
    aggregation scope
    status
    archived
    license
    CC-BY-4.0
    direction
    higherIsBetter
    conditions
    language
    en
    benchmark
    name
    Aider-Polyglot
    language
    en
    modality
    text
    verified
    false
    max_score
    1
    categories
    • general
    • code
    created_at
    2025-09-05T00:00:00.000000+00:00
    updated_at
    2025-09-30T00:00:00.000000+00:00
    description
    A coding benchmark that evaluates LLMs on 225 challenging Exercism programming exercises across C++, Go, Java, JavaScript, Python, and Rust. Models receive two attempts to solve each problem, with test error feedback provided after the first attempt if it fails. The benchmark measures both initial problem-solving ability and capacity to edit code based on error feedback, providing an end-to-end evaluation of code generation and editing capabilities across multiple programming languages.
    benchmark_id
    aider-polyglot
    multilingual
    false
    implementation_link
    https://github.com/Aider-AI/polyglot-benchmark
    analysis_method
    Accuracy
    is_self_reported
    true
    verified_by_llmstats
    false
    attribution
    Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
    license_url
    https://creativecommons.org/licenses/by/4.0/
    metric_kind
    Aider-Polyglot · Normalized (reported)
    rank_method
    competition rank among reported evaluations in identical conditions, including unlinked observations
    source_file
    data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
    archive_kind
    llmStats
    evidence_url
    https://openai.com/index/gpt-4-1/
    git_revision
    872b75f63b8d853c46ed17856424029a0c47b1d2
    published_at
    2025-10-24T17:47:59Z
    snapshot_date
    2025-10-24T17:47:59Z
    source_revision
    6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
    publisher_anchor
    openai/gpt-4o-2024-08-06
    metric_population
    5
    current_catalog_url
    https://llm-stats.com/
    benchmark_evaluation
    true
    source model id
    gpt-4o-2024-08-06
    Original result ↗

    LLM Stats · GitHub archive

    0.307

    source id
    llmStatsArchive
    leaderboard name
    Aider-Polyglot · Raw
    dimension
    general
    metric
    Aider-Polyglot · Raw
    rank
    5
    aggregation scope
    status
    archived
    license
    CC-BY-4.0
    direction
    higherIsBetter
    conditions
    language
    en
    benchmark
    name
    Aider-Polyglot
    language
    en
    modality
    text
    verified
    false
    max_score
    1
    categories
    • general
    • code
    created_at
    2025-09-05T00:00:00.000000+00:00
    updated_at
    2025-09-30T00:00:00.000000+00:00
    description
    A coding benchmark that evaluates LLMs on 225 challenging Exercism programming exercises across C++, Go, Java, JavaScript, Python, and Rust. Models receive two attempts to solve each problem, with test error feedback provided after the first attempt if it fails. The benchmark measures both initial problem-solving ability and capacity to edit code based on error feedback, providing an end-to-end evaluation of code generation and editing capabilities across multiple programming languages.
    benchmark_id
    aider-polyglot
    multilingual
    false
    implementation_link
    https://github.com/Aider-AI/polyglot-benchmark
    analysis_method
    Accuracy
    is_self_reported
    true
    verified_by_llmstats
    false
    attribution
    Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
    license_url
    https://creativecommons.org/licenses/by/4.0/
    metric_kind
    Aider-Polyglot · Raw
    rank_method
    competition rank among reported evaluations in identical conditions, including unlinked observations
    source_file
    data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
    archive_kind
    llmStats
    evidence_url
    https://openai.com/index/gpt-4-1/
    git_revision
    872b75f63b8d853c46ed17856424029a0c47b1d2
    published_at
    2025-10-24T17:47:59Z
    snapshot_date
    2025-10-24T17:47:59Z
    source_revision
    6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
    publisher_anchor
    openai/gpt-4o-2024-08-06
    metric_population
    5
    current_catalog_url
    https://llm-stats.com/
    benchmark_evaluation
    true
    source model id
    gpt-4o-2024-08-06
    Original result ↗

    Kaggle Benchmarks

    32.4%

    source id
    kaggleBenchmarks
    leaderboard name
    MultiLoKo Bengali · EM Bengali
    dimension
    reasoning
    metric
    MultiLoKo Bengali · EM Bengali
    rank
    6
    aggregation scope
    kaggle
    true
    status
    published
    license
    Apache-2.0
    direction
    higherIsBetter
    conditions
    task
    id
    31
    name
    EM Bengali
    version
    1
    sort_order
    DESCENDING
    display_type
    percentages
    benchmark
    id
    54
    name
    MultiLoKo Bengali
    type
    INDIVIDUAL
    owner
    metaresearch
    version
    1
    categories
    • multilingual and cross-lingual capabilities
    • general knowledge and reasoning
    version_id
    40
    maintainer_tier
    Kaggle-maintained
    maintenance_level
    KAGGLE_MAINTAINED
    candidate
    candidate_class
    modelVersion
    record_kind
    aggregate
    attribution
    Kaggle · metaresearch · MultiLoKo Bengali
    license_url
    https://www.apache.org/licenses/LICENSE-2.0
    metric_kind
    MultiLoKo Bengali · EM Bengali
    rank_method
    competition rank among reported evaluations in identical conditions, including unlinked observations
    source_file
    40/leaderboard.json
    source_page
    https://www.kaggle.com/benchmarks/metaresearch/multiloko-bengali/versions/1
    measurements
    • name
      EM Bengali
      unit
      percent
      scale
      1
      metric
      MultiLoKo Bengali · EM Bengali
      result
      resultCase
      numericResult
      booleanResult
      false
      numericResult
      value
      0.324
      standardDeviation
      0
      confidenceInterval
      0.0580128150237069
      hasStandardDeviation
      false
      hasConfidenceInterval
      true
      confidenceIntervalNullable
      0.0580128150237069
      hasUnevenConfidenceInterval
      false
      taskVersionId
      0
      evaluationDate
      2025-06-16T00:00:00Z
      hasBooleanResult
      false
      hasNumericResult
      true
      hasTaskVersionId
      false
      hasEvaluationDate
      true
      numericResultNullable
      value
      0.324
      standardDeviation
      0
      confidenceInterval
      0.0580128150237069
      hasStandardDeviation
      false
      hasConfidenceInterval
      true
      confidenceIntervalNullable
      0.0580128150237069
      hasUnevenConfidenceInterval
      false
      hasNumericResultPublic
      false
      customAdditionalResults
        hasNumericResultPrivate
        false
        task_id
        31
        sort_order
        DESCENDING
        source_name
        EM Bengali
        display_type
        percentages
        task_version
        1
        task_display_type
        PERCENTAGES
        normalization_status
        normalized
      published_at
      2025-07-15T11:49:31.790Z
      candidate_class
      modelVersion
      source_revision
      b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
      publisher_anchor
      openai/gpt-4o-2024-08-06
      metric_population
      51
      benchmark_evaluation
      true
      task_metadata_source
      benchmarkVersion
      candidate_metadata_source
      benchmarkVersionMappings
      evaluated at
      2025-06-16T00:00:00.000Z
      source model id
      GPT-4o
      confidence lower
      0.2659871849762931
      confidence upper
      0.38201281502370693
      Original result ↗

      Published prices

      1–3 / 3

      openai/gpt-4o-2024-08-06 ↗

      $1.25

      Inference provider
      openRouter
      Platform
      OpenRouter
      Unit
      cache_read_tokens_per_million
      Currency
      USD
      Billing conditions

      Safety assessments

      1–1 / 1

      Foundation Model Transparency Index 2025 ↗

      35 / 100

      source id
      stanfordFMTI
      canonical model id
      openai/gpt-4o-2024-08-06
      taxonomy
      transparency
      hazard category
      transparency
      hazard name
      Organization transparency
      policy area
      OpenAI
      assessment type
      organization-level disclosure
      metric
      disclosure_score
      result label
      35 / 100
      sample size
      100
      updated at
      2026-11-09T18:08:05.227Z

      Source specification observations

      1–3 / 3

      openRouter ↗

      field
      context_length
      value
      128000
      normalized value
      128000
      scope
      source-model
      source id
      openRouter
      confidence
      1

      openRouter ↗

      field
      knowledge_cutoff
      reported field
      knowledge_cutoff
      value
      2023-10-31
      normalized value
      2023-10-31
      precision
      day
      scope
      source-model
      source id
      openRouter
      confidence
      1

      openRouter ↗

      field
      max_output_tokens
      value
      16384
      normalized value
      16384
      scope
      source-model
      source id
      openRouter
      confidence
      1

      Original sources

      1–7 / 7

      OpenRouter · Model page ↗

      GitHub · Catalog ↗

      GitHub · Catalog ↗

      Kaggle · Catalog ↗

      OpenRouter · Catalog ↗

      OpenRouter · Catalog ↗

      www.swebench.com · Catalog ↗

      Source links

      1–7 / 7

      Explore more models

      Models

      Loading
      EVIDENCE / COMPARISON

      Benchmarks

      Explore every benchmark

      Loading

      These charts contain model-linked results. The charts above also include unlinked source profiles. Compare scores within each evaluation. Model quality, usage, transparency and system performance measure different things.

      Install TensorTrace