OpenAI: GPT-4o-mini (2024-07-18)

GPT-4o mini is OpenAI's newest model after [GPT-4 Omni](/models/openai/gpt-4o), supporting both text and image inputs with text outputs. As their most advanced small model, it is many multiples more …

openai/gpt-4o-mini-2024-07-18

GPT-4o mini is OpenAI's newest model after [GPT-4 Omni](/models/openai/gpt-4o), supporting both text and image inputs with text outputs. As their most advanced small model, it is many multiples more affordable...

Specifications & capabilities

Input
text, image, file
Output
text
Context window · tokens
128000
Capabilities
audioInput, fileInput, multimodal, promptCaching, structuredOutputs, toolCalling, visionInput

Evaluations

1–20 / 186

LLM Stats · GitHub archive

0.087

source id
llmStatsArchive
leaderboard name
SWE-Bench Verified · Normalized (reported)
dimension
general
metric
SWE-Bench Verified · Normalized (reported)
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
SWE-Bench Verified
language
en
modality
text
verified
false
max_score
1
categories
  • reasoning
  • frontend_development
  • code
created_at
2025-07-19T19:56:13.812805+00:00
paper_link
https://arxiv.org/abs/2310.06770
updated_at
2025-07-19T19:56:13.812805+00:00
description
A verified subset of 500 software engineering problems from real GitHub issues, validated by human annotators for evaluating language models' ability to resolve real-world coding issues by generating patches for Python codebases.
benchmark_id
swe-bench-verified
multilingual
false
analysis_method
Pass Rate
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
SWE-Bench Verified · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-mini-2024-07-18/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/gpt-4-1/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-mini-2024-07-18
metric_population
1
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-mini-2024-07-18
Original result ↗

LLM Stats · GitHub archive

0.087

source id
llmStatsArchive
leaderboard name
SWE-Bench Verified · Raw
dimension
general
metric
SWE-Bench Verified · Raw
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
SWE-Bench Verified
language
en
modality
text
verified
false
max_score
1
categories
  • reasoning
  • frontend_development
  • code
created_at
2025-07-19T19:56:13.812805+00:00
paper_link
https://arxiv.org/abs/2310.06770
updated_at
2025-07-19T19:56:13.812805+00:00
description
A verified subset of 500 software engineering problems from real GitHub issues, validated by human annotators for evaluating language models' ability to resolve real-world coding issues by generating patches for Python codebases.
benchmark_id
swe-bench-verified
multilingual
false
analysis_method
Pass Rate
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
SWE-Bench Verified · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-mini-2024-07-18/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/gpt-4-1/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-mini-2024-07-18
metric_population
1
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-mini-2024-07-18
Original result ↗

LLM Stats · GitHub archive

0.87

source id
llmStatsArchive
leaderboard name
MGSM · Raw
dimension
general
metric
MGSM · Raw
rank
2
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
MGSM
language
en
modality
text
verified
false
max_score
1
categories
  • math
  • reasoning
created_at
2025-07-19T19:56:13.669061+00:00
paper_link
https://arxiv.org/abs/2210.03057
updated_at
2025-07-19T19:56:13.669061+00:00
description
MGSM (Multilingual Grade School Math) is a benchmark of grade-school math problems. Contains 250 grade-school math problems manually translated from the GSM8K dataset into ten typologically diverse languages: Spanish, French, German, Russian, Chinese, Japanese, Thai, Swahili, Bengali, and Telugu. Evaluates multilingual mathematical reasoning capabilities.
benchmark_id
mgsm
multilingual
true
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
MGSM · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-mini-2024-07-18/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/blog/gpt-4o-mini-announcement
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-mini-2024-07-18
metric_population
3
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-mini-2024-07-18
Original result ↗

LLM Stats · GitHub archive

0.87

source id
llmStatsArchive
leaderboard name
MGSM · Normalized (reported)
dimension
general
metric
MGSM · Normalized (reported)
rank
2
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
MGSM
language
en
modality
text
verified
false
max_score
1
categories
  • math
  • reasoning
created_at
2025-07-19T19:56:13.669061+00:00
paper_link
https://arxiv.org/abs/2210.03057
updated_at
2025-07-19T19:56:13.669061+00:00
description
MGSM (Multilingual Grade School Math) is a benchmark of grade-school math problems. Contains 250 grade-school math problems manually translated from the GSM8K dataset into ten typologically diverse languages: Spanish, French, German, Russian, Chinese, Japanese, Thai, Swahili, Bengali, and Telugu. Evaluates multilingual mathematical reasoning capabilities.
benchmark_id
mgsm
multilingual
true
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
MGSM · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-mini-2024-07-18/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/blog/gpt-4o-mini-announcement
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-mini-2024-07-18
metric_population
3
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-mini-2024-07-18
Original result ↗

LLM Stats · GitHub archive

0.797

source id
llmStatsArchive
leaderboard name
DROP · Normalized (reported)
dimension
general
metric
DROP · Normalized (reported)
rank
2
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
DROP
language
en
modality
text
verified
false
max_score
1
categories
  • reasoning
  • math
created_at
2025-07-19T19:56:12.981569+00:00
paper_link
https://arxiv.org/abs/1903.00161
updated_at
2025-07-19T19:56:12.981569+00:00
description
DROP (Discrete Reasoning Over Paragraphs) is a reading comprehension benchmark requiring discrete reasoning over paragraph content. It contains crowdsourced, adversarially-created questions that require resolving references and performing discrete operations like addition, counting, or sorting, demanding comprehensive paragraph understanding beyond paraphrase-and-entity-typing shortcuts.
benchmark_id
drop
multilingual
false
analysis_method
F1 Score
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
DROP · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-mini-2024-07-18/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/blog/gpt-4o-mini-announcement
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-mini-2024-07-18
metric_population
2
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-mini-2024-07-18
Original result ↗

LLM Stats · GitHub archive

0.797

source id
llmStatsArchive
leaderboard name
DROP · Raw
dimension
general
metric
DROP · Raw
rank
2
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
DROP
language
en
modality
text
verified
false
max_score
1
categories
  • reasoning
  • math
created_at
2025-07-19T19:56:12.981569+00:00
paper_link
https://arxiv.org/abs/1903.00161
updated_at
2025-07-19T19:56:12.981569+00:00
description
DROP (Discrete Reasoning Over Paragraphs) is a reading comprehension benchmark requiring discrete reasoning over paragraph content. It contains crowdsourced, adversarially-created questions that require resolving references and performing discrete operations like addition, counting, or sorting, demanding comprehensive paragraph understanding beyond paraphrase-and-entity-typing shortcuts.
benchmark_id
drop
multilingual
false
analysis_method
F1 Score
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
DROP · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-mini-2024-07-18/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/blog/gpt-4o-mini-announcement
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-mini-2024-07-18
metric_population
2
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-mini-2024-07-18
Original result ↗

LLM Stats · GitHub archive

0.594

source id
llmStatsArchive
leaderboard name
MMMU · Raw
dimension
general
metric
MMMU · Raw
rank
4
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
MMMU
language
en
modality
multimodal
verified
false
max_score
1
categories
  • multimodal
  • reasoning
  • general
created_at
2025-07-19T19:56:12.130105+00:00
paper_link
https://arxiv.org/abs/2311.16502
updated_at
2025-07-19T19:56:12.130105+00:00
description
MMMU (Massive Multi-discipline Multimodal Understanding) is a benchmark designed to evaluate multimodal models on college-level subject knowledge and deliberate reasoning. Contains 11.5K meticulously collected multimodal questions from college exams, quizzes, and textbooks, covering six core disciplines: Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering across 30 subjects and 183 subfields.
benchmark_id
mmmu
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
MMMU · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-mini-2024-07-18/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/blog/gpt-4o-mini-announcement
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-mini-2024-07-18
metric_population
4
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-mini-2024-07-18
Original result ↗

LLM Stats · GitHub archive

0.594

source id
llmStatsArchive
leaderboard name
MMMU · Normalized (reported)
dimension
general
metric
MMMU · Normalized (reported)
rank
4
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
MMMU
language
en
modality
multimodal
verified
false
max_score
1
categories
  • multimodal
  • reasoning
  • general
created_at
2025-07-19T19:56:12.130105+00:00
paper_link
https://arxiv.org/abs/2311.16502
updated_at
2025-07-19T19:56:12.130105+00:00
description
MMMU (Massive Multi-discipline Multimodal Understanding) is a benchmark designed to evaluate multimodal models on college-level subject knowledge and deliberate reasoning. Contains 11.5K meticulously collected multimodal questions from college exams, quizzes, and textbooks, covering six core disciplines: Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering across 30 subjects and 183 subfields.
benchmark_id
mmmu
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
MMMU · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-mini-2024-07-18/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/blog/gpt-4o-mini-announcement
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-mini-2024-07-18
metric_population
4
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-mini-2024-07-18
Original result ↗

LLM Stats · GitHub archive

0.702

source id
llmStatsArchive
leaderboard name
MATH · Raw
dimension
general
metric
MATH · Raw
rank
5
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
MATH
language
en
modality
text
verified
false
max_score
1
categories
  • math
  • reasoning
created_at
2025-07-19T19:56:11.804258+00:00
paper_link
https://arxiv.org/abs/2103.03874
updated_at
2025-07-19T19:56:11.804258+00:00
description
MATH dataset contains 12,500 challenging competition mathematics problems from AMC 10, AMC 12, AIME, and other mathematics competitions. Each problem includes full step-by-step solutions and spans multiple difficulty levels (1-5) across seven mathematical subjects including Prealgebra, Algebra, Number Theory, Counting and Probability, Geometry, Intermediate Algebra, and Precalculus.
benchmark_id
math
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
MATH · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-mini-2024-07-18/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/blog/gpt-4o-mini-announcement
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-mini-2024-07-18
metric_population
7
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-mini-2024-07-18
Original result ↗

LLM Stats · GitHub archive

0.702

source id
llmStatsArchive
leaderboard name
MATH · Normalized (reported)
dimension
general
metric
MATH · Normalized (reported)
rank
5
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
MATH
language
en
modality
text
verified
false
max_score
1
categories
  • math
  • reasoning
created_at
2025-07-19T19:56:11.804258+00:00
paper_link
https://arxiv.org/abs/2103.03874
updated_at
2025-07-19T19:56:11.804258+00:00
description
MATH dataset contains 12,500 challenging competition mathematics problems from AMC 10, AMC 12, AIME, and other mathematics competitions. Each problem includes full step-by-step solutions and spans multiple difficulty levels (1-5) across seven mathematical subjects including Prealgebra, Algebra, Number Theory, Counting and Probability, Geometry, Intermediate Algebra, and Precalculus.
benchmark_id
math
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
MATH · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-mini-2024-07-18/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/blog/gpt-4o-mini-announcement
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-mini-2024-07-18
metric_population
7
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-mini-2024-07-18
Original result ↗

LLM Stats · GitHub archive

0.872

source id
llmStatsArchive
leaderboard name
HumanEval · Raw
dimension
general
metric
HumanEval · Raw
rank
6
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
HumanEval
language
en
modality
text
verified
false
max_score
1
categories
  • reasoning
  • code
created_at
2025-07-19T19:56:12.595263+00:00
paper_link
https://arxiv.org/abs/2107.03374
updated_at
2025-07-19T19:56:12.595263+00:00
description
A benchmark that measures functional correctness for synthesizing programs from docstrings, consisting of 164 original programming problems assessing language comprehension, algorithms, and simple mathematics
benchmark_id
humaneval
multilingual
false
analysis_method
Pass@1
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
HumanEval · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-mini-2024-07-18/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/blog/gpt-4o-mini-announcement
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-mini-2024-07-18
metric_population
9
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-mini-2024-07-18
Original result ↗

LLM Stats · GitHub archive

0.872

source id
llmStatsArchive
leaderboard name
HumanEval · Normalized (reported)
dimension
general
metric
HumanEval · Normalized (reported)
rank
6
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
HumanEval
language
en
modality
text
verified
false
max_score
1
categories
  • reasoning
  • code
created_at
2025-07-19T19:56:12.595263+00:00
paper_link
https://arxiv.org/abs/2107.03374
updated_at
2025-07-19T19:56:12.595263+00:00
description
A benchmark that measures functional correctness for synthesizing programs from docstrings, consisting of 164 original programming problems assessing language comprehension, algorithms, and simple mathematics
benchmark_id
humaneval
multilingual
false
analysis_method
Pass@1
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
HumanEval · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-mini-2024-07-18/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/blog/gpt-4o-mini-announcement
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-mini-2024-07-18
metric_population
9
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-mini-2024-07-18
Original result ↗

LLM Stats · GitHub archive

0.82

source id
llmStatsArchive
leaderboard name
MMLU · Raw
dimension
general
metric
MMLU · Raw
rank
6
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
MMLU
language
en
modality
text
verified
false
max_score
1
categories
  • general
  • reasoning
  • language
  • math
created_at
2025-07-19T19:56:11.200416+00:00
paper_link
https://arxiv.org/abs/2009.03300
updated_at
2025-07-19T19:56:11.200416+00:00
description
Massive Multitask Language Understanding benchmark testing knowledge across 57 diverse subjects including STEM, humanities, social sciences, and professional domains
benchmark_id
mmlu
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
MMLU · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-mini-2024-07-18/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/blog/gpt-4o-mini-announcement
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-mini-2024-07-18
metric_population
8
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-mini-2024-07-18
Original result ↗

LLM Stats · GitHub archive

0.82

source id
llmStatsArchive
leaderboard name
MMLU · Normalized (reported)
dimension
general
metric
MMLU · Normalized (reported)
rank
6
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
MMLU
language
en
modality
text
verified
false
max_score
1
categories
  • general
  • reasoning
  • language
  • math
created_at
2025-07-19T19:56:11.200416+00:00
paper_link
https://arxiv.org/abs/2009.03300
updated_at
2025-07-19T19:56:11.200416+00:00
description
Massive Multitask Language Understanding benchmark testing knowledge across 57 diverse subjects including STEM, humanities, social sciences, and professional domains
benchmark_id
mmlu
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
MMLU · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-mini-2024-07-18/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/blog/gpt-4o-mini-announcement
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-mini-2024-07-18
metric_population
8
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-mini-2024-07-18
Original result ↗

LLM Stats · GitHub archive

0.567

source id
llmStatsArchive
leaderboard name
MathVista · Normalized (reported)
dimension
general
metric
MathVista · Normalized (reported)
rank
6
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
MathVista
language
en
modality
multimodal
verified
false
max_score
1
categories
  • math
  • vision
  • multimodal
created_at
2025-07-19T19:56:12.069611+00:00
paper_link
https://arxiv.org/abs/2310.02255
updated_at
2025-07-19T19:56:12.069611+00:00
description
MathVista evaluates mathematical reasoning of foundation models in visual contexts. It consists of 6,141 examples derived from 28 existing multimodal datasets and 3 newly created datasets (IQTest, FunctionQA, and PaperQA), combining challenges from diverse mathematical and visual tasks to assess models' ability to understand complex figures and perform rigorous reasoning.
benchmark_id
mathvista
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
MathVista · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-mini-2024-07-18/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/blog/gpt-4o-mini-announcement
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-mini-2024-07-18
metric_population
6
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-mini-2024-07-18
Original result ↗

LLM Stats · GitHub archive

0.567

source id
llmStatsArchive
leaderboard name
MathVista · Raw
dimension
general
metric
MathVista · Raw
rank
6
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
MathVista
language
en
modality
multimodal
verified
false
max_score
1
categories
  • math
  • vision
  • multimodal
created_at
2025-07-19T19:56:12.069611+00:00
paper_link
https://arxiv.org/abs/2310.02255
updated_at
2025-07-19T19:56:12.069611+00:00
description
MathVista evaluates mathematical reasoning of foundation models in visual contexts. It consists of 6,141 examples derived from 28 existing multimodal datasets and 3 newly created datasets (IQTest, FunctionQA, and PaperQA), combining challenges from diverse mathematical and visual tasks to assess models' ability to understand complex figures and perform rigorous reasoning.
benchmark_id
mathvista
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
MathVista · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-mini-2024-07-18/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/blog/gpt-4o-mini-announcement
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-mini-2024-07-18
metric_population
6
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-mini-2024-07-18
Original result ↗

LLM Stats · GitHub archive

0.402

source id
llmStatsArchive
leaderboard name
GPQA · Raw
dimension
general
metric
GPQA · Raw
rank
8
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
GPQA
language
en
modality
text
verified
false
max_score
1
categories
  • reasoning
  • general
created_at
2025-07-19T19:56:11.588605+00:00
paper_link
https://arxiv.org/abs/2311.12022
updated_at
2025-07-19T19:56:11.588605+00:00
description
A challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry. Questions are Google-proof and extremely difficult, with PhD experts reaching 65% accuracy.
benchmark_id
gpqa
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
GPQA · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-mini-2024-07-18/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/blog/gpt-4o-mini-announcement
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-mini-2024-07-18
metric_population
11
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-mini-2024-07-18
Original result ↗

LLM Stats · GitHub archive

0.402

source id
llmStatsArchive
leaderboard name
GPQA · Normalized (reported)
dimension
general
metric
GPQA · Normalized (reported)
rank
8
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
GPQA
language
en
modality
text
verified
false
max_score
1
categories
  • reasoning
  • general
created_at
2025-07-19T19:56:11.588605+00:00
paper_link
https://arxiv.org/abs/2311.12022
updated_at
2025-07-19T19:56:11.588605+00:00
description
A challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry. Questions are Google-proof and extremely difficult, with PhD experts reaching 65% accuracy.
benchmark_id
gpqa
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
GPQA · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-mini-2024-07-18/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/blog/gpt-4o-mini-announcement
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-mini-2024-07-18
metric_population
11
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-mini-2024-07-18
Original result ↗

Kaggle Benchmarks

4.4%

source id
kaggleBenchmarks
leaderboard name
MultiLoKo Urdu · EM Urdu
dimension
reasoning
metric
MultiLoKo Urdu · EM Urdu
rank
12
aggregation scope
kaggle
true
status
published
license
Apache-2.0
direction
higherIsBetter
conditions
task
id
58
name
EM Urdu
version
1
sort_order
DESCENDING
display_type
percentages
benchmark
id
81
name
MultiLoKo Urdu
type
INDIVIDUAL
owner
metaresearch
version
1
categories
  • multilingual and cross-lingual capabilities
  • general knowledge and reasoning
version_id
67
maintainer_tier
Kaggle-maintained
maintenance_level
KAGGLE_MAINTAINED
candidate
candidate_class
modelVersion
record_kind
aggregate
attribution
Kaggle · metaresearch · MultiLoKo Urdu
license_url
https://www.apache.org/licenses/LICENSE-2.0
metric_kind
MultiLoKo Urdu · EM Urdu
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
67/leaderboard.json
source_page
https://www.kaggle.com/benchmarks/metaresearch/multiloko-urdu/versions/1
measurements
  • name
    EM Urdu
    unit
    percent
    scale
    1
    metric
    MultiLoKo Urdu · EM Urdu
    result
    resultCase
    numericResult
    booleanResult
    false
    numericResult
    value
    0.044
    standardDeviation
    0
    confidenceInterval
    0.0254233848127016
    hasStandardDeviation
    false
    hasConfidenceInterval
    true
    confidenceIntervalNullable
    0.0254233848127016
    hasUnevenConfidenceInterval
    false
    taskVersionId
    0
    evaluationDate
    2025-06-16T00:00:00Z
    hasBooleanResult
    false
    hasNumericResult
    true
    hasTaskVersionId
    false
    hasEvaluationDate
    true
    numericResultNullable
    value
    0.044
    standardDeviation
    0
    confidenceInterval
    0.0254233848127016
    hasStandardDeviation
    false
    hasConfidenceInterval
    true
    confidenceIntervalNullable
    0.0254233848127016
    hasUnevenConfidenceInterval
    false
    hasNumericResultPublic
    false
    customAdditionalResults
      hasNumericResultPrivate
      false
      task_id
      58
      sort_order
      DESCENDING
      source_name
      EM Urdu
      display_type
      percentages
      task_version
      1
      task_display_type
      PERCENTAGES
      normalization_status
      normalized
    published_at
    2025-07-15T11:49:31.790Z
    candidate_class
    modelVersion
    source_revision
    b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
    publisher_anchor
    openai/gpt-4o-mini-2024-07-18
    metric_population
    51
    benchmark_evaluation
    true
    task_metadata_source
    benchmarkVersion
    candidate_metadata_source
    benchmarkVersionMappings
    evaluated at
    2025-06-16T00:00:00.000Z
    source model id
    GPT-4o mini
    confidence lower
    0.018576615187298397
    confidence upper
    0.0694233848127016
    Original result ↗

    Kaggle Benchmarks

    13.6%

    source id
    kaggleBenchmarks
    leaderboard name
    MultiLoKo Malay · EM Malay
    dimension
    reasoning
    metric
    MultiLoKo Malay · EM Malay
    rank
    23
    aggregation scope
    kaggle
    true
    status
    published
    license
    Apache-2.0
    direction
    higherIsBetter
    conditions
    task
    id
    45
    name
    EM Malay
    version
    1
    sort_order
    DESCENDING
    display_type
    percentages
    benchmark
    id
    68
    name
    MultiLoKo Malay
    type
    INDIVIDUAL
    owner
    metaresearch
    version
    1
    categories
    • multilingual and cross-lingual capabilities
    • general knowledge and reasoning
    version_id
    54
    maintainer_tier
    Kaggle-maintained
    maintenance_level
    KAGGLE_MAINTAINED
    candidate
    candidate_class
    modelVersion
    record_kind
    aggregate
    attribution
    Kaggle · metaresearch · MultiLoKo Malay
    license_url
    https://www.apache.org/licenses/LICENSE-2.0
    metric_kind
    MultiLoKo Malay · EM Malay
    rank_method
    competition rank among reported evaluations in identical conditions, including unlinked observations
    source_file
    54/leaderboard.json
    source_page
    https://www.kaggle.com/benchmarks/metaresearch/multiloko-malay/versions/1
    measurements
    • name
      EM Malay
      unit
      percent
      scale
      1
      metric
      MultiLoKo Malay · EM Malay
      result
      resultCase
      numericResult
      booleanResult
      false
      numericResult
      value
      0.136
      standardDeviation
      0
      confidenceInterval
      0.0424917298902665
      hasStandardDeviation
      false
      hasConfidenceInterval
      true
      confidenceIntervalNullable
      0.0424917298902665
      hasUnevenConfidenceInterval
      false
      taskVersionId
      0
      evaluationDate
      2025-06-16T00:00:00Z
      hasBooleanResult
      false
      hasNumericResult
      true
      hasTaskVersionId
      false
      hasEvaluationDate
      true
      numericResultNullable
      value
      0.136
      standardDeviation
      0
      confidenceInterval
      0.0424917298902665
      hasStandardDeviation
      false
      hasConfidenceInterval
      true
      confidenceIntervalNullable
      0.0424917298902665
      hasUnevenConfidenceInterval
      false
      hasNumericResultPublic
      false
      customAdditionalResults
        hasNumericResultPrivate
        false
        task_id
        45
        sort_order
        DESCENDING
        source_name
        EM Malay
        display_type
        percentages
        task_version
        1
        task_display_type
        PERCENTAGES
        normalization_status
        normalized
      published_at
      2025-07-15T11:49:31.790Z
      candidate_class
      modelVersion
      source_revision
      b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
      publisher_anchor
      openai/gpt-4o-mini-2024-07-18
      metric_population
      51
      benchmark_evaluation
      true
      task_metadata_source
      benchmarkVersion
      candidate_metadata_source
      benchmarkVersionMappings
      evaluated at
      2025-06-16T00:00:00.000Z
      source model id
      GPT-4o mini
      confidence lower
      0.0935082701097335
      confidence upper
      0.17849172989026652
      Original result ↗

      Published prices

      1–3 / 3

      Provider specifications & access

      1–1 / 1

      OpenAI GPT-4o mini · azure-openai v2024-07-18 ↗

      source model id
      azure-openai/gpt-4o-mini@2024-07-18
      hosting
      foundry-managed-compute
      platforms
      • Microsoft Foundry
      license
      custom
      context length
      128000
      max input tokens
      128000
      max output tokens
      16384
      architecture
      input_modalities
      • text
      • image
      • audio
      output_modalities
      • text
      capabilities
      • toolCalling

      Safety assessments

      1–1 / 1

      Foundation Model Transparency Index 2025 ↗

      35 / 100

      source id
      stanfordFMTI
      canonical model id
      openai/gpt-4o-mini-2024-07-18
      taxonomy
      transparency
      hazard category
      transparency
      hazard name
      Organization transparency
      policy area
      OpenAI
      assessment type
      organization-level disclosure
      metric
      disclosure_score
      result label
      35 / 100
      sample size
      100
      updated at
      2026-11-09T18:08:05.227Z

      Source specification observations

      1–7 / 7

      azureFoundry ↗

      field
      context_length
      value
      128000
      normalized value
      128000
      scope
      source-model
      source id
      azureFoundry
      confidence
      1

      openRouter ↗

      field
      context_length
      value
      128000
      normalized value
      128000
      scope
      source-model
      source id
      openRouter
      confidence
      1

      openRouter ↗

      field
      knowledge_cutoff
      reported field
      knowledge_cutoff
      value
      2023-10-31
      normalized value
      2023-10-31
      precision
      day
      scope
      source-model
      source id
      openRouter
      confidence
      1

      azureFoundry ↗

      field
      max_input_tokens
      value
      128000
      normalized value
      128000
      scope
      source-model
      source id
      azureFoundry
      confidence
      1

      azureFoundry ↗

      field
      max_output_tokens
      value
      16384
      normalized value
      16384
      scope
      source-model
      source id
      azureFoundry
      confidence
      1

      openRouter ↗

      field
      max_output_tokens
      value
      16384
      normalized value
      16384
      scope
      source-model
      source id
      openRouter
      confidence
      1

      azureFoundry ↗

      field
      knowledge_cutoff
      reported field
      training_data_cutoff
      value
      September 2023
      normalized value
      2023-09
      precision
      month
      scope
      source-model
      source id
      azureFoundry
      confidence
      1

      Original sources

      1–11 / 11

      Microsoft Foundry · Model page ↗

      OpenRouter · Model page ↗

      Microsoft Foundry · Model documentation ↗

      GitHub · Catalog ↗

      GitHub · Catalog ↗

      Kaggle · Catalog ↗

      LMArena · Catalog ↗

      Microsoft Foundry · Catalog ↗

      OpenRouter · Catalog ↗

      crfm.stanford.edu · Catalog ↗

      livebench.ai · Catalog ↗

      Source links

      1–8 / 8

      Explore more models

      Models

      Loading
      EVIDENCE / COMPARISON

      Benchmarks

      Explore every benchmark

      Loading

      These charts contain model-linked results. The charts above also include unlinked source profiles. Compare scores within each evaluation. Model quality, usage, transparency and system performance measure different things.

      Install TensorTrace