OpenAI: GPT-4o-mini (2024-07-18)
GPT-4o mini is OpenAI's newest model after [GPT-4 Omni](/models/openai/gpt-4o), supporting both text and image inputs with text outputs. As their most advanced small model, it is many multiples more …
openai/gpt-4o-mini-2024-07-18
GPT-4o mini is OpenAI's newest model after [GPT-4 Omni](/models/openai/gpt-4o), supporting both text and image inputs with text outputs. As their most advanced small model, it is many multiples more affordable...
Specifications & capabilities
- Input
- text, image, file
- Output
- text
- Context window · tokens
- 128000
- Capabilities
- audioInput, fileInput, multimodal, promptCaching, structuredOutputs, toolCalling, visionInput
Evaluations
1–20 / 186
0.087
- source id
- llmStatsArchive
- leaderboard name
- SWE-Bench Verified · Normalized (reported)
- dimension
- general
- metric
- SWE-Bench Verified · Normalized (reported)
- rank
- 1
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- SWE-Bench Verified
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- categories
- reasoning
- frontend_development
- code
- created_at
- 2025-07-19T19:56:13.812805+00:00
- paper_link
- https://arxiv.org/abs/2310.06770
- updated_at
- 2025-07-19T19:56:13.812805+00:00
- description
- A verified subset of 500 software engineering problems from real GitHub issues, validated by human annotators for evaluating language models' ability to resolve real-world coding issues by generating patches for Python codebases.
- benchmark_id
- swe-bench-verified
- multilingual
- false
- analysis_method
- Pass Rate
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- SWE-Bench Verified · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-mini-2024-07-18/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/gpt-4-1/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-mini-2024-07-18
- metric_population
- 1
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-mini-2024-07-18
Original result ↗0.087
- source id
- llmStatsArchive
- leaderboard name
- SWE-Bench Verified · Raw
- dimension
- general
- metric
- SWE-Bench Verified · Raw
- rank
- 1
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- SWE-Bench Verified
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- categories
- reasoning
- frontend_development
- code
- created_at
- 2025-07-19T19:56:13.812805+00:00
- paper_link
- https://arxiv.org/abs/2310.06770
- updated_at
- 2025-07-19T19:56:13.812805+00:00
- description
- A verified subset of 500 software engineering problems from real GitHub issues, validated by human annotators for evaluating language models' ability to resolve real-world coding issues by generating patches for Python codebases.
- benchmark_id
- swe-bench-verified
- multilingual
- false
- analysis_method
- Pass Rate
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- SWE-Bench Verified · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-mini-2024-07-18/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/gpt-4-1/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-mini-2024-07-18
- metric_population
- 1
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-mini-2024-07-18
Original result ↗0.87
- source id
- llmStatsArchive
- leaderboard name
- MGSM · Raw
- dimension
- general
- metric
- MGSM · Raw
- rank
- 2
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- MGSM
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:13.669061+00:00
- paper_link
- https://arxiv.org/abs/2210.03057
- updated_at
- 2025-07-19T19:56:13.669061+00:00
- description
- MGSM (Multilingual Grade School Math) is a benchmark of grade-school math problems. Contains 250 grade-school math problems manually translated from the GSM8K dataset into ten typologically diverse languages: Spanish, French, German, Russian, Chinese, Japanese, Thai, Swahili, Bengali, and Telugu. Evaluates multilingual mathematical reasoning capabilities.
- benchmark_id
- mgsm
- multilingual
- true
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- MGSM · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-mini-2024-07-18/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/blog/gpt-4o-mini-announcement
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-mini-2024-07-18
- metric_population
- 3
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-mini-2024-07-18
Original result ↗0.87
- source id
- llmStatsArchive
- leaderboard name
- MGSM · Normalized (reported)
- dimension
- general
- metric
- MGSM · Normalized (reported)
- rank
- 2
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- MGSM
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:13.669061+00:00
- paper_link
- https://arxiv.org/abs/2210.03057
- updated_at
- 2025-07-19T19:56:13.669061+00:00
- description
- MGSM (Multilingual Grade School Math) is a benchmark of grade-school math problems. Contains 250 grade-school math problems manually translated from the GSM8K dataset into ten typologically diverse languages: Spanish, French, German, Russian, Chinese, Japanese, Thai, Swahili, Bengali, and Telugu. Evaluates multilingual mathematical reasoning capabilities.
- benchmark_id
- mgsm
- multilingual
- true
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- MGSM · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-mini-2024-07-18/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/blog/gpt-4o-mini-announcement
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-mini-2024-07-18
- metric_population
- 3
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-mini-2024-07-18
Original result ↗0.797
- source id
- llmStatsArchive
- leaderboard name
- DROP · Normalized (reported)
- dimension
- general
- metric
- DROP · Normalized (reported)
- rank
- 2
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- DROP
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:12.981569+00:00
- paper_link
- https://arxiv.org/abs/1903.00161
- updated_at
- 2025-07-19T19:56:12.981569+00:00
- description
- DROP (Discrete Reasoning Over Paragraphs) is a reading comprehension benchmark requiring discrete reasoning over paragraph content. It contains crowdsourced, adversarially-created questions that require resolving references and performing discrete operations like addition, counting, or sorting, demanding comprehensive paragraph understanding beyond paraphrase-and-entity-typing shortcuts.
- benchmark_id
- drop
- multilingual
- false
- analysis_method
- F1 Score
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- DROP · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-mini-2024-07-18/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/blog/gpt-4o-mini-announcement
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-mini-2024-07-18
- metric_population
- 2
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-mini-2024-07-18
Original result ↗0.797
- source id
- llmStatsArchive
- leaderboard name
- DROP · Raw
- dimension
- general
- metric
- DROP · Raw
- rank
- 2
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- DROP
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:12.981569+00:00
- paper_link
- https://arxiv.org/abs/1903.00161
- updated_at
- 2025-07-19T19:56:12.981569+00:00
- description
- DROP (Discrete Reasoning Over Paragraphs) is a reading comprehension benchmark requiring discrete reasoning over paragraph content. It contains crowdsourced, adversarially-created questions that require resolving references and performing discrete operations like addition, counting, or sorting, demanding comprehensive paragraph understanding beyond paraphrase-and-entity-typing shortcuts.
- benchmark_id
- drop
- multilingual
- false
- analysis_method
- F1 Score
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- DROP · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-mini-2024-07-18/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/blog/gpt-4o-mini-announcement
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-mini-2024-07-18
- metric_population
- 2
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-mini-2024-07-18
Original result ↗0.594
- source id
- llmStatsArchive
- leaderboard name
- MMMU · Raw
- dimension
- general
- metric
- MMMU · Raw
- rank
- 4
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- MMMU
- language
- en
- modality
- multimodal
- verified
- false
- max_score
- 1
- categories
- multimodal
- reasoning
- general
- created_at
- 2025-07-19T19:56:12.130105+00:00
- paper_link
- https://arxiv.org/abs/2311.16502
- updated_at
- 2025-07-19T19:56:12.130105+00:00
- description
- MMMU (Massive Multi-discipline Multimodal Understanding) is a benchmark designed to evaluate multimodal models on college-level subject knowledge and deliberate reasoning. Contains 11.5K meticulously collected multimodal questions from college exams, quizzes, and textbooks, covering six core disciplines: Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering across 30 subjects and 183 subfields.
- benchmark_id
- mmmu
- multilingual
- false
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- MMMU · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-mini-2024-07-18/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/blog/gpt-4o-mini-announcement
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-mini-2024-07-18
- metric_population
- 4
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-mini-2024-07-18
Original result ↗0.594
- source id
- llmStatsArchive
- leaderboard name
- MMMU · Normalized (reported)
- dimension
- general
- metric
- MMMU · Normalized (reported)
- rank
- 4
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- MMMU
- language
- en
- modality
- multimodal
- verified
- false
- max_score
- 1
- categories
- multimodal
- reasoning
- general
- created_at
- 2025-07-19T19:56:12.130105+00:00
- paper_link
- https://arxiv.org/abs/2311.16502
- updated_at
- 2025-07-19T19:56:12.130105+00:00
- description
- MMMU (Massive Multi-discipline Multimodal Understanding) is a benchmark designed to evaluate multimodal models on college-level subject knowledge and deliberate reasoning. Contains 11.5K meticulously collected multimodal questions from college exams, quizzes, and textbooks, covering six core disciplines: Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering across 30 subjects and 183 subfields.
- benchmark_id
- mmmu
- multilingual
- false
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- MMMU · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-mini-2024-07-18/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/blog/gpt-4o-mini-announcement
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-mini-2024-07-18
- metric_population
- 4
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-mini-2024-07-18
Original result ↗0.702
- source id
- llmStatsArchive
- leaderboard name
- MATH · Raw
- dimension
- general
- metric
- MATH · Raw
- rank
- 5
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- MATH
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:11.804258+00:00
- paper_link
- https://arxiv.org/abs/2103.03874
- updated_at
- 2025-07-19T19:56:11.804258+00:00
- description
- MATH dataset contains 12,500 challenging competition mathematics problems from AMC 10, AMC 12, AIME, and other mathematics competitions. Each problem includes full step-by-step solutions and spans multiple difficulty levels (1-5) across seven mathematical subjects including Prealgebra, Algebra, Number Theory, Counting and Probability, Geometry, Intermediate Algebra, and Precalculus.
- benchmark_id
- math
- multilingual
- false
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- MATH · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-mini-2024-07-18/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/blog/gpt-4o-mini-announcement
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-mini-2024-07-18
- metric_population
- 7
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-mini-2024-07-18
Original result ↗0.702
- source id
- llmStatsArchive
- leaderboard name
- MATH · Normalized (reported)
- dimension
- general
- metric
- MATH · Normalized (reported)
- rank
- 5
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- MATH
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:11.804258+00:00
- paper_link
- https://arxiv.org/abs/2103.03874
- updated_at
- 2025-07-19T19:56:11.804258+00:00
- description
- MATH dataset contains 12,500 challenging competition mathematics problems from AMC 10, AMC 12, AIME, and other mathematics competitions. Each problem includes full step-by-step solutions and spans multiple difficulty levels (1-5) across seven mathematical subjects including Prealgebra, Algebra, Number Theory, Counting and Probability, Geometry, Intermediate Algebra, and Precalculus.
- benchmark_id
- math
- multilingual
- false
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- MATH · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-mini-2024-07-18/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/blog/gpt-4o-mini-announcement
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-mini-2024-07-18
- metric_population
- 7
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-mini-2024-07-18
Original result ↗0.872
- source id
- llmStatsArchive
- leaderboard name
- HumanEval · Raw
- dimension
- general
- metric
- HumanEval · Raw
- rank
- 6
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- HumanEval
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:12.595263+00:00
- paper_link
- https://arxiv.org/abs/2107.03374
- updated_at
- 2025-07-19T19:56:12.595263+00:00
- description
- A benchmark that measures functional correctness for synthesizing programs from docstrings, consisting of 164 original programming problems assessing language comprehension, algorithms, and simple mathematics
- benchmark_id
- humaneval
- multilingual
- false
- analysis_method
- Pass@1
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- HumanEval · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-mini-2024-07-18/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/blog/gpt-4o-mini-announcement
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-mini-2024-07-18
- metric_population
- 9
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-mini-2024-07-18
Original result ↗0.872
- source id
- llmStatsArchive
- leaderboard name
- HumanEval · Normalized (reported)
- dimension
- general
- metric
- HumanEval · Normalized (reported)
- rank
- 6
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- HumanEval
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:12.595263+00:00
- paper_link
- https://arxiv.org/abs/2107.03374
- updated_at
- 2025-07-19T19:56:12.595263+00:00
- description
- A benchmark that measures functional correctness for synthesizing programs from docstrings, consisting of 164 original programming problems assessing language comprehension, algorithms, and simple mathematics
- benchmark_id
- humaneval
- multilingual
- false
- analysis_method
- Pass@1
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- HumanEval · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-mini-2024-07-18/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/blog/gpt-4o-mini-announcement
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-mini-2024-07-18
- metric_population
- 9
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-mini-2024-07-18
Original result ↗0.82
- source id
- llmStatsArchive
- leaderboard name
- MMLU · Raw
- dimension
- general
- metric
- MMLU · Raw
- rank
- 6
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- MMLU
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- categories
- general
- reasoning
- language
- math
- created_at
- 2025-07-19T19:56:11.200416+00:00
- paper_link
- https://arxiv.org/abs/2009.03300
- updated_at
- 2025-07-19T19:56:11.200416+00:00
- description
- Massive Multitask Language Understanding benchmark testing knowledge across 57 diverse subjects including STEM, humanities, social sciences, and professional domains
- benchmark_id
- mmlu
- multilingual
- false
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- MMLU · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-mini-2024-07-18/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/blog/gpt-4o-mini-announcement
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-mini-2024-07-18
- metric_population
- 8
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-mini-2024-07-18
Original result ↗0.82
- source id
- llmStatsArchive
- leaderboard name
- MMLU · Normalized (reported)
- dimension
- general
- metric
- MMLU · Normalized (reported)
- rank
- 6
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- MMLU
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- categories
- general
- reasoning
- language
- math
- created_at
- 2025-07-19T19:56:11.200416+00:00
- paper_link
- https://arxiv.org/abs/2009.03300
- updated_at
- 2025-07-19T19:56:11.200416+00:00
- description
- Massive Multitask Language Understanding benchmark testing knowledge across 57 diverse subjects including STEM, humanities, social sciences, and professional domains
- benchmark_id
- mmlu
- multilingual
- false
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- MMLU · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-mini-2024-07-18/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/blog/gpt-4o-mini-announcement
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-mini-2024-07-18
- metric_population
- 8
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-mini-2024-07-18
Original result ↗0.567
- source id
- llmStatsArchive
- leaderboard name
- MathVista · Normalized (reported)
- dimension
- general
- metric
- MathVista · Normalized (reported)
- rank
- 6
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- MathVista
- language
- en
- modality
- multimodal
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:12.069611+00:00
- paper_link
- https://arxiv.org/abs/2310.02255
- updated_at
- 2025-07-19T19:56:12.069611+00:00
- description
- MathVista evaluates mathematical reasoning of foundation models in visual contexts. It consists of 6,141 examples derived from 28 existing multimodal datasets and 3 newly created datasets (IQTest, FunctionQA, and PaperQA), combining challenges from diverse mathematical and visual tasks to assess models' ability to understand complex figures and perform rigorous reasoning.
- benchmark_id
- mathvista
- multilingual
- false
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- MathVista · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-mini-2024-07-18/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/blog/gpt-4o-mini-announcement
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-mini-2024-07-18
- metric_population
- 6
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-mini-2024-07-18
Original result ↗0.567
- source id
- llmStatsArchive
- leaderboard name
- MathVista · Raw
- dimension
- general
- metric
- MathVista · Raw
- rank
- 6
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- MathVista
- language
- en
- modality
- multimodal
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:12.069611+00:00
- paper_link
- https://arxiv.org/abs/2310.02255
- updated_at
- 2025-07-19T19:56:12.069611+00:00
- description
- MathVista evaluates mathematical reasoning of foundation models in visual contexts. It consists of 6,141 examples derived from 28 existing multimodal datasets and 3 newly created datasets (IQTest, FunctionQA, and PaperQA), combining challenges from diverse mathematical and visual tasks to assess models' ability to understand complex figures and perform rigorous reasoning.
- benchmark_id
- mathvista
- multilingual
- false
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- MathVista · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-mini-2024-07-18/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/blog/gpt-4o-mini-announcement
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-mini-2024-07-18
- metric_population
- 6
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-mini-2024-07-18
Original result ↗0.402
- source id
- llmStatsArchive
- leaderboard name
- GPQA · Raw
- dimension
- general
- metric
- GPQA · Raw
- rank
- 8
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- GPQA
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:11.588605+00:00
- paper_link
- https://arxiv.org/abs/2311.12022
- updated_at
- 2025-07-19T19:56:11.588605+00:00
- description
- A challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry. Questions are Google-proof and extremely difficult, with PhD experts reaching 65% accuracy.
- benchmark_id
- gpqa
- multilingual
- false
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- GPQA · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-mini-2024-07-18/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/blog/gpt-4o-mini-announcement
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-mini-2024-07-18
- metric_population
- 11
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-mini-2024-07-18
Original result ↗0.402
- source id
- llmStatsArchive
- leaderboard name
- GPQA · Normalized (reported)
- dimension
- general
- metric
- GPQA · Normalized (reported)
- rank
- 8
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- GPQA
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:11.588605+00:00
- paper_link
- https://arxiv.org/abs/2311.12022
- updated_at
- 2025-07-19T19:56:11.588605+00:00
- description
- A challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry. Questions are Google-proof and extremely difficult, with PhD experts reaching 65% accuracy.
- benchmark_id
- gpqa
- multilingual
- false
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- GPQA · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-mini-2024-07-18/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/blog/gpt-4o-mini-announcement
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-mini-2024-07-18
- metric_population
- 11
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-mini-2024-07-18
Original result ↗4.4%
- source id
- kaggleBenchmarks
- leaderboard name
- MultiLoKo Urdu · EM Urdu
- dimension
- reasoning
- metric
- MultiLoKo Urdu · EM Urdu
- rank
- 12
- aggregation scope
- kaggle
- true
- status
- published
- license
- Apache-2.0
- direction
- higherIsBetter
- conditions
- task
- id
- 58
- name
- EM Urdu
- version
- 1
- sort_order
- DESCENDING
- display_type
- percentages
- benchmark
- id
- 81
- name
- MultiLoKo Urdu
- type
- INDIVIDUAL
- owner
- metaresearch
- version
- 1
- categories
- multilingual and cross-lingual capabilities
- general knowledge and reasoning
- version_id
- 67
- maintainer_tier
- Kaggle-maintained
- maintenance_level
- KAGGLE_MAINTAINED
- candidate
- candidate_class
- modelVersion
- record_kind
- aggregate
- attribution
- Kaggle · metaresearch · MultiLoKo Urdu
- license_url
- https://www.apache.org/licenses/LICENSE-2.0
- metric_kind
- MultiLoKo Urdu · EM Urdu
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- 67/leaderboard.json
- source_page
- https://www.kaggle.com/benchmarks/metaresearch/multiloko-urdu/versions/1
- measurements
- name
- EM Urdu
- unit
- percent
- scale
- 1
- metric
- MultiLoKo Urdu · EM Urdu
- result
- resultCase
- numericResult
- booleanResult
- false
- numericResult
- value
- 0.044
- standardDeviation
- 0
- confidenceInterval
- 0.0254233848127016
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0254233848127016
- hasUnevenConfidenceInterval
- false
- taskVersionId
- 0
- evaluationDate
- 2025-06-16T00:00:00Z
- hasBooleanResult
- false
- hasNumericResult
- true
- hasTaskVersionId
- false
- hasEvaluationDate
- true
- numericResultNullable
- value
- 0.044
- standardDeviation
- 0
- confidenceInterval
- 0.0254233848127016
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0254233848127016
- hasUnevenConfidenceInterval
- false
- hasNumericResultPublic
- false
- hasNumericResultPrivate
- false
- task_id
- 58
- sort_order
- DESCENDING
- source_name
- EM Urdu
- display_type
- percentages
- task_version
- 1
- task_display_type
- PERCENTAGES
- normalization_status
- normalized
- published_at
- 2025-07-15T11:49:31.790Z
- candidate_class
- modelVersion
- source_revision
- b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
- publisher_anchor
- openai/gpt-4o-mini-2024-07-18
- metric_population
- 51
- benchmark_evaluation
- true
- task_metadata_source
- benchmarkVersion
- candidate_metadata_source
- benchmarkVersionMappings
- evaluated at
- 2025-06-16T00:00:00.000Z
- source model id
- GPT-4o mini
- confidence lower
- 0.018576615187298397
- confidence upper
- 0.0694233848127016
Original result ↗13.6%
- source id
- kaggleBenchmarks
- leaderboard name
- MultiLoKo Malay · EM Malay
- dimension
- reasoning
- metric
- MultiLoKo Malay · EM Malay
- rank
- 23
- aggregation scope
- kaggle
- true
- status
- published
- license
- Apache-2.0
- direction
- higherIsBetter
- conditions
- task
- id
- 45
- name
- EM Malay
- version
- 1
- sort_order
- DESCENDING
- display_type
- percentages
- benchmark
- id
- 68
- name
- MultiLoKo Malay
- type
- INDIVIDUAL
- owner
- metaresearch
- version
- 1
- categories
- multilingual and cross-lingual capabilities
- general knowledge and reasoning
- version_id
- 54
- maintainer_tier
- Kaggle-maintained
- maintenance_level
- KAGGLE_MAINTAINED
- candidate
- candidate_class
- modelVersion
- record_kind
- aggregate
- attribution
- Kaggle · metaresearch · MultiLoKo Malay
- license_url
- https://www.apache.org/licenses/LICENSE-2.0
- metric_kind
- MultiLoKo Malay · EM Malay
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- 54/leaderboard.json
- source_page
- https://www.kaggle.com/benchmarks/metaresearch/multiloko-malay/versions/1
- measurements
- name
- EM Malay
- unit
- percent
- scale
- 1
- metric
- MultiLoKo Malay · EM Malay
- result
- resultCase
- numericResult
- booleanResult
- false
- numericResult
- value
- 0.136
- standardDeviation
- 0
- confidenceInterval
- 0.0424917298902665
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0424917298902665
- hasUnevenConfidenceInterval
- false
- taskVersionId
- 0
- evaluationDate
- 2025-06-16T00:00:00Z
- hasBooleanResult
- false
- hasNumericResult
- true
- hasTaskVersionId
- false
- hasEvaluationDate
- true
- numericResultNullable
- value
- 0.136
- standardDeviation
- 0
- confidenceInterval
- 0.0424917298902665
- hasStandardDeviation
- false
- hasConfidenceInterval
- true
- confidenceIntervalNullable
- 0.0424917298902665
- hasUnevenConfidenceInterval
- false
- hasNumericResultPublic
- false
- hasNumericResultPrivate
- false
- task_id
- 45
- sort_order
- DESCENDING
- source_name
- EM Malay
- display_type
- percentages
- task_version
- 1
- task_display_type
- PERCENTAGES
- normalization_status
- normalized
- published_at
- 2025-07-15T11:49:31.790Z
- candidate_class
- modelVersion
- source_revision
- b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
- publisher_anchor
- openai/gpt-4o-mini-2024-07-18
- metric_population
- 51
- benchmark_evaluation
- true
- task_metadata_source
- benchmarkVersion
- candidate_metadata_source
- benchmarkVersionMappings
- evaluated at
- 2025-06-16T00:00:00.000Z
- source model id
- GPT-4o mini
- confidence lower
- 0.0935082701097335
- confidence upper
- 0.17849172989026652
Original result ↗Published prices
1–3 / 3
$0.08
- Inference provider
- openRouter
- Platform
- OpenRouter
- Unit
- cache_read_tokens_per_million
- Currency
- USD
$0.15
- Inference provider
- openRouter
- Platform
- OpenRouter
- Unit
- input_tokens_per_million
- Currency
- USD
$0.60
- Inference provider
- openRouter
- Platform
- OpenRouter
- Unit
- output_tokens_per_million
- Currency
- USD
Provider specifications & access
1–1 / 1
- source model id
- azure-openai/gpt-4o-mini@2024-07-18
- hosting
- foundry-managed-compute
- license
- custom
- context length
- 128000
- max input tokens
- 128000
- max output tokens
- 16384
Safety assessments
1–1 / 1
35 / 100
- source id
- stanfordFMTI
- canonical model id
- openai/gpt-4o-mini-2024-07-18
- taxonomy
- transparency
- hazard category
- transparency
- hazard name
- Organization transparency
- policy area
- OpenAI
- assessment type
- organization-level disclosure
- metric
- disclosure_score
- result label
- 35 / 100
- sample size
- 100
- updated at
- 2026-11-09T18:08:05.227Z
Source specification observations
1–7 / 7
- field
- context_length
- value
- 128000
- normalized value
- 128000
- scope
- source-model
- source id
- azureFoundry
- confidence
- 1
- field
- context_length
- value
- 128000
- normalized value
- 128000
- scope
- source-model
- source id
- openRouter
- confidence
- 1
- field
- knowledge_cutoff
- reported field
- knowledge_cutoff
- value
- 2023-10-31
- normalized value
- 2023-10-31
- precision
- day
- scope
- source-model
- source id
- openRouter
- confidence
- 1
- field
- max_input_tokens
- value
- 128000
- normalized value
- 128000
- scope
- source-model
- source id
- azureFoundry
- confidence
- 1
- field
- max_output_tokens
- value
- 16384
- normalized value
- 16384
- scope
- source-model
- source id
- azureFoundry
- confidence
- 1
- field
- max_output_tokens
- value
- 16384
- normalized value
- 16384
- scope
- source-model
- source id
- openRouter
- confidence
- 1
- field
- knowledge_cutoff
- reported field
- training_data_cutoff
- value
- September 2023
- normalized value
- 2023-09
- precision
- month
- scope
- source-model
- source id
- azureFoundry
- confidence
- 1
Explore more models