OpenAI: GPT-4o (2024-08-06)
The 2024-08-06 version of GPT-4o offers improved performance in structured outputs, with the ability to supply a JSON schema in the respone_format. Read more [here](https://openai.com/index/introduci…
openai/gpt-4o-2024-08-06
The 2024-08-06 version of GPT-4o offers improved performance in structured outputs, with the ability to supply a JSON schema in the respone_format. Read more [here](https://openai.com/index/introducing-structured-outputs-in-the-api/). GPT-4o ("o" for "omni") is...
Specifications & capabilities
- Input
- text, image, file
- Output
- text
- Context window · tokens
- 128000
- Knowledge cutoff
- 2023-10-31
- Capabilities
- fileInput, multimodal, promptCaching, structuredOutputs, toolCalling, visionInput
Evaluations
1–20 / 143
0.665
- source id
- llmStatsArchive
- leaderboard name
- ComplexFuncBench · Normalized (reported)
- dimension
- general
- metric
- ComplexFuncBench · Normalized (reported)
- rank
- 1
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- ComplexFuncBench
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:15.336577+00:00
- paper_link
- https://arxiv.org/abs/2501.10132
- updated_at
- 2025-07-19T19:56:15.336577+00:00
- description
- ComplexFuncBench is a benchmark designed to evaluate large language models' capabilities in handling complex function calling scenarios. It encompasses multi-step and constrained function calling tasks that require long-parameter filling, parameter value reasoning, and managing contexts up to 128k tokens. The benchmark includes 1,000 samples across five real-world scenarios.
- benchmark_id
- complexfuncbench
- multilingual
- false
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- ComplexFuncBench · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/gpt-4-1/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 2
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-2024-08-06
Original result ↗0.665
- source id
- llmStatsArchive
- leaderboard name
- ComplexFuncBench · Raw
- dimension
- general
- metric
- ComplexFuncBench · Raw
- rank
- 1
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- ComplexFuncBench
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:15.336577+00:00
- paper_link
- https://arxiv.org/abs/2501.10132
- updated_at
- 2025-07-19T19:56:15.336577+00:00
- description
- ComplexFuncBench is a benchmark designed to evaluate large language models' capabilities in handling complex function calling scenarios. It encompasses multi-step and constrained function calling tasks that require long-parameter filling, parameter value reasoning, and managing contexts up to 128k tokens. The benchmark includes 1,000 samples across five real-world scenarios.
- benchmark_id
- complexfuncbench
- multilingual
- false
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- ComplexFuncBench · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/gpt-4-1/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 2
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-2024-08-06
Original result ↗0.853
- source id
- llmStatsArchive
- leaderboard name
- CharXiv-D · Normalized (reported)
- dimension
- general
- metric
- CharXiv-D · Normalized (reported)
- rank
- 2
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- CharXiv-D
- language
- en
- modality
- multimodal
- verified
- false
- max_score
- 1
- categories
- reasoning
- vision
- multimodal
- created_at
- 2025-07-19T19:56:15.325204+00:00
- paper_link
- https://arxiv.org/abs/2406.18521
- updated_at
- 2025-07-19T19:56:15.325204+00:00
- description
- CharXiv-D is the descriptive questions subset of the CharXiv benchmark, designed to assess multimodal large language models' ability to extract basic information from scientific charts. It contains descriptive questions covering information extraction, enumeration, pattern recognition, and counting across 2,323 diverse charts from arXiv papers, all curated and verified by human experts.
- benchmark_id
- charxiv-d
- multilingual
- false
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- CharXiv-D · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/gpt-4-1/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 2
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-2024-08-06
Original result ↗0.853
- source id
- llmStatsArchive
- leaderboard name
- CharXiv-D · Raw
- dimension
- general
- metric
- CharXiv-D · Raw
- rank
- 2
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- CharXiv-D
- language
- en
- modality
- multimodal
- verified
- false
- max_score
- 1
- categories
- reasoning
- vision
- multimodal
- created_at
- 2025-07-19T19:56:15.325204+00:00
- paper_link
- https://arxiv.org/abs/2406.18521
- updated_at
- 2025-07-19T19:56:15.325204+00:00
- description
- CharXiv-D is the descriptive questions subset of the CharXiv benchmark, designed to assess multimodal large language models' ability to extract basic information from scientific charts. It contains descriptive questions covering information extraction, enumeration, pattern recognition, and counting across 2,323 diverse charts from arXiv papers, all curated and verified by human experts.
- benchmark_id
- charxiv-d
- multilingual
- false
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- CharXiv-D · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/gpt-4-1/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 2
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-2024-08-06
Original result ↗0.603
- source id
- llmStatsArchive
- leaderboard name
- TAU-bench Retail · Raw
- dimension
- general
- metric
- TAU-bench Retail · Raw
- rank
- 2
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- TAU-bench Retail
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:14.965635+00:00
- paper_link
- https://arxiv.org/abs/2406.12045
- updated_at
- 2025-07-19T19:56:14.965635+00:00
- description
- A benchmark for evaluating tool-agent-user interaction in retail environments. Tests language agents' ability to handle dynamic conversations with users while using domain-specific API tools and following policy guidelines. Evaluates agents on tasks like order cancellations, address changes, and order status checks through multi-turn conversations.
- benchmark_id
- tau-bench-retail
- multilingual
- false
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- TAU-bench Retail · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/gpt-4-1/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 2
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-2024-08-06
Original result ↗0.603
- source id
- llmStatsArchive
- leaderboard name
- TAU-bench Retail · Normalized (reported)
- dimension
- general
- metric
- TAU-bench Retail · Normalized (reported)
- rank
- 2
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- TAU-bench Retail
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:14.965635+00:00
- paper_link
- https://arxiv.org/abs/2406.12045
- updated_at
- 2025-07-19T19:56:14.965635+00:00
- description
- A benchmark for evaluating tool-agent-user interaction in retail environments. Tests language agents' ability to handle dynamic conversations with users while using domain-specific API tools and following policy guidelines. Evaluates agents on tasks like order cancellations, address changes, and order status checks through multi-turn conversations.
- benchmark_id
- tau-bench-retail
- multilingual
- false
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- TAU-bench Retail · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/gpt-4-1/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 2
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-2024-08-06
Original result ↗0.428
- source id
- llmStatsArchive
- leaderboard name
- TAU-bench Airline · Normalized (reported)
- dimension
- general
- metric
- TAU-bench Airline · Normalized (reported)
- rank
- 2
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- TAU-bench Airline
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:14.993213+00:00
- paper_link
- https://arxiv.org/abs/2406.12045
- updated_at
- 2025-07-19T19:56:14.993213+00:00
- description
- Part of τ-bench (TAU-bench), a benchmark for Tool-Agent-User interaction in real-world domains. The airline domain evaluates language agents' ability to interact with users through dynamic conversations while following domain-specific rules and using API tools. Agents must handle airline-related tasks and policies reliably.
- benchmark_id
- tau-bench-airline
- multilingual
- false
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- TAU-bench Airline · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/gpt-4-1/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 2
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-2024-08-06
Original result ↗0.428
- source id
- llmStatsArchive
- leaderboard name
- TAU-bench Airline · Raw
- dimension
- general
- metric
- TAU-bench Airline · Raw
- rank
- 2
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- TAU-bench Airline
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:14.993213+00:00
- paper_link
- https://arxiv.org/abs/2406.12045
- updated_at
- 2025-07-19T19:56:14.993213+00:00
- description
- Part of τ-bench (TAU-bench), a benchmark for Tool-Agent-User interaction in real-world domains. The airline domain evaluates language agents' ability to interact with users through dynamic conversations while following domain-specific rules and using API tools. Agents must handle airline-related tasks and policies reliably.
- benchmark_id
- tau-bench-airline
- multilingual
- false
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- TAU-bench Airline · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/gpt-4-1/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 2
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-2024-08-06
Original result ↗0.417
- source id
- llmStatsArchive
- leaderboard name
- Graphwalks BFS <128k · Raw
- dimension
- general
- metric
- Graphwalks BFS <128k · Raw
- rank
- 2
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- Graphwalks BFS <128k
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- categories
- reasoning
- spatial_reasoning
- created_at
- 2025-07-19T19:56:15.287324+00:00
- updated_at
- 2025-07-19T19:56:15.287324+00:00
- description
- A graph reasoning benchmark that evaluates language models' ability to perform breadth-first search (BFS) operations on graphs with context length under 128k tokens, returning nodes reachable at specified depths.
- benchmark_id
- graphwalks-bfs-<128k
- multilingual
- false
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- Graphwalks BFS <128k · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/gpt-4-1/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 2
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-2024-08-06
Original result ↗0.417
- source id
- llmStatsArchive
- leaderboard name
- Graphwalks BFS <128k · Normalized (reported)
- dimension
- general
- metric
- Graphwalks BFS <128k · Normalized (reported)
- rank
- 2
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- Graphwalks BFS <128k
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- categories
- reasoning
- spatial_reasoning
- created_at
- 2025-07-19T19:56:15.287324+00:00
- updated_at
- 2025-07-19T19:56:15.287324+00:00
- description
- A graph reasoning benchmark that evaluates language models' ability to perform breadth-first search (BFS) operations on graphs with context length under 128k tokens, returning nodes reachable at specified depths.
- benchmark_id
- graphwalks-bfs-<128k
- multilingual
- false
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- Graphwalks BFS <128k · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/gpt-4-1/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 2
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-2024-08-06
Original result ↗0.399
- source id
- llmStatsArchive
- leaderboard name
- MultiChallenge (o3-mini grader) · Raw
- dimension
- general
- metric
- MultiChallenge (o3-mini grader) · Raw
- rank
- 2
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- MultiChallenge (o3-mini grader)
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:15.235758+00:00
- paper_link
- https://arxiv.org/abs/2501.17399
- updated_at
- 2025-07-19T19:56:15.235758+00:00
- description
- A realistic multi-turn conversation evaluation benchmark that challenges frontier LLMs across four key areas: instruction retention, inference memory, reliable versioned editing, and self-coherence. Despite near-perfect scores on existing benchmarks, frontier models achieve less than 50% accuracy on MultiChallenge.
- benchmark_id
- multichallenge-(o3-mini-grader)
- multilingual
- false
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- MultiChallenge (o3-mini grader) · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/gpt-4-1/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 2
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-2024-08-06
Original result ↗0.399
- source id
- llmStatsArchive
- leaderboard name
- MultiChallenge (o3-mini grader) · Normalized (reported)
- dimension
- general
- metric
- MultiChallenge (o3-mini grader) · Normalized (reported)
- rank
- 2
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- MultiChallenge (o3-mini grader)
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:15.235758+00:00
- paper_link
- https://arxiv.org/abs/2501.17399
- updated_at
- 2025-07-19T19:56:15.235758+00:00
- description
- A realistic multi-turn conversation evaluation benchmark that challenges frontier LLMs across four key areas: instruction retention, inference memory, reliable versioned editing, and self-coherence. Despite near-perfect scores on existing benchmarks, frontier models achieve less than 50% accuracy on MultiChallenge.
- benchmark_id
- multichallenge-(o3-mini-grader)
- multilingual
- false
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- MultiChallenge (o3-mini grader) · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/gpt-4-1/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 2
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-2024-08-06
Original result ↗0.354
- source id
- llmStatsArchive
- leaderboard name
- Graphwalks parents <128k · Raw
- dimension
- general
- metric
- Graphwalks parents <128k · Raw
- rank
- 2
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- Graphwalks parents <128k
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- categories
- reasoning
- spatial_reasoning
- created_at
- 2025-07-19T19:56:15.303643+00:00
- updated_at
- 2025-07-19T19:56:15.303643+00:00
- description
- A graph reasoning benchmark that evaluates language models' ability to find parent nodes in graphs with context length under 128k tokens, requiring understanding of graph structure and edge relationships.
- benchmark_id
- graphwalks-parents-<128k
- multilingual
- false
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- Graphwalks parents <128k · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/gpt-4-1/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 2
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-2024-08-06
Original result ↗0.354
- source id
- llmStatsArchive
- leaderboard name
- Graphwalks parents <128k · Normalized (reported)
- dimension
- general
- metric
- Graphwalks parents <128k · Normalized (reported)
- rank
- 2
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- Graphwalks parents <128k
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- categories
- reasoning
- spatial_reasoning
- created_at
- 2025-07-19T19:56:15.303643+00:00
- updated_at
- 2025-07-19T19:56:15.303643+00:00
- description
- A graph reasoning benchmark that evaluates language models' ability to find parent nodes in graphs with context length under 128k tokens, requiring understanding of graph structure and edge relationships.
- benchmark_id
- graphwalks-parents-<128k
- multilingual
- false
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- Graphwalks parents <128k · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/gpt-4-1/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 2
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-2024-08-06
Original result ↗0.319
- source id
- llmStatsArchive
- leaderboard name
- OpenAI-MRCR: 2 needle 128k · Raw
- dimension
- general
- metric
- OpenAI-MRCR: 2 needle 128k · Raw
- rank
- 2
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- OpenAI-MRCR: 2 needle 128k
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:15.266878+00:00
- paper_link
- https://arxiv.org/abs/2403.05530
- updated_at
- 2025-07-19T19:56:15.266878+00:00
- description
- Multi-round Co-reference Resolution (MRCR) benchmark for evaluating an LLM's ability to distinguish between multiple needles hidden in long context. Models are given a long, multi-turn synthetic conversation and must retrieve a specific instance of a repeated request, requiring reasoning and disambiguation skills beyond simple retrieval.
- benchmark_id
- openai-mrcr:-2-needle-128k
- multilingual
- false
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- OpenAI-MRCR: 2 needle 128k · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/gpt-4-1/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 2
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-2024-08-06
Original result ↗0.319
- source id
- llmStatsArchive
- leaderboard name
- OpenAI-MRCR: 2 needle 128k · Normalized (reported)
- dimension
- general
- metric
- OpenAI-MRCR: 2 needle 128k · Normalized (reported)
- rank
- 2
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- OpenAI-MRCR: 2 needle 128k
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:15.266878+00:00
- paper_link
- https://arxiv.org/abs/2403.05530
- updated_at
- 2025-07-19T19:56:15.266878+00:00
- description
- Multi-round Co-reference Resolution (MRCR) benchmark for evaluating an LLM's ability to distinguish between multiple needles hidden in long context. Models are given a long, multi-turn synthetic conversation and must retrieve a specific instance of a repeated request, requiring reasoning and disambiguation skills beyond simple retrieval.
- benchmark_id
- openai-mrcr:-2-needle-128k
- multilingual
- false
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- OpenAI-MRCR: 2 needle 128k · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/gpt-4-1/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 2
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-2024-08-06
Original result ↗0.292
- source id
- llmStatsArchive
- leaderboard name
- Internal API instruction following (hard) · Raw
- dimension
- general
- metric
- Internal API instruction following (hard) · Raw
- rank
- 2
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- Internal API instruction following (hard)
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:15.222560+00:00
- updated_at
- 2025-07-19T19:56:15.222560+00:00
- description
- Internal API instruction following (hard) benchmark - specific documentation not found in official sources
- benchmark_id
- internal-api-instruction-following-(hard)
- multilingual
- false
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- Internal API instruction following (hard) · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/gpt-4-1/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 2
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-2024-08-06
Original result ↗0.292
- source id
- llmStatsArchive
- leaderboard name
- Internal API instruction following (hard) · Normalized (reported)
- dimension
- general
- metric
- Internal API instruction following (hard) · Normalized (reported)
- rank
- 2
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- Internal API instruction following (hard)
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:15.222560+00:00
- updated_at
- 2025-07-19T19:56:15.222560+00:00
- description
- Internal API instruction following (hard) benchmark - specific documentation not found in official sources
- benchmark_id
- internal-api-instruction-following-(hard)
- multilingual
- false
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- Internal API instruction following (hard) · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/gpt-4-1/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 2
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-2024-08-06
Original result ↗0.857
- source id
- llmStatsArchive
- leaderboard name
- MMLU · Raw
- dimension
- general
- metric
- MMLU · Raw
- rank
- 3
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- MMLU
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- categories
- general
- reasoning
- language
- math
- created_at
- 2025-07-19T19:56:11.200416+00:00
- paper_link
- https://arxiv.org/abs/2009.03300
- updated_at
- 2025-07-19T19:56:11.200416+00:00
- description
- Massive Multitask Language Understanding benchmark testing knowledge across 57 diverse subjects including STEM, humanities, social sciences, and professional domains
- benchmark_id
- mmlu
- multilingual
- false
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- MMLU · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/gpt-4-1/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 8
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-2024-08-06
Original result ↗0.857
- source id
- llmStatsArchive
- leaderboard name
- MMLU · Normalized (reported)
- dimension
- general
- metric
- MMLU · Normalized (reported)
- rank
- 3
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- MMLU
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- categories
- general
- reasoning
- language
- math
- created_at
- 2025-07-19T19:56:11.200416+00:00
- paper_link
- https://arxiv.org/abs/2009.03300
- updated_at
- 2025-07-19T19:56:11.200416+00:00
- description
- Massive Multitask Language Understanding benchmark testing knowledge across 57 diverse subjects including STEM, humanities, social sciences, and professional domains
- benchmark_id
- mmlu
- multilingual
- false
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- MMLU · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/gpt-4-1/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/gpt-4o-2024-08-06
- metric_population
- 8
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- gpt-4o-2024-08-06
Original result ↗Published prices
1–3 / 3
$1.25
- Inference provider
- openRouter
- Platform
- OpenRouter
- Unit
- cache_read_tokens_per_million
- Currency
- USD
$2.50
- Inference provider
- openRouter
- Platform
- OpenRouter
- Unit
- input_tokens_per_million
- Currency
- USD
$10.00
- Inference provider
- openRouter
- Platform
- OpenRouter
- Unit
- output_tokens_per_million
- Currency
- USD
Safety assessments
1–1 / 1
35 / 100
- source id
- stanfordFMTI
- canonical model id
- openai/gpt-4o-2024-08-06
- taxonomy
- transparency
- hazard category
- transparency
- hazard name
- Organization transparency
- policy area
- OpenAI
- assessment type
- organization-level disclosure
- metric
- disclosure_score
- result label
- 35 / 100
- sample size
- 100
- updated at
- 2026-11-09T18:08:05.227Z
Source specification observations
1–3 / 3
- field
- context_length
- value
- 128000
- normalized value
- 128000
- scope
- source-model
- source id
- openRouter
- confidence
- 1
- field
- knowledge_cutoff
- reported field
- knowledge_cutoff
- value
- 2023-10-31
- normalized value
- 2023-10-31
- precision
- day
- scope
- source-model
- source id
- openRouter
- confidence
- 1
- field
- max_output_tokens
- value
- 16384
- normalized value
- 16384
- scope
- source-model
- source id
- openRouter
- confidence
- 1
Explore more models