OpenAI: GPT-4o (2024-08-06)

The 2024-08-06 version of GPT-4o offers improved performance in structured outputs, with the ability to supply a JSON schema in the respone_format. Read more [here](https://openai.com/index/introduci…

openai/gpt-4o-2024-08-06

The 2024-08-06 version of GPT-4o offers improved performance in structured outputs, with the ability to supply a JSON schema in the respone_format. Read more [here](https://openai.com/index/introducing-structured-outputs-in-the-api/). GPT-4o ("o" for "omni") is...

Specifications & capabilities

Input
text, image, file
Output
text
Context window · tokens
128000
Knowledge cutoff
2023-10-31
Capabilities
fileInput, multimodal, promptCaching, structuredOutputs, toolCalling, visionInput

Evaluations

1–20 / 143

LLM Stats · GitHub archive

0.665

source id
llmStatsArchive
leaderboard name
ComplexFuncBench · Normalized (reported)
dimension
general
metric
ComplexFuncBench · Normalized (reported)
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
ComplexFuncBench
language
en
modality
text
verified
false
max_score
1
categories
  • long_context
  • reasoning
created_at
2025-07-19T19:56:15.336577+00:00
paper_link
https://arxiv.org/abs/2501.10132
updated_at
2025-07-19T19:56:15.336577+00:00
description
ComplexFuncBench is a benchmark designed to evaluate large language models' capabilities in handling complex function calling scenarios. It encompasses multi-step and constrained function calling tasks that require long-parameter filling, parameter value reasoning, and managing contexts up to 128k tokens. The benchmark includes 1,000 samples across five real-world scenarios.
benchmark_id
complexfuncbench
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
ComplexFuncBench · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/gpt-4-1/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-2024-08-06
metric_population
2
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-2024-08-06
Original result ↗

LLM Stats · GitHub archive

0.665

source id
llmStatsArchive
leaderboard name
ComplexFuncBench · Raw
dimension
general
metric
ComplexFuncBench · Raw
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
ComplexFuncBench
language
en
modality
text
verified
false
max_score
1
categories
  • long_context
  • reasoning
created_at
2025-07-19T19:56:15.336577+00:00
paper_link
https://arxiv.org/abs/2501.10132
updated_at
2025-07-19T19:56:15.336577+00:00
description
ComplexFuncBench is a benchmark designed to evaluate large language models' capabilities in handling complex function calling scenarios. It encompasses multi-step and constrained function calling tasks that require long-parameter filling, parameter value reasoning, and managing contexts up to 128k tokens. The benchmark includes 1,000 samples across five real-world scenarios.
benchmark_id
complexfuncbench
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
ComplexFuncBench · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/gpt-4-1/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-2024-08-06
metric_population
2
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-2024-08-06
Original result ↗

LLM Stats · GitHub archive

0.853

source id
llmStatsArchive
leaderboard name
CharXiv-D · Normalized (reported)
dimension
general
metric
CharXiv-D · Normalized (reported)
rank
2
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
CharXiv-D
language
en
modality
multimodal
verified
false
max_score
1
categories
  • reasoning
  • vision
  • multimodal
created_at
2025-07-19T19:56:15.325204+00:00
paper_link
https://arxiv.org/abs/2406.18521
updated_at
2025-07-19T19:56:15.325204+00:00
description
CharXiv-D is the descriptive questions subset of the CharXiv benchmark, designed to assess multimodal large language models' ability to extract basic information from scientific charts. It contains descriptive questions covering information extraction, enumeration, pattern recognition, and counting across 2,323 diverse charts from arXiv papers, all curated and verified by human experts.
benchmark_id
charxiv-d
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
CharXiv-D · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/gpt-4-1/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-2024-08-06
metric_population
2
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-2024-08-06
Original result ↗

LLM Stats · GitHub archive

0.853

source id
llmStatsArchive
leaderboard name
CharXiv-D · Raw
dimension
general
metric
CharXiv-D · Raw
rank
2
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
CharXiv-D
language
en
modality
multimodal
verified
false
max_score
1
categories
  • reasoning
  • vision
  • multimodal
created_at
2025-07-19T19:56:15.325204+00:00
paper_link
https://arxiv.org/abs/2406.18521
updated_at
2025-07-19T19:56:15.325204+00:00
description
CharXiv-D is the descriptive questions subset of the CharXiv benchmark, designed to assess multimodal large language models' ability to extract basic information from scientific charts. It contains descriptive questions covering information extraction, enumeration, pattern recognition, and counting across 2,323 diverse charts from arXiv papers, all curated and verified by human experts.
benchmark_id
charxiv-d
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
CharXiv-D · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/gpt-4-1/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-2024-08-06
metric_population
2
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-2024-08-06
Original result ↗

LLM Stats · GitHub archive

0.603

source id
llmStatsArchive
leaderboard name
TAU-bench Retail · Raw
dimension
general
metric
TAU-bench Retail · Raw
rank
2
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
TAU-bench Retail
language
en
modality
text
verified
false
max_score
1
categories
  • reasoning
  • communication
created_at
2025-07-19T19:56:14.965635+00:00
paper_link
https://arxiv.org/abs/2406.12045
updated_at
2025-07-19T19:56:14.965635+00:00
description
A benchmark for evaluating tool-agent-user interaction in retail environments. Tests language agents' ability to handle dynamic conversations with users while using domain-specific API tools and following policy guidelines. Evaluates agents on tasks like order cancellations, address changes, and order status checks through multi-turn conversations.
benchmark_id
tau-bench-retail
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
TAU-bench Retail · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/gpt-4-1/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-2024-08-06
metric_population
2
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-2024-08-06
Original result ↗

LLM Stats · GitHub archive

0.603

source id
llmStatsArchive
leaderboard name
TAU-bench Retail · Normalized (reported)
dimension
general
metric
TAU-bench Retail · Normalized (reported)
rank
2
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
TAU-bench Retail
language
en
modality
text
verified
false
max_score
1
categories
  • reasoning
  • communication
created_at
2025-07-19T19:56:14.965635+00:00
paper_link
https://arxiv.org/abs/2406.12045
updated_at
2025-07-19T19:56:14.965635+00:00
description
A benchmark for evaluating tool-agent-user interaction in retail environments. Tests language agents' ability to handle dynamic conversations with users while using domain-specific API tools and following policy guidelines. Evaluates agents on tasks like order cancellations, address changes, and order status checks through multi-turn conversations.
benchmark_id
tau-bench-retail
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
TAU-bench Retail · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/gpt-4-1/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-2024-08-06
metric_population
2
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-2024-08-06
Original result ↗

LLM Stats · GitHub archive

0.428

source id
llmStatsArchive
leaderboard name
TAU-bench Airline · Normalized (reported)
dimension
general
metric
TAU-bench Airline · Normalized (reported)
rank
2
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
TAU-bench Airline
language
en
modality
text
verified
false
max_score
1
categories
  • reasoning
  • communication
created_at
2025-07-19T19:56:14.993213+00:00
paper_link
https://arxiv.org/abs/2406.12045
updated_at
2025-07-19T19:56:14.993213+00:00
description
Part of τ-bench (TAU-bench), a benchmark for Tool-Agent-User interaction in real-world domains. The airline domain evaluates language agents' ability to interact with users through dynamic conversations while following domain-specific rules and using API tools. Agents must handle airline-related tasks and policies reliably.
benchmark_id
tau-bench-airline
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
TAU-bench Airline · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/gpt-4-1/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-2024-08-06
metric_population
2
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-2024-08-06
Original result ↗

LLM Stats · GitHub archive

0.428

source id
llmStatsArchive
leaderboard name
TAU-bench Airline · Raw
dimension
general
metric
TAU-bench Airline · Raw
rank
2
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
TAU-bench Airline
language
en
modality
text
verified
false
max_score
1
categories
  • reasoning
  • communication
created_at
2025-07-19T19:56:14.993213+00:00
paper_link
https://arxiv.org/abs/2406.12045
updated_at
2025-07-19T19:56:14.993213+00:00
description
Part of τ-bench (TAU-bench), a benchmark for Tool-Agent-User interaction in real-world domains. The airline domain evaluates language agents' ability to interact with users through dynamic conversations while following domain-specific rules and using API tools. Agents must handle airline-related tasks and policies reliably.
benchmark_id
tau-bench-airline
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
TAU-bench Airline · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/gpt-4-1/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-2024-08-06
metric_population
2
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-2024-08-06
Original result ↗

LLM Stats · GitHub archive

0.417

source id
llmStatsArchive
leaderboard name
Graphwalks BFS <128k · Raw
dimension
general
metric
Graphwalks BFS <128k · Raw
rank
2
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
Graphwalks BFS <128k
language
en
modality
text
verified
false
max_score
1
categories
  • reasoning
  • spatial_reasoning
created_at
2025-07-19T19:56:15.287324+00:00
updated_at
2025-07-19T19:56:15.287324+00:00
description
A graph reasoning benchmark that evaluates language models' ability to perform breadth-first search (BFS) operations on graphs with context length under 128k tokens, returning nodes reachable at specified depths.
benchmark_id
graphwalks-bfs-<128k
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
Graphwalks BFS <128k · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/gpt-4-1/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-2024-08-06
metric_population
2
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-2024-08-06
Original result ↗

LLM Stats · GitHub archive

0.417

source id
llmStatsArchive
leaderboard name
Graphwalks BFS <128k · Normalized (reported)
dimension
general
metric
Graphwalks BFS <128k · Normalized (reported)
rank
2
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
Graphwalks BFS <128k
language
en
modality
text
verified
false
max_score
1
categories
  • reasoning
  • spatial_reasoning
created_at
2025-07-19T19:56:15.287324+00:00
updated_at
2025-07-19T19:56:15.287324+00:00
description
A graph reasoning benchmark that evaluates language models' ability to perform breadth-first search (BFS) operations on graphs with context length under 128k tokens, returning nodes reachable at specified depths.
benchmark_id
graphwalks-bfs-<128k
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
Graphwalks BFS <128k · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/gpt-4-1/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-2024-08-06
metric_population
2
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-2024-08-06
Original result ↗

LLM Stats · GitHub archive

0.399

source id
llmStatsArchive
leaderboard name
MultiChallenge (o3-mini grader) · Raw
dimension
general
metric
MultiChallenge (o3-mini grader) · Raw
rank
2
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
MultiChallenge (o3-mini grader)
language
en
modality
text
verified
false
max_score
1
categories
  • reasoning
  • language
created_at
2025-07-19T19:56:15.235758+00:00
paper_link
https://arxiv.org/abs/2501.17399
updated_at
2025-07-19T19:56:15.235758+00:00
description
A realistic multi-turn conversation evaluation benchmark that challenges frontier LLMs across four key areas: instruction retention, inference memory, reliable versioned editing, and self-coherence. Despite near-perfect scores on existing benchmarks, frontier models achieve less than 50% accuracy on MultiChallenge.
benchmark_id
multichallenge-(o3-mini-grader)
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
MultiChallenge (o3-mini grader) · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/gpt-4-1/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-2024-08-06
metric_population
2
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-2024-08-06
Original result ↗

LLM Stats · GitHub archive

0.399

source id
llmStatsArchive
leaderboard name
MultiChallenge (o3-mini grader) · Normalized (reported)
dimension
general
metric
MultiChallenge (o3-mini grader) · Normalized (reported)
rank
2
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
MultiChallenge (o3-mini grader)
language
en
modality
text
verified
false
max_score
1
categories
  • reasoning
  • language
created_at
2025-07-19T19:56:15.235758+00:00
paper_link
https://arxiv.org/abs/2501.17399
updated_at
2025-07-19T19:56:15.235758+00:00
description
A realistic multi-turn conversation evaluation benchmark that challenges frontier LLMs across four key areas: instruction retention, inference memory, reliable versioned editing, and self-coherence. Despite near-perfect scores on existing benchmarks, frontier models achieve less than 50% accuracy on MultiChallenge.
benchmark_id
multichallenge-(o3-mini-grader)
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
MultiChallenge (o3-mini grader) · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/gpt-4-1/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-2024-08-06
metric_population
2
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-2024-08-06
Original result ↗

LLM Stats · GitHub archive

0.354

source id
llmStatsArchive
leaderboard name
Graphwalks parents <128k · Raw
dimension
general
metric
Graphwalks parents <128k · Raw
rank
2
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
Graphwalks parents <128k
language
en
modality
text
verified
false
max_score
1
categories
  • reasoning
  • spatial_reasoning
created_at
2025-07-19T19:56:15.303643+00:00
updated_at
2025-07-19T19:56:15.303643+00:00
description
A graph reasoning benchmark that evaluates language models' ability to find parent nodes in graphs with context length under 128k tokens, requiring understanding of graph structure and edge relationships.
benchmark_id
graphwalks-parents-<128k
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
Graphwalks parents <128k · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/gpt-4-1/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-2024-08-06
metric_population
2
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-2024-08-06
Original result ↗

LLM Stats · GitHub archive

0.354

source id
llmStatsArchive
leaderboard name
Graphwalks parents <128k · Normalized (reported)
dimension
general
metric
Graphwalks parents <128k · Normalized (reported)
rank
2
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
Graphwalks parents <128k
language
en
modality
text
verified
false
max_score
1
categories
  • reasoning
  • spatial_reasoning
created_at
2025-07-19T19:56:15.303643+00:00
updated_at
2025-07-19T19:56:15.303643+00:00
description
A graph reasoning benchmark that evaluates language models' ability to find parent nodes in graphs with context length under 128k tokens, requiring understanding of graph structure and edge relationships.
benchmark_id
graphwalks-parents-<128k
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
Graphwalks parents <128k · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/gpt-4-1/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-2024-08-06
metric_population
2
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-2024-08-06
Original result ↗

LLM Stats · GitHub archive

0.319

source id
llmStatsArchive
leaderboard name
OpenAI-MRCR: 2 needle 128k · Raw
dimension
general
metric
OpenAI-MRCR: 2 needle 128k · Raw
rank
2
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
OpenAI-MRCR: 2 needle 128k
language
en
modality
text
verified
false
max_score
1
categories
  • long_context
  • reasoning
created_at
2025-07-19T19:56:15.266878+00:00
paper_link
https://arxiv.org/abs/2403.05530
updated_at
2025-07-19T19:56:15.266878+00:00
description
Multi-round Co-reference Resolution (MRCR) benchmark for evaluating an LLM's ability to distinguish between multiple needles hidden in long context. Models are given a long, multi-turn synthetic conversation and must retrieve a specific instance of a repeated request, requiring reasoning and disambiguation skills beyond simple retrieval.
benchmark_id
openai-mrcr:-2-needle-128k
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
OpenAI-MRCR: 2 needle 128k · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/gpt-4-1/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-2024-08-06
metric_population
2
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-2024-08-06
Original result ↗

LLM Stats · GitHub archive

0.319

source id
llmStatsArchive
leaderboard name
OpenAI-MRCR: 2 needle 128k · Normalized (reported)
dimension
general
metric
OpenAI-MRCR: 2 needle 128k · Normalized (reported)
rank
2
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
OpenAI-MRCR: 2 needle 128k
language
en
modality
text
verified
false
max_score
1
categories
  • long_context
  • reasoning
created_at
2025-07-19T19:56:15.266878+00:00
paper_link
https://arxiv.org/abs/2403.05530
updated_at
2025-07-19T19:56:15.266878+00:00
description
Multi-round Co-reference Resolution (MRCR) benchmark for evaluating an LLM's ability to distinguish between multiple needles hidden in long context. Models are given a long, multi-turn synthetic conversation and must retrieve a specific instance of a repeated request, requiring reasoning and disambiguation skills beyond simple retrieval.
benchmark_id
openai-mrcr:-2-needle-128k
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
OpenAI-MRCR: 2 needle 128k · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/gpt-4-1/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-2024-08-06
metric_population
2
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-2024-08-06
Original result ↗

LLM Stats · GitHub archive

0.292

source id
llmStatsArchive
leaderboard name
Internal API instruction following (hard) · Raw
dimension
general
metric
Internal API instruction following (hard) · Raw
rank
2
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
Internal API instruction following (hard)
language
en
modality
text
verified
false
max_score
1
categories
  • general
created_at
2025-07-19T19:56:15.222560+00:00
updated_at
2025-07-19T19:56:15.222560+00:00
description
Internal API instruction following (hard) benchmark - specific documentation not found in official sources
benchmark_id
internal-api-instruction-following-(hard)
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
Internal API instruction following (hard) · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/gpt-4-1/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-2024-08-06
metric_population
2
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-2024-08-06
Original result ↗

LLM Stats · GitHub archive

0.292

source id
llmStatsArchive
leaderboard name
Internal API instruction following (hard) · Normalized (reported)
dimension
general
metric
Internal API instruction following (hard) · Normalized (reported)
rank
2
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
Internal API instruction following (hard)
language
en
modality
text
verified
false
max_score
1
categories
  • general
created_at
2025-07-19T19:56:15.222560+00:00
updated_at
2025-07-19T19:56:15.222560+00:00
description
Internal API instruction following (hard) benchmark - specific documentation not found in official sources
benchmark_id
internal-api-instruction-following-(hard)
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
Internal API instruction following (hard) · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/gpt-4-1/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-2024-08-06
metric_population
2
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-2024-08-06
Original result ↗

LLM Stats · GitHub archive

0.857

source id
llmStatsArchive
leaderboard name
MMLU · Raw
dimension
general
metric
MMLU · Raw
rank
3
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
MMLU
language
en
modality
text
verified
false
max_score
1
categories
  • general
  • reasoning
  • language
  • math
created_at
2025-07-19T19:56:11.200416+00:00
paper_link
https://arxiv.org/abs/2009.03300
updated_at
2025-07-19T19:56:11.200416+00:00
description
Massive Multitask Language Understanding benchmark testing knowledge across 57 diverse subjects including STEM, humanities, social sciences, and professional domains
benchmark_id
mmlu
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
MMLU · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/gpt-4-1/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-2024-08-06
metric_population
8
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-2024-08-06
Original result ↗

LLM Stats · GitHub archive

0.857

source id
llmStatsArchive
leaderboard name
MMLU · Normalized (reported)
dimension
general
metric
MMLU · Normalized (reported)
rank
3
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
MMLU
language
en
modality
text
verified
false
max_score
1
categories
  • general
  • reasoning
  • language
  • math
created_at
2025-07-19T19:56:11.200416+00:00
paper_link
https://arxiv.org/abs/2009.03300
updated_at
2025-07-19T19:56:11.200416+00:00
description
Massive Multitask Language Understanding benchmark testing knowledge across 57 diverse subjects including STEM, humanities, social sciences, and professional domains
benchmark_id
mmlu
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
MMLU · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/gpt-4o-2024-08-06/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/gpt-4-1/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/gpt-4o-2024-08-06
metric_population
8
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
gpt-4o-2024-08-06
Original result ↗

Published prices

1–3 / 3

openai/gpt-4o-2024-08-06 ↗

$1.25

Inference provider
openRouter
Platform
OpenRouter
Unit
cache_read_tokens_per_million
Currency
USD
Billing conditions

Safety assessments

1–1 / 1

Foundation Model Transparency Index 2025 ↗

35 / 100

source id
stanfordFMTI
canonical model id
openai/gpt-4o-2024-08-06
taxonomy
transparency
hazard category
transparency
hazard name
Organization transparency
policy area
OpenAI
assessment type
organization-level disclosure
metric
disclosure_score
result label
35 / 100
sample size
100
updated at
2026-11-09T18:08:05.227Z

Source specification observations

1–3 / 3

openRouter ↗

field
context_length
value
128000
normalized value
128000
scope
source-model
source id
openRouter
confidence
1

openRouter ↗

field
knowledge_cutoff
reported field
knowledge_cutoff
value
2023-10-31
normalized value
2023-10-31
precision
day
scope
source-model
source id
openRouter
confidence
1

openRouter ↗

field
max_output_tokens
value
16384
normalized value
16384
scope
source-model
source id
openRouter
confidence
1

Original sources

1–7 / 7

OpenRouter · Model page ↗

GitHub · Catalog ↗

GitHub · Catalog ↗

Kaggle · Catalog ↗

OpenRouter · Catalog ↗

OpenRouter · Catalog ↗

www.swebench.com · Catalog ↗

Source links

1–7 / 7

Explore more models

Models

Loading
EVIDENCE / COMPARISON

Benchmarks

Explore every benchmark

Loading

These charts contain model-linked results. The charts above also include unlinked source profiles. Compare scores within each evaluation. Model quality, usage, transparency and system performance measure different things.

Install TensorTrace