OpenAI: o3

o3 is a well-rounded and powerful model across domains. It sets a new standard for math, science, coding, and visual reasoning tasks. It also excels at technical writing and instruction-following. Us…

openai/o3-2025-04-16

o3 is a well-rounded and powerful model across domains. It sets a new standard for math, science, coding, and visual reasoning tasks. It also excels at technical writing and instruction-following. Use it to think through multi-step problems that involve analysis across text, code, and images.

Specifications & capabilities

Input
image, text, file
Output
text
Context window · tokens
200000
Capabilities
fileInput, multimodal, promptCaching, reasoning, structuredOutputs, toolCalling, visionInput
Description source ↗

Evaluations

1–20 / 328

LLM Stats · GitHub archive

0.868

source id
llmStatsArchive
leaderboard name
MathVista · Normalized (reported)
dimension
general
metric
MathVista · Normalized (reported)
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
MathVista
language
en
modality
multimodal
verified
false
max_score
1
categories
  • math
  • vision
  • multimodal
created_at
2025-07-19T19:56:12.069611+00:00
paper_link
https://arxiv.org/abs/2310.02255
updated_at
2025-07-19T19:56:12.069611+00:00
description
MathVista evaluates mathematical reasoning of foundation models in visual contexts. It consists of 6,141 examples derived from 28 existing multimodal datasets and 3 newly created datasets (IQTest, FunctionQA, and PaperQA), combining challenges from diverse mathematical and visual tasks to assess models' ability to understand complex figures and perform rigorous reasoning.
benchmark_id
mathvista
multilingual
false
analysis_method
accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
MathVista · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/o3-2025-04-16/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/introducing-o3-and-o4-mini/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/o3-2025-04-16
metric_population
4
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
o3-2025-04-16
Original result ↗

LLM Stats · GitHub archive

0.868

source id
llmStatsArchive
leaderboard name
MathVista · Raw
dimension
general
metric
MathVista · Raw
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
MathVista
language
en
modality
multimodal
verified
false
max_score
1
categories
  • math
  • vision
  • multimodal
created_at
2025-07-19T19:56:12.069611+00:00
paper_link
https://arxiv.org/abs/2310.02255
updated_at
2025-07-19T19:56:12.069611+00:00
description
MathVista evaluates mathematical reasoning of foundation models in visual contexts. It consists of 6,141 examples derived from 28 existing multimodal datasets and 3 newly created datasets (IQTest, FunctionQA, and PaperQA), combining challenges from diverse mathematical and visual tasks to assess models' ability to understand complex figures and perform rigorous reasoning.
benchmark_id
mathvista
multilingual
false
analysis_method
accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
MathVista · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/o3-2025-04-16/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/introducing-o3-and-o4-mini/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/o3-2025-04-16
metric_population
4
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
o3-2025-04-16
Original result ↗

LLM Stats · GitHub archive

0.864

source id
llmStatsArchive
leaderboard name
AIME 2025 · Raw
dimension
general
metric
AIME 2025 · Raw
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
AIME 2025
language
en
modality
text
verified
false
max_score
1
categories
  • math
  • reasoning
created_at
2025-09-05T00:00:00.000000+00:00
paper_link
https://arxiv.org/abs/2503.21380
updated_at
2025-09-05T00:00:00.000000+00:00
description
All 30 problems from the 2025 American Invitational Mathematics Examination (AIME I and AIME II), testing olympiad-level mathematical reasoning with integer answers from 000-999. Used as an AI benchmark to evaluate large language models' ability to solve complex mathematical problems requiring multi-step logical deductions and structured symbolic reasoning.
benchmark_id
aime-2025
multilingual
false
analysis_method
pass@1 (no tools)
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
AIME 2025 · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/o3-2025-04-16/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/introducing-o3-and-o4-mini/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/o3-2025-04-16
metric_population
1
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
o3-2025-04-16
Original result ↗

LLM Stats · GitHub archive

0.864

source id
llmStatsArchive
leaderboard name
AIME 2025 · Normalized (reported)
dimension
general
metric
AIME 2025 · Normalized (reported)
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
AIME 2025
language
en
modality
text
verified
false
max_score
1
categories
  • math
  • reasoning
created_at
2025-09-05T00:00:00.000000+00:00
paper_link
https://arxiv.org/abs/2503.21380
updated_at
2025-09-05T00:00:00.000000+00:00
description
All 30 problems from the 2025 American Invitational Mathematics Examination (AIME I and AIME II), testing olympiad-level mathematical reasoning with integer answers from 000-999. Used as an AI benchmark to evaluate large language models' ability to solve complex mathematical problems requiring multi-step logical deductions and structured symbolic reasoning.
benchmark_id
aime-2025
multilingual
false
analysis_method
pass@1 (no tools)
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
AIME 2025 · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/o3-2025-04-16/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/introducing-o3-and-o4-mini/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/o3-2025-04-16
metric_population
1
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
o3-2025-04-16
Original result ↗

LLM Stats · GitHub archive

0.813

source id
llmStatsArchive
leaderboard name
Aider-Polyglot · Raw
dimension
general
metric
Aider-Polyglot · Raw
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
Aider-Polyglot
language
en
modality
text
verified
false
max_score
1
categories
  • general
  • code
created_at
2025-09-05T00:00:00.000000+00:00
updated_at
2025-09-30T00:00:00.000000+00:00
description
A coding benchmark that evaluates LLMs on 225 challenging Exercism programming exercises across C++, Go, Java, JavaScript, Python, and Rust. Models receive two attempts to solve each problem, with test error feedback provided after the first attempt if it fails. The benchmark measures both initial problem-solving ability and capacity to edit code based on error feedback, providing an end-to-end evaluation of code generation and editing capabilities across multiple programming languages.
benchmark_id
aider-polyglot
multilingual
false
implementation_link
https://github.com/Aider-AI/polyglot-benchmark
analysis_method
accuracy (whole)
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
Aider-Polyglot · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/o3-2025-04-16/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/introducing-o3-and-o4-mini/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/o3-2025-04-16
metric_population
1
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
o3-2025-04-16
Original result ↗

LLM Stats · GitHub archive

0.813

source id
llmStatsArchive
leaderboard name
Aider-Polyglot · Normalized (reported)
dimension
general
metric
Aider-Polyglot · Normalized (reported)
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
Aider-Polyglot
language
en
modality
text
verified
false
max_score
1
categories
  • general
  • code
created_at
2025-09-05T00:00:00.000000+00:00
updated_at
2025-09-30T00:00:00.000000+00:00
description
A coding benchmark that evaluates LLMs on 225 challenging Exercism programming exercises across C++, Go, Java, JavaScript, Python, and Rust. Models receive two attempts to solve each problem, with test error feedback provided after the first attempt if it fails. The benchmark measures both initial problem-solving ability and capacity to edit code based on error feedback, providing an end-to-end evaluation of code generation and editing capabilities across multiple programming languages.
benchmark_id
aider-polyglot
multilingual
false
implementation_link
https://github.com/Aider-AI/polyglot-benchmark
analysis_method
accuracy (whole)
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
Aider-Polyglot · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/o3-2025-04-16/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/introducing-o3-and-o4-mini/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/o3-2025-04-16
metric_population
1
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
o3-2025-04-16
Original result ↗

LLM Stats · GitHub archive

0.691

source id
llmStatsArchive
leaderboard name
SWE-Bench Verified · Normalized (reported)
dimension
general
metric
SWE-Bench Verified · Normalized (reported)
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
SWE-Bench Verified
language
en
modality
text
verified
false
max_score
1
categories
  • reasoning
  • frontend_development
  • code
created_at
2025-07-19T19:56:13.812805+00:00
paper_link
https://arxiv.org/abs/2310.06770
updated_at
2025-07-19T19:56:13.812805+00:00
description
A verified subset of 500 software engineering problems from real GitHub issues, validated by human annotators for evaluating language models' ability to resolve real-world coding issues by generating patches for Python codebases.
benchmark_id
swe-bench-verified
multilingual
false
analysis_method
accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
SWE-Bench Verified · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/o3-2025-04-16/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/introducing-o3-and-o4-mini/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/o3-2025-04-16
metric_population
2
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
o3-2025-04-16
Original result ↗

LLM Stats · GitHub archive

0.691

source id
llmStatsArchive
leaderboard name
SWE-Bench Verified · Raw
dimension
general
metric
SWE-Bench Verified · Raw
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
SWE-Bench Verified
language
en
modality
text
verified
false
max_score
1
categories
  • reasoning
  • frontend_development
  • code
created_at
2025-07-19T19:56:13.812805+00:00
paper_link
https://arxiv.org/abs/2310.06770
updated_at
2025-07-19T19:56:13.812805+00:00
description
A verified subset of 500 software engineering problems from real GitHub issues, validated by human annotators for evaluating language models' ability to resolve real-world coding issues by generating patches for Python codebases.
benchmark_id
swe-bench-verified
multilingual
false
analysis_method
accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
SWE-Bench Verified · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/o3-2025-04-16/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/introducing-o3-and-o4-mini/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/o3-2025-04-16
metric_population
2
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
o3-2025-04-16
Original result ↗

LLM Stats · GitHub archive

0.63

source id
llmStatsArchive
leaderboard name
Tau-bench · Raw
dimension
general
metric
Tau-bench · Raw
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
Tau-bench
language
en
modality
text
verified
false
max_score
1
categories
  • general
  • reasoning
created_at
2025-07-19T19:56:15.219001+00:00
paper_link
https://arxiv.org/abs/2406.12045
updated_at
2025-07-19T19:56:15.219001+00:00
description
τ-bench: A benchmark for tool-agent-user interaction in real-world domains. Tests language agents' ability to interact with users and follow domain-specific rules through dynamic conversations using API tools and policy guidelines across retail and airline domains. Evaluates consistency and reliability of agent behavior over multiple trials.
benchmark_id
tau-bench
multilingual
false
analysis_method
accuracy (avg Airline/Retail)
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
Tau-bench · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/o3-2025-04-16/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/introducing-o3-and-o4-mini/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/o3-2025-04-16
metric_population
1
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
o3-2025-04-16
Original result ↗

LLM Stats · GitHub archive

0.63

source id
llmStatsArchive
leaderboard name
Tau-bench · Normalized (reported)
dimension
general
metric
Tau-bench · Normalized (reported)
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
Tau-bench
language
en
modality
text
verified
false
max_score
1
categories
  • general
  • reasoning
created_at
2025-07-19T19:56:15.219001+00:00
paper_link
https://arxiv.org/abs/2406.12045
updated_at
2025-07-19T19:56:15.219001+00:00
description
τ-bench: A benchmark for tool-agent-user interaction in real-world domains. Tests language agents' ability to interact with users and follow domain-specific rules through dynamic conversations using API tools and policy guidelines across retail and airline domains. Evaluates consistency and reliability of agent behavior over multiple trials.
benchmark_id
tau-bench
multilingual
false
analysis_method
accuracy (avg Airline/Retail)
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
Tau-bench · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/o3-2025-04-16/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/introducing-o3-and-o4-mini/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/o3-2025-04-16
metric_population
1
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
o3-2025-04-16
Original result ↗

LLM Stats · GitHub archive

0.565

source id
llmStatsArchive
leaderboard name
Scale MultiChallenge · Raw
dimension
general
metric
Scale MultiChallenge · Raw
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
Scale MultiChallenge
language
en
modality
text
verified
false
max_score
1
categories
  • reasoning
  • communication
  • general
created_at
2025-07-19T19:56:15.205789+00:00
paper_link
https://arxiv.org/abs/2501.17399
updated_at
2025-07-19T19:56:15.205789+00:00
description
MultiChallenge is a realistic multi-turn conversation evaluation benchmark developed by Scale AI that evaluates large language models on four challenging conversation categories: instruction retention, inference memory of user information, reliable versioned editing, and self-coherence. Each challenge requires accurate instruction-following, context allocation, and in-context reasoning. Despite achieving near-perfect scores on existing multi-turn evaluation benchmarks, all frontier models have less than 50% accuracy on MultiChallenge.
benchmark_id
scale-multichallenge
multilingual
false
analysis_method
accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
Scale MultiChallenge · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/o3-2025-04-16/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/introducing-o3-and-o4-mini/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/o3-2025-04-16
metric_population
2
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
o3-2025-04-16
Original result ↗

LLM Stats · GitHub archive

0.565

source id
llmStatsArchive
leaderboard name
Scale MultiChallenge · Normalized (reported)
dimension
general
metric
Scale MultiChallenge · Normalized (reported)
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
Scale MultiChallenge
language
en
modality
text
verified
false
max_score
1
categories
  • reasoning
  • communication
  • general
created_at
2025-07-19T19:56:15.205789+00:00
paper_link
https://arxiv.org/abs/2501.17399
updated_at
2025-07-19T19:56:15.205789+00:00
description
MultiChallenge is a realistic multi-turn conversation evaluation benchmark developed by Scale AI that evaluates large language models on four challenging conversation categories: instruction retention, inference memory of user information, reliable versioned editing, and self-coherence. Each challenge requires accurate instruction-following, context allocation, and in-context reasoning. Despite achieving near-perfect scores on existing multi-turn evaluation benchmarks, all frontier models have less than 50% accuracy on MultiChallenge.
benchmark_id
scale-multichallenge
multilingual
false
analysis_method
accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
Scale MultiChallenge · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/o3-2025-04-16/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/introducing-o3-and-o4-mini/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/o3-2025-04-16
metric_population
2
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
o3-2025-04-16
Original result ↗

Kaggle Benchmarks

50%

source id
kaggleBenchmarks
leaderboard name
ITBench · SRE Score
dimension
reasoning
metric
ITBench · SRE Score
rank
1
aggregation scope
kaggle
true
status
published
license
Apache-2.0
direction
higherIsBetter
conditions
task
id
1172
name
SRE Score
version
8
sort_order
DESCENDING
display_type
rawPercentages
benchmark
id
246
name
ITBench
type
INDIVIDUAL
owner
ibm-research
version
1
categories
  • evaluation
  • general knowledge and reasoning
version_id
225
maintainer_tier
Kaggle-maintained
maintenance_level
KAGGLE_MAINTAINED
candidate
candidate_class
modelVersion
record_kind
aggregate
attribution
Kaggle · ibm-research · ITBench
license_url
https://www.apache.org/licenses/LICENSE-2.0
metric_kind
ITBench · SRE Score
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
225/leaderboard.json
source_page
https://www.kaggle.com/benchmarks/ibm-research/itbench/versions/1
measurements
  • name
    SRE Score
    unit
    percent
    scale
    100
    metric
    ITBench · SRE Score
    result
    resultCase
    numericResult
    booleanResult
    false
    numericResult
    value
    50
    standardDeviation
    0
    confidenceInterval
    0
    hasStandardDeviation
    false
    hasConfidenceInterval
    false
    hasUnevenConfidenceInterval
    false
    taskVersionId
    0
    evaluationDate
    2025-12-01T22:27:57.070731Z
    hasBooleanResult
    false
    hasNumericResult
    true
    hasTaskVersionId
    false
    hasEvaluationDate
    true
    numericResultNullable
    value
    50
    standardDeviation
    0
    confidenceInterval
    0
    hasStandardDeviation
    false
    hasConfidenceInterval
    false
    hasUnevenConfidenceInterval
    false
    hasNumericResultPublic
    false
    customAdditionalResults
      hasNumericResultPrivate
      false
      task_id
      1172
      task_url
      /benchmarks/tasks/benchmarkler/itbench-sre-main
      sort_order
      DESCENDING
      source_name
      SRE Score
      display_type
      rawPercentages
      task_version
      8
      task_display_type
      RAW_VALUES
      normalization_status
      normalized
    published_at
    2025-12-02T12:36:17.407125500Z
    candidate_class
    modelVersion
    source_revision
    b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
    publisher_anchor
    openai/o3-2025-04-16
    metric_population
    13
    benchmark_evaluation
    true
    task_metadata_source
    benchmarkVersion
    candidate_metadata_source
    benchmarkVersionMappings
    evaluated at
    2025-12-01T22:27:57.070Z
    source model id
    o3
    Original result ↗

    LLM Stats · GitHub archive

    0.202

    source id
    llmStatsArchive
    leaderboard name
    Humanity's Last Exam · Raw
    dimension
    general
    metric
    Humanity's Last Exam · Raw
    rank
    1
    aggregation scope
    status
    archived
    license
    CC-BY-4.0
    direction
    higherIsBetter
    conditions
    language
    en
    benchmark
    name
    Humanity's Last Exam
    language
    en
    modality
    multimodal
    verified
    false
    max_score
    1
    categories
    • general
    created_at
    2025-07-19T19:56:12.507693+00:00
    paper_link
    https://arxiv.org/abs/2501.14249
    updated_at
    2025-07-19T19:56:12.507693+00:00
    description
    A multi-modal benchmark at the frontier of human knowledge with 2,500 questions across dozens of subjects including mathematics, humanities, and natural sciences, created by nearly 1000 subject expert contributors from over 500 institutions
    benchmark_id
    humanity's-last-exam
    multilingual
    false
    analysis_method
    accuracy (no tools)
    is_self_reported
    true
    verified_by_llmstats
    false
    attribution
    Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
    license_url
    https://creativecommons.org/licenses/by/4.0/
    metric_kind
    Humanity's Last Exam · Raw
    rank_method
    competition rank among reported evaluations in identical conditions, including unlinked observations
    source_file
    data/organizations/openai/models/o3-2025-04-16/benchmarks.json
    archive_kind
    llmStats
    evidence_url
    https://openai.com/index/introducing-o3-and-o4-mini/
    git_revision
    872b75f63b8d853c46ed17856424029a0c47b1d2
    published_at
    2025-10-24T17:47:59Z
    snapshot_date
    2025-10-24T17:47:59Z
    source_revision
    6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
    publisher_anchor
    openai/o3-2025-04-16
    metric_population
    2
    current_catalog_url
    https://llm-stats.com/
    benchmark_evaluation
    true
    source model id
    o3-2025-04-16
    Original result ↗

    LLM Stats · GitHub archive

    0.202

    source id
    llmStatsArchive
    leaderboard name
    Humanity's Last Exam · Normalized (reported)
    dimension
    general
    metric
    Humanity's Last Exam · Normalized (reported)
    rank
    1
    aggregation scope
    status
    archived
    license
    CC-BY-4.0
    direction
    higherIsBetter
    conditions
    language
    en
    benchmark
    name
    Humanity's Last Exam
    language
    en
    modality
    multimodal
    verified
    false
    max_score
    1
    categories
    • general
    created_at
    2025-07-19T19:56:12.507693+00:00
    paper_link
    https://arxiv.org/abs/2501.14249
    updated_at
    2025-07-19T19:56:12.507693+00:00
    description
    A multi-modal benchmark at the frontier of human knowledge with 2,500 questions across dozens of subjects including mathematics, humanities, and natural sciences, created by nearly 1000 subject expert contributors from over 500 institutions
    benchmark_id
    humanity's-last-exam
    multilingual
    false
    analysis_method
    accuracy (no tools)
    is_self_reported
    true
    verified_by_llmstats
    false
    attribution
    Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
    license_url
    https://creativecommons.org/licenses/by/4.0/
    metric_kind
    Humanity's Last Exam · Normalized (reported)
    rank_method
    competition rank among reported evaluations in identical conditions, including unlinked observations
    source_file
    data/organizations/openai/models/o3-2025-04-16/benchmarks.json
    archive_kind
    llmStats
    evidence_url
    https://openai.com/index/introducing-o3-and-o4-mini/
    git_revision
    872b75f63b8d853c46ed17856424029a0c47b1d2
    published_at
    2025-10-24T17:47:59Z
    snapshot_date
    2025-10-24T17:47:59Z
    source_revision
    6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
    publisher_anchor
    openai/o3-2025-04-16
    metric_population
    2
    current_catalog_url
    https://llm-stats.com/
    benchmark_evaluation
    true
    source model id
    o3-2025-04-16
    Original result ↗

    LLM Stats · GitHub archive

    0.158

    source id
    llmStatsArchive
    leaderboard name
    FrontierMath · Normalized (reported)
    dimension
    general
    metric
    FrontierMath · Normalized (reported)
    rank
    1
    aggregation scope
    status
    archived
    license
    CC-BY-4.0
    direction
    higherIsBetter
    conditions
    language
    en
    benchmark
    name
    FrontierMath
    language
    en
    modality
    text
    verified
    false
    max_score
    1
    categories
    • math
    • reasoning
    created_at
    2025-07-19T19:56:15.179213+00:00
    paper_link
    https://arxiv.org/abs/2411.04872
    updated_at
    2025-07-19T19:56:15.179213+00:00
    description
    A benchmark of hundreds of original, exceptionally challenging mathematics problems crafted and vetted by expert mathematicians, covering most major branches of modern mathematics from number theory and real analysis to algebraic geometry and category theory.
    benchmark_id
    frontiermath
    multilingual
    false
    analysis_method
    accuracy
    is_self_reported
    true
    verified_by_llmstats
    false
    attribution
    Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
    license_url
    https://creativecommons.org/licenses/by/4.0/
    metric_kind
    FrontierMath · Normalized (reported)
    rank_method
    competition rank among reported evaluations in identical conditions, including unlinked observations
    source_file
    data/organizations/openai/models/o3-2025-04-16/benchmarks.json
    archive_kind
    llmStats
    evidence_url
    https://www.youtube.com/live/SKBG1sqdyIU?si=lWccKHt8bnttuYta
    git_revision
    872b75f63b8d853c46ed17856424029a0c47b1d2
    published_at
    2025-10-24T17:47:59Z
    snapshot_date
    2025-10-24T17:47:59Z
    source_revision
    6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
    publisher_anchor
    openai/o3-2025-04-16
    metric_population
    1
    current_catalog_url
    https://llm-stats.com/
    benchmark_evaluation
    true
    source model id
    o3-2025-04-16
    Original result ↗

    LLM Stats · GitHub archive

    0.158

    source id
    llmStatsArchive
    leaderboard name
    FrontierMath · Raw
    dimension
    general
    metric
    FrontierMath · Raw
    rank
    1
    aggregation scope
    status
    archived
    license
    CC-BY-4.0
    direction
    higherIsBetter
    conditions
    language
    en
    benchmark
    name
    FrontierMath
    language
    en
    modality
    text
    verified
    false
    max_score
    1
    categories
    • math
    • reasoning
    created_at
    2025-07-19T19:56:15.179213+00:00
    paper_link
    https://arxiv.org/abs/2411.04872
    updated_at
    2025-07-19T19:56:15.179213+00:00
    description
    A benchmark of hundreds of original, exceptionally challenging mathematics problems crafted and vetted by expert mathematicians, covering most major branches of modern mathematics from number theory and real analysis to algebraic geometry and category theory.
    benchmark_id
    frontiermath
    multilingual
    false
    analysis_method
    accuracy
    is_self_reported
    true
    verified_by_llmstats
    false
    attribution
    Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
    license_url
    https://creativecommons.org/licenses/by/4.0/
    metric_kind
    FrontierMath · Raw
    rank_method
    competition rank among reported evaluations in identical conditions, including unlinked observations
    source_file
    data/organizations/openai/models/o3-2025-04-16/benchmarks.json
    archive_kind
    llmStats
    evidence_url
    https://www.youtube.com/live/SKBG1sqdyIU?si=lWccKHt8bnttuYta
    git_revision
    872b75f63b8d853c46ed17856424029a0c47b1d2
    published_at
    2025-10-24T17:47:59Z
    snapshot_date
    2025-10-24T17:47:59Z
    source_revision
    6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
    publisher_anchor
    openai/o3-2025-04-16
    metric_population
    1
    current_catalog_url
    https://llm-stats.com/
    benchmark_evaluation
    true
    source model id
    o3-2025-04-16
    Original result ↗

    Kaggle Benchmarks

    934

    source id
    kaggleBenchmarks
    leaderboard name
    Chess Suite · Text Input With Openings
    dimension
    general
    metric
    Chess Suite · Text Input With Openings
    rank
    2
    aggregation scope
    kaggle
    true
    status
    published
    license
    Apache-2.0
    direction
    higherIsBetter
    conditions
    task
    id
    1869
    name
    Text Input With Openings
    version
    1
    sort_order
    DESCENDING
    display_type
    counts
    benchmark
    id
    86
    name
    Chess Suite
    type
    SUITE
    owner
    kaggle
    version
    2
    categories
      version_id
      437
      maintainer_tier
      Kaggle-published
      candidate
      candidate_class
      modelVersion
      record_kind
      aggregate
      attribution
      Kaggle · kaggle · Chess Suite
      license_url
      https://www.apache.org/licenses/LICENSE-2.0
      metric_kind
      Chess Suite · Text Input With Openings
      rank_method
      competition rank among reported evaluations in identical conditions, including unlinked observations
      source_file
      437/leaderboard.json
      source_page
      https://www.kaggle.com/benchmarks/kaggle/chess-suite/versions/2
      measurements
      • name
        Text Input With Openings
        unit
        score
        scale
        1
        metric
        Chess Suite · Text Input With Openings
        result
        resultCase
        numericResult
        booleanResult
        false
        numericResult
        value
        934
        standardDeviation
        0
        confidenceInterval
        0
        hasStandardDeviation
        false
        hasConfidenceInterval
        false
        unevenConfidenceInterval
        plus
        98
        minus
        95
        hasUnevenConfidenceInterval
        true
        taskVersionId
        0
        evaluationDate
        2026-01-13T00:00:00Z
        hasBooleanResult
        false
        hasNumericResult
        true
        hasTaskVersionId
        false
        hasEvaluationDate
        true
        numericResultNullable
        value
        934
        standardDeviation
        0
        confidenceInterval
        0
        hasStandardDeviation
        false
        hasConfidenceInterval
        false
        unevenConfidenceInterval
        plus
        98
        minus
        95
        hasUnevenConfidenceInterval
        true
        hasNumericResultPublic
        false
        customAdditionalResults
          hasNumericResultPrivate
          false
          task_id
          1869
          task_url
          /benchmarks/kaggle/chess-text-openings/versions/2
          sort_order
          DESCENDING
          source_name
          Text Input With Openings
          display_type
          counts
          task_version
          1
          task_display_type
          COUNTS
          normalization_status
          normalized
        published_at
        2026-02-02T15:46:09.865215700Z
        candidate_class
        modelVersion
        source_revision
        b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
        publisher_anchor
        openai/o3-2025-04-16
        metric_population
        8
        benchmark_evaluation
        true
        task_metadata_source
        benchmarkVersion
        candidate_metadata_source
        benchmarkVersionMappings
        evaluated at
        2026-01-13T00:00:00.000Z
        source model id
        o3
        confidence lower
        839
        confidence upper
        1032
        Original result ↗

        LLM Stats · GitHub archive

        0.916

        source id
        llmStatsArchive
        leaderboard name
        AIME 2024 · Normalized (reported)
        dimension
        general
        metric
        AIME 2024 · Normalized (reported)
        rank
        2
        aggregation scope
        status
        archived
        license
        CC-BY-4.0
        direction
        higherIsBetter
        conditions
        language
        en
        benchmark
        name
        AIME 2024
        language
        en
        modality
        text
        verified
        false
        max_score
        1
        categories
        • math
        • reasoning
        created_at
        2025-07-19T19:56:11.941652+00:00
        paper_link
        https://arxiv.org/html/2503.21380v2
        updated_at
        2025-09-30T00:00:00.000000+00:00
        description
        American Invitational Mathematics Examination 2024, consisting of 30 challenging mathematical reasoning problems from AIME I and AIME II competitions. Each problem requires an integer answer between 0-999 and tests advanced mathematical reasoning across algebra, geometry, combinatorics, and number theory. Used as a benchmark for evaluating mathematical reasoning capabilities in large language models at Olympiad-level difficulty.
        benchmark_id
        aime-2024
        multilingual
        false
        analysis_method
        accuracy (no tools)
        is_self_reported
        true
        verified_by_llmstats
        false
        attribution
        Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
        license_url
        https://creativecommons.org/licenses/by/4.0/
        metric_kind
        AIME 2024 · Normalized (reported)
        rank_method
        competition rank among reported evaluations in identical conditions, including unlinked observations
        source_file
        data/organizations/openai/models/o3-2025-04-16/benchmarks.json
        archive_kind
        llmStats
        evidence_url
        https://openai.com/index/introducing-o3-and-o4-mini/
        git_revision
        872b75f63b8d853c46ed17856424029a0c47b1d2
        published_at
        2025-10-24T17:47:59Z
        snapshot_date
        2025-10-24T17:47:59Z
        source_revision
        6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
        publisher_anchor
        openai/o3-2025-04-16
        metric_population
        2
        current_catalog_url
        https://llm-stats.com/
        benchmark_evaluation
        true
        source model id
        o3-2025-04-16
        Original result ↗

        LLM Stats · GitHub archive

        0.916

        source id
        llmStatsArchive
        leaderboard name
        AIME 2024 · Raw
        dimension
        general
        metric
        AIME 2024 · Raw
        rank
        2
        aggregation scope
        status
        archived
        license
        CC-BY-4.0
        direction
        higherIsBetter
        conditions
        language
        en
        benchmark
        name
        AIME 2024
        language
        en
        modality
        text
        verified
        false
        max_score
        1
        categories
        • math
        • reasoning
        created_at
        2025-07-19T19:56:11.941652+00:00
        paper_link
        https://arxiv.org/html/2503.21380v2
        updated_at
        2025-09-30T00:00:00.000000+00:00
        description
        American Invitational Mathematics Examination 2024, consisting of 30 challenging mathematical reasoning problems from AIME I and AIME II competitions. Each problem requires an integer answer between 0-999 and tests advanced mathematical reasoning across algebra, geometry, combinatorics, and number theory. Used as a benchmark for evaluating mathematical reasoning capabilities in large language models at Olympiad-level difficulty.
        benchmark_id
        aime-2024
        multilingual
        false
        analysis_method
        accuracy (no tools)
        is_self_reported
        true
        verified_by_llmstats
        false
        attribution
        Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
        license_url
        https://creativecommons.org/licenses/by/4.0/
        metric_kind
        AIME 2024 · Raw
        rank_method
        competition rank among reported evaluations in identical conditions, including unlinked observations
        source_file
        data/organizations/openai/models/o3-2025-04-16/benchmarks.json
        archive_kind
        llmStats
        evidence_url
        https://openai.com/index/introducing-o3-and-o4-mini/
        git_revision
        872b75f63b8d853c46ed17856424029a0c47b1d2
        published_at
        2025-10-24T17:47:59Z
        snapshot_date
        2025-10-24T17:47:59Z
        source_revision
        6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
        publisher_anchor
        openai/o3-2025-04-16
        metric_population
        2
        current_catalog_url
        https://llm-stats.com/
        benchmark_evaluation
        true
        source model id
        o3-2025-04-16
        Original result ↗

        Published prices

        1–8 / 8

        openai/o3 ↗

        $0.50

        Inference provider
        openRouter
        Platform
        OpenRouter
        Unit
        cache_read_tokens_per_million
        Currency
        USD
        Billing conditions

        openai/o3 ↗

        $2.00

        Inference provider
        openRouter
        Platform
        OpenRouter
        Unit
        input_tokens_per_million
        Currency
        USD
        Billing conditions

        openai/o3 ↗

        $8.00

        Inference provider
        openRouter
        Platform
        OpenRouter
        Unit
        output_tokens_per_million
        Currency
        USD
        Billing conditions

        openai/o3 ↗

        $0.01

        Inference provider
        openRouter
        Platform
        OpenRouter
        Unit
        web_search
        Currency
        USD
        Billing conditions

        openai/o3:batch ↗

        $0.25

        Inference provider
        openRouter
        Platform
        OpenRouter
        Unit
        cache_read_tokens_per_million
        Currency
        USD
        Billing conditions
        mode
        batch

        openai/o3:batch ↗

        $1.00

        Inference provider
        openRouter
        Platform
        OpenRouter
        Unit
        input_tokens_per_million
        Currency
        USD
        Billing conditions
        mode
        batch

        openai/o3:batch ↗

        $4.00

        Inference provider
        openRouter
        Platform
        OpenRouter
        Unit
        output_tokens_per_million
        Currency
        USD
        Billing conditions
        mode
        batch

        openai/o3:batch ↗

        $0.01

        Inference provider
        openRouter
        Platform
        OpenRouter
        Unit
        web_search
        Currency
        USD
        Billing conditions
        mode
        batch

        Provider specifications & access

        1–1 / 1

        OpenAI o3 · azure-openai v2025-04-16 ↗

        source model id
        azure-openai/o3@2025-04-16
        hosting
        foundry-managed-compute
        platforms
        • Microsoft Foundry
        license
        custom
        context length
        200000
        max output tokens
        100000
        architecture
        input_modalities
        • text
        • image
        output_modalities
        • text
        capabilities
        • reasoning
        • toolCalling

        Safety assessments

        1–1 / 1

        Foundation Model Transparency Index 2025 ↗

        35 / 100

        source id
        stanfordFMTI
        canonical model id
        openai/o3-2025-04-16
        taxonomy
        transparency
        hazard category
        transparency
        hazard name
        Organization transparency
        policy area
        OpenAI
        assessment type
        organization-level disclosure
        metric
        disclosure_score
        result label
        35 / 100
        sample size
        100
        updated at
        2026-11-09T18:08:05.227Z

        Source specification observations

        1–9 / 9

        azureFoundry ↗

        field
        context_length
        value
        200000
        normalized value
        200000
        scope
        source-model
        source id
        azureFoundry
        confidence
        1

        openRouter ↗

        field
        context_length
        value
        200000
        normalized value
        200000
        scope
        source-model
        source id
        openRouter
        confidence
        1

        openRouter ↗

        field
        context_length
        value
        200000
        normalized value
        200000
        scope
        source-model
        source id
        openRouter
        confidence
        1

        openRouter ↗

        field
        knowledge_cutoff
        reported field
        knowledge_cutoff
        value
        2024-06-30
        normalized value
        2024-06-30
        precision
        day
        scope
        source-model
        source id
        openRouter
        confidence
        1

        openRouter ↗

        field
        knowledge_cutoff
        reported field
        knowledge_cutoff
        value
        2024-06-30
        normalized value
        2024-06-30
        precision
        day
        scope
        source-model
        source id
        openRouter
        confidence
        1

        azureFoundry ↗

        field
        max_output_tokens
        value
        100000
        normalized value
        100000
        scope
        source-model
        source id
        azureFoundry
        confidence
        1

        openRouter ↗

        field
        max_output_tokens
        value
        100000
        normalized value
        100000
        scope
        source-model
        source id
        openRouter
        confidence
        1

        openRouter ↗

        field
        max_output_tokens
        value
        100000
        normalized value
        100000
        scope
        source-model
        source id
        openRouter
        confidence
        1

        azureFoundry ↗

        field
        knowledge_cutoff
        reported field
        training_data_cutoff
        value
        May 2024
        normalized value
        2024-05
        precision
        month
        scope
        source-model
        source id
        azureFoundry
        confidence
        1

        Original sources

        1–13 / 13

        OpenAI · Model reference ↗

        Microsoft Foundry · Model page ↗

        OpenRouter · Model page ↗

        OpenRouter · Model page ↗

        Microsoft Foundry · Model documentation ↗

        GitHub · Catalog ↗

        GitHub · Catalog ↗

        Kaggle · Catalog ↗

        LMArena · Catalog ↗

        Microsoft Foundry · Catalog ↗

        OpenRouter · Catalog ↗

        gorilla.cs.berkeley.edu · Catalog ↗

        www.swebench.com · Catalog ↗

        Source links

        1–10 / 10

        Explore more models

        Models

        Loading
        EVIDENCE / COMPARISON

        Benchmarks

        Explore every benchmark

        Loading

        These charts contain model-linked results. The charts above also include unlinked source profiles. Compare scores within each evaluation. Model quality, usage, transparency and system performance measure different things.

        Install TensorTrace