OpenAI: o3
o3 is a well-rounded and powerful model across domains. It sets a new standard for math, science, coding, and visual reasoning tasks. It also excels at technical writing and instruction-following. Us…
openai/o3-2025-04-16
o3 is a well-rounded and powerful model across domains. It sets a new standard for math, science, coding, and visual reasoning tasks. It also excels at technical writing and instruction-following. Use it to think through multi-step problems that involve analysis across text, code, and images.
Specifications & capabilities
- Input
- image, text, file
- Output
- text
- Context window · tokens
- 200000
- Capabilities
- fileInput, multimodal, promptCaching, reasoning, structuredOutputs, toolCalling, visionInput
Description source ↗Evaluations
1–20 / 328
0.868
- source id
- llmStatsArchive
- leaderboard name
- MathVista · Normalized (reported)
- dimension
- general
- metric
- MathVista · Normalized (reported)
- rank
- 1
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- MathVista
- language
- en
- modality
- multimodal
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:12.069611+00:00
- paper_link
- https://arxiv.org/abs/2310.02255
- updated_at
- 2025-07-19T19:56:12.069611+00:00
- description
- MathVista evaluates mathematical reasoning of foundation models in visual contexts. It consists of 6,141 examples derived from 28 existing multimodal datasets and 3 newly created datasets (IQTest, FunctionQA, and PaperQA), combining challenges from diverse mathematical and visual tasks to assess models' ability to understand complex figures and perform rigorous reasoning.
- benchmark_id
- mathvista
- multilingual
- false
- analysis_method
- accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- MathVista · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/o3-2025-04-16/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/introducing-o3-and-o4-mini/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/o3-2025-04-16
- metric_population
- 4
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- o3-2025-04-16
Original result ↗0.868
- source id
- llmStatsArchive
- leaderboard name
- MathVista · Raw
- dimension
- general
- metric
- MathVista · Raw
- rank
- 1
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- MathVista
- language
- en
- modality
- multimodal
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:12.069611+00:00
- paper_link
- https://arxiv.org/abs/2310.02255
- updated_at
- 2025-07-19T19:56:12.069611+00:00
- description
- MathVista evaluates mathematical reasoning of foundation models in visual contexts. It consists of 6,141 examples derived from 28 existing multimodal datasets and 3 newly created datasets (IQTest, FunctionQA, and PaperQA), combining challenges from diverse mathematical and visual tasks to assess models' ability to understand complex figures and perform rigorous reasoning.
- benchmark_id
- mathvista
- multilingual
- false
- analysis_method
- accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- MathVista · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/o3-2025-04-16/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/introducing-o3-and-o4-mini/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/o3-2025-04-16
- metric_population
- 4
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- o3-2025-04-16
Original result ↗0.864
- source id
- llmStatsArchive
- leaderboard name
- AIME 2025 · Raw
- dimension
- general
- metric
- AIME 2025 · Raw
- rank
- 1
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- AIME 2025
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-09-05T00:00:00.000000+00:00
- paper_link
- https://arxiv.org/abs/2503.21380
- updated_at
- 2025-09-05T00:00:00.000000+00:00
- description
- All 30 problems from the 2025 American Invitational Mathematics Examination (AIME I and AIME II), testing olympiad-level mathematical reasoning with integer answers from 000-999. Used as an AI benchmark to evaluate large language models' ability to solve complex mathematical problems requiring multi-step logical deductions and structured symbolic reasoning.
- benchmark_id
- aime-2025
- multilingual
- false
- analysis_method
- pass@1 (no tools)
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- AIME 2025 · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/o3-2025-04-16/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/introducing-o3-and-o4-mini/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/o3-2025-04-16
- metric_population
- 1
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- o3-2025-04-16
Original result ↗0.864
- source id
- llmStatsArchive
- leaderboard name
- AIME 2025 · Normalized (reported)
- dimension
- general
- metric
- AIME 2025 · Normalized (reported)
- rank
- 1
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- AIME 2025
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-09-05T00:00:00.000000+00:00
- paper_link
- https://arxiv.org/abs/2503.21380
- updated_at
- 2025-09-05T00:00:00.000000+00:00
- description
- All 30 problems from the 2025 American Invitational Mathematics Examination (AIME I and AIME II), testing olympiad-level mathematical reasoning with integer answers from 000-999. Used as an AI benchmark to evaluate large language models' ability to solve complex mathematical problems requiring multi-step logical deductions and structured symbolic reasoning.
- benchmark_id
- aime-2025
- multilingual
- false
- analysis_method
- pass@1 (no tools)
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- AIME 2025 · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/o3-2025-04-16/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/introducing-o3-and-o4-mini/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/o3-2025-04-16
- metric_population
- 1
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- o3-2025-04-16
Original result ↗0.813
- source id
- llmStatsArchive
- leaderboard name
- Aider-Polyglot · Raw
- dimension
- general
- metric
- Aider-Polyglot · Raw
- rank
- 1
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- Aider-Polyglot
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-09-05T00:00:00.000000+00:00
- updated_at
- 2025-09-30T00:00:00.000000+00:00
- description
- A coding benchmark that evaluates LLMs on 225 challenging Exercism programming exercises across C++, Go, Java, JavaScript, Python, and Rust. Models receive two attempts to solve each problem, with test error feedback provided after the first attempt if it fails. The benchmark measures both initial problem-solving ability and capacity to edit code based on error feedback, providing an end-to-end evaluation of code generation and editing capabilities across multiple programming languages.
- benchmark_id
- aider-polyglot
- multilingual
- false
- implementation_link
- https://github.com/Aider-AI/polyglot-benchmark
- analysis_method
- accuracy (whole)
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- Aider-Polyglot · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/o3-2025-04-16/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/introducing-o3-and-o4-mini/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/o3-2025-04-16
- metric_population
- 1
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- o3-2025-04-16
Original result ↗0.813
- source id
- llmStatsArchive
- leaderboard name
- Aider-Polyglot · Normalized (reported)
- dimension
- general
- metric
- Aider-Polyglot · Normalized (reported)
- rank
- 1
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- Aider-Polyglot
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-09-05T00:00:00.000000+00:00
- updated_at
- 2025-09-30T00:00:00.000000+00:00
- description
- A coding benchmark that evaluates LLMs on 225 challenging Exercism programming exercises across C++, Go, Java, JavaScript, Python, and Rust. Models receive two attempts to solve each problem, with test error feedback provided after the first attempt if it fails. The benchmark measures both initial problem-solving ability and capacity to edit code based on error feedback, providing an end-to-end evaluation of code generation and editing capabilities across multiple programming languages.
- benchmark_id
- aider-polyglot
- multilingual
- false
- implementation_link
- https://github.com/Aider-AI/polyglot-benchmark
- analysis_method
- accuracy (whole)
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- Aider-Polyglot · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/o3-2025-04-16/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/introducing-o3-and-o4-mini/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/o3-2025-04-16
- metric_population
- 1
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- o3-2025-04-16
Original result ↗0.691
- source id
- llmStatsArchive
- leaderboard name
- SWE-Bench Verified · Normalized (reported)
- dimension
- general
- metric
- SWE-Bench Verified · Normalized (reported)
- rank
- 1
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- SWE-Bench Verified
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- categories
- reasoning
- frontend_development
- code
- created_at
- 2025-07-19T19:56:13.812805+00:00
- paper_link
- https://arxiv.org/abs/2310.06770
- updated_at
- 2025-07-19T19:56:13.812805+00:00
- description
- A verified subset of 500 software engineering problems from real GitHub issues, validated by human annotators for evaluating language models' ability to resolve real-world coding issues by generating patches for Python codebases.
- benchmark_id
- swe-bench-verified
- multilingual
- false
- analysis_method
- accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- SWE-Bench Verified · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/o3-2025-04-16/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/introducing-o3-and-o4-mini/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/o3-2025-04-16
- metric_population
- 2
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- o3-2025-04-16
Original result ↗0.691
- source id
- llmStatsArchive
- leaderboard name
- SWE-Bench Verified · Raw
- dimension
- general
- metric
- SWE-Bench Verified · Raw
- rank
- 1
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- SWE-Bench Verified
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- categories
- reasoning
- frontend_development
- code
- created_at
- 2025-07-19T19:56:13.812805+00:00
- paper_link
- https://arxiv.org/abs/2310.06770
- updated_at
- 2025-07-19T19:56:13.812805+00:00
- description
- A verified subset of 500 software engineering problems from real GitHub issues, validated by human annotators for evaluating language models' ability to resolve real-world coding issues by generating patches for Python codebases.
- benchmark_id
- swe-bench-verified
- multilingual
- false
- analysis_method
- accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- SWE-Bench Verified · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/o3-2025-04-16/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/introducing-o3-and-o4-mini/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/o3-2025-04-16
- metric_population
- 2
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- o3-2025-04-16
Original result ↗0.63
- source id
- llmStatsArchive
- leaderboard name
- Tau-bench · Raw
- dimension
- general
- metric
- Tau-bench · Raw
- rank
- 1
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- Tau-bench
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:15.219001+00:00
- paper_link
- https://arxiv.org/abs/2406.12045
- updated_at
- 2025-07-19T19:56:15.219001+00:00
- description
- τ-bench: A benchmark for tool-agent-user interaction in real-world domains. Tests language agents' ability to interact with users and follow domain-specific rules through dynamic conversations using API tools and policy guidelines across retail and airline domains. Evaluates consistency and reliability of agent behavior over multiple trials.
- benchmark_id
- tau-bench
- multilingual
- false
- analysis_method
- accuracy (avg Airline/Retail)
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- Tau-bench · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/o3-2025-04-16/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/introducing-o3-and-o4-mini/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/o3-2025-04-16
- metric_population
- 1
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- o3-2025-04-16
Original result ↗0.63
- source id
- llmStatsArchive
- leaderboard name
- Tau-bench · Normalized (reported)
- dimension
- general
- metric
- Tau-bench · Normalized (reported)
- rank
- 1
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- Tau-bench
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:15.219001+00:00
- paper_link
- https://arxiv.org/abs/2406.12045
- updated_at
- 2025-07-19T19:56:15.219001+00:00
- description
- τ-bench: A benchmark for tool-agent-user interaction in real-world domains. Tests language agents' ability to interact with users and follow domain-specific rules through dynamic conversations using API tools and policy guidelines across retail and airline domains. Evaluates consistency and reliability of agent behavior over multiple trials.
- benchmark_id
- tau-bench
- multilingual
- false
- analysis_method
- accuracy (avg Airline/Retail)
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- Tau-bench · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/o3-2025-04-16/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/introducing-o3-and-o4-mini/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/o3-2025-04-16
- metric_population
- 1
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- o3-2025-04-16
Original result ↗0.565
- source id
- llmStatsArchive
- leaderboard name
- Scale MultiChallenge · Raw
- dimension
- general
- metric
- Scale MultiChallenge · Raw
- rank
- 1
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- Scale MultiChallenge
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- categories
- reasoning
- communication
- general
- created_at
- 2025-07-19T19:56:15.205789+00:00
- paper_link
- https://arxiv.org/abs/2501.17399
- updated_at
- 2025-07-19T19:56:15.205789+00:00
- description
- MultiChallenge is a realistic multi-turn conversation evaluation benchmark developed by Scale AI that evaluates large language models on four challenging conversation categories: instruction retention, inference memory of user information, reliable versioned editing, and self-coherence. Each challenge requires accurate instruction-following, context allocation, and in-context reasoning. Despite achieving near-perfect scores on existing multi-turn evaluation benchmarks, all frontier models have less than 50% accuracy on MultiChallenge.
- benchmark_id
- scale-multichallenge
- multilingual
- false
- analysis_method
- accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- Scale MultiChallenge · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/o3-2025-04-16/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/introducing-o3-and-o4-mini/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/o3-2025-04-16
- metric_population
- 2
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- o3-2025-04-16
Original result ↗0.565
- source id
- llmStatsArchive
- leaderboard name
- Scale MultiChallenge · Normalized (reported)
- dimension
- general
- metric
- Scale MultiChallenge · Normalized (reported)
- rank
- 1
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- Scale MultiChallenge
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- categories
- reasoning
- communication
- general
- created_at
- 2025-07-19T19:56:15.205789+00:00
- paper_link
- https://arxiv.org/abs/2501.17399
- updated_at
- 2025-07-19T19:56:15.205789+00:00
- description
- MultiChallenge is a realistic multi-turn conversation evaluation benchmark developed by Scale AI that evaluates large language models on four challenging conversation categories: instruction retention, inference memory of user information, reliable versioned editing, and self-coherence. Each challenge requires accurate instruction-following, context allocation, and in-context reasoning. Despite achieving near-perfect scores on existing multi-turn evaluation benchmarks, all frontier models have less than 50% accuracy on MultiChallenge.
- benchmark_id
- scale-multichallenge
- multilingual
- false
- analysis_method
- accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- Scale MultiChallenge · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/o3-2025-04-16/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/introducing-o3-and-o4-mini/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/o3-2025-04-16
- metric_population
- 2
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- o3-2025-04-16
Original result ↗50%
- source id
- kaggleBenchmarks
- leaderboard name
- ITBench · SRE Score
- dimension
- reasoning
- metric
- ITBench · SRE Score
- rank
- 1
- aggregation scope
- kaggle
- true
- status
- published
- license
- Apache-2.0
- direction
- higherIsBetter
- conditions
- task
- id
- 1172
- name
- SRE Score
- version
- 8
- sort_order
- DESCENDING
- display_type
- rawPercentages
- benchmark
- id
- 246
- name
- ITBench
- type
- INDIVIDUAL
- owner
- ibm-research
- version
- 1
- categories
- evaluation
- general knowledge and reasoning
- version_id
- 225
- maintainer_tier
- Kaggle-maintained
- maintenance_level
- KAGGLE_MAINTAINED
- candidate
- candidate_class
- modelVersion
- record_kind
- aggregate
- attribution
- Kaggle · ibm-research · ITBench
- license_url
- https://www.apache.org/licenses/LICENSE-2.0
- metric_kind
- ITBench · SRE Score
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- 225/leaderboard.json
- source_page
- https://www.kaggle.com/benchmarks/ibm-research/itbench/versions/1
- measurements
- name
- SRE Score
- unit
- percent
- scale
- 100
- metric
- ITBench · SRE Score
- result
- resultCase
- numericResult
- booleanResult
- false
- numericResult
- value
- 50
- standardDeviation
- 0
- confidenceInterval
- 0
- hasStandardDeviation
- false
- hasConfidenceInterval
- false
- hasUnevenConfidenceInterval
- false
- taskVersionId
- 0
- evaluationDate
- 2025-12-01T22:27:57.070731Z
- hasBooleanResult
- false
- hasNumericResult
- true
- hasTaskVersionId
- false
- hasEvaluationDate
- true
- numericResultNullable
- value
- 50
- standardDeviation
- 0
- confidenceInterval
- 0
- hasStandardDeviation
- false
- hasConfidenceInterval
- false
- hasUnevenConfidenceInterval
- false
- hasNumericResultPublic
- false
- hasNumericResultPrivate
- false
- task_id
- 1172
- task_url
- /benchmarks/tasks/benchmarkler/itbench-sre-main
- sort_order
- DESCENDING
- source_name
- SRE Score
- display_type
- rawPercentages
- task_version
- 8
- task_display_type
- RAW_VALUES
- normalization_status
- normalized
- published_at
- 2025-12-02T12:36:17.407125500Z
- candidate_class
- modelVersion
- source_revision
- b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
- publisher_anchor
- openai/o3-2025-04-16
- metric_population
- 13
- benchmark_evaluation
- true
- task_metadata_source
- benchmarkVersion
- candidate_metadata_source
- benchmarkVersionMappings
- evaluated at
- 2025-12-01T22:27:57.070Z
- source model id
- o3
Original result ↗0.202
- source id
- llmStatsArchive
- leaderboard name
- Humanity's Last Exam · Raw
- dimension
- general
- metric
- Humanity's Last Exam · Raw
- rank
- 1
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- Humanity's Last Exam
- language
- en
- modality
- multimodal
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:12.507693+00:00
- paper_link
- https://arxiv.org/abs/2501.14249
- updated_at
- 2025-07-19T19:56:12.507693+00:00
- description
- A multi-modal benchmark at the frontier of human knowledge with 2,500 questions across dozens of subjects including mathematics, humanities, and natural sciences, created by nearly 1000 subject expert contributors from over 500 institutions
- benchmark_id
- humanity's-last-exam
- multilingual
- false
- analysis_method
- accuracy (no tools)
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- Humanity's Last Exam · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/o3-2025-04-16/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/introducing-o3-and-o4-mini/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/o3-2025-04-16
- metric_population
- 2
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- o3-2025-04-16
Original result ↗0.202
- source id
- llmStatsArchive
- leaderboard name
- Humanity's Last Exam · Normalized (reported)
- dimension
- general
- metric
- Humanity's Last Exam · Normalized (reported)
- rank
- 1
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- Humanity's Last Exam
- language
- en
- modality
- multimodal
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:12.507693+00:00
- paper_link
- https://arxiv.org/abs/2501.14249
- updated_at
- 2025-07-19T19:56:12.507693+00:00
- description
- A multi-modal benchmark at the frontier of human knowledge with 2,500 questions across dozens of subjects including mathematics, humanities, and natural sciences, created by nearly 1000 subject expert contributors from over 500 institutions
- benchmark_id
- humanity's-last-exam
- multilingual
- false
- analysis_method
- accuracy (no tools)
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- Humanity's Last Exam · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/o3-2025-04-16/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/introducing-o3-and-o4-mini/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/o3-2025-04-16
- metric_population
- 2
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- o3-2025-04-16
Original result ↗0.158
- source id
- llmStatsArchive
- leaderboard name
- FrontierMath · Normalized (reported)
- dimension
- general
- metric
- FrontierMath · Normalized (reported)
- rank
- 1
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- FrontierMath
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:15.179213+00:00
- paper_link
- https://arxiv.org/abs/2411.04872
- updated_at
- 2025-07-19T19:56:15.179213+00:00
- description
- A benchmark of hundreds of original, exceptionally challenging mathematics problems crafted and vetted by expert mathematicians, covering most major branches of modern mathematics from number theory and real analysis to algebraic geometry and category theory.
- benchmark_id
- frontiermath
- multilingual
- false
- analysis_method
- accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- FrontierMath · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/o3-2025-04-16/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://www.youtube.com/live/SKBG1sqdyIU?si=lWccKHt8bnttuYta
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/o3-2025-04-16
- metric_population
- 1
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- o3-2025-04-16
Original result ↗0.158
- source id
- llmStatsArchive
- leaderboard name
- FrontierMath · Raw
- dimension
- general
- metric
- FrontierMath · Raw
- rank
- 1
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- FrontierMath
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:15.179213+00:00
- paper_link
- https://arxiv.org/abs/2411.04872
- updated_at
- 2025-07-19T19:56:15.179213+00:00
- description
- A benchmark of hundreds of original, exceptionally challenging mathematics problems crafted and vetted by expert mathematicians, covering most major branches of modern mathematics from number theory and real analysis to algebraic geometry and category theory.
- benchmark_id
- frontiermath
- multilingual
- false
- analysis_method
- accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- FrontierMath · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/o3-2025-04-16/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://www.youtube.com/live/SKBG1sqdyIU?si=lWccKHt8bnttuYta
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/o3-2025-04-16
- metric_population
- 1
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- o3-2025-04-16
Original result ↗934
- source id
- kaggleBenchmarks
- leaderboard name
- Chess Suite · Text Input With Openings
- dimension
- general
- metric
- Chess Suite · Text Input With Openings
- rank
- 2
- aggregation scope
- kaggle
- true
- status
- published
- license
- Apache-2.0
- direction
- higherIsBetter
- conditions
- task
- id
- 1869
- name
- Text Input With Openings
- version
- 1
- sort_order
- DESCENDING
- display_type
- counts
- benchmark
- id
- 86
- name
- Chess Suite
- type
- SUITE
- owner
- kaggle
- version
- 2
- version_id
- 437
- maintainer_tier
- Kaggle-published
- candidate
- candidate_class
- modelVersion
- record_kind
- aggregate
- attribution
- Kaggle · kaggle · Chess Suite
- license_url
- https://www.apache.org/licenses/LICENSE-2.0
- metric_kind
- Chess Suite · Text Input With Openings
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- 437/leaderboard.json
- source_page
- https://www.kaggle.com/benchmarks/kaggle/chess-suite/versions/2
- measurements
- name
- Text Input With Openings
- unit
- score
- scale
- 1
- metric
- Chess Suite · Text Input With Openings
- result
- resultCase
- numericResult
- booleanResult
- false
- numericResult
- value
- 934
- standardDeviation
- 0
- confidenceInterval
- 0
- hasStandardDeviation
- false
- hasConfidenceInterval
- false
- unevenConfidenceInterval
- plus
- 98
- minus
- 95
- hasUnevenConfidenceInterval
- true
- taskVersionId
- 0
- evaluationDate
- 2026-01-13T00:00:00Z
- hasBooleanResult
- false
- hasNumericResult
- true
- hasTaskVersionId
- false
- hasEvaluationDate
- true
- numericResultNullable
- value
- 934
- standardDeviation
- 0
- confidenceInterval
- 0
- hasStandardDeviation
- false
- hasConfidenceInterval
- false
- unevenConfidenceInterval
- plus
- 98
- minus
- 95
- hasUnevenConfidenceInterval
- true
- hasNumericResultPublic
- false
- hasNumericResultPrivate
- false
- task_id
- 1869
- task_url
- /benchmarks/kaggle/chess-text-openings/versions/2
- sort_order
- DESCENDING
- source_name
- Text Input With Openings
- display_type
- counts
- task_version
- 1
- task_display_type
- COUNTS
- normalization_status
- normalized
- published_at
- 2026-02-02T15:46:09.865215700Z
- candidate_class
- modelVersion
- source_revision
- b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
- publisher_anchor
- openai/o3-2025-04-16
- metric_population
- 8
- benchmark_evaluation
- true
- task_metadata_source
- benchmarkVersion
- candidate_metadata_source
- benchmarkVersionMappings
- evaluated at
- 2026-01-13T00:00:00.000Z
- source model id
- o3
- confidence lower
- 839
- confidence upper
- 1032
Original result ↗0.916
- source id
- llmStatsArchive
- leaderboard name
- AIME 2024 · Normalized (reported)
- dimension
- general
- metric
- AIME 2024 · Normalized (reported)
- rank
- 2
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- AIME 2024
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:11.941652+00:00
- paper_link
- https://arxiv.org/html/2503.21380v2
- updated_at
- 2025-09-30T00:00:00.000000+00:00
- description
- American Invitational Mathematics Examination 2024, consisting of 30 challenging mathematical reasoning problems from AIME I and AIME II competitions. Each problem requires an integer answer between 0-999 and tests advanced mathematical reasoning across algebra, geometry, combinatorics, and number theory. Used as a benchmark for evaluating mathematical reasoning capabilities in large language models at Olympiad-level difficulty.
- benchmark_id
- aime-2024
- multilingual
- false
- analysis_method
- accuracy (no tools)
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- AIME 2024 · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/o3-2025-04-16/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/introducing-o3-and-o4-mini/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/o3-2025-04-16
- metric_population
- 2
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- o3-2025-04-16
Original result ↗0.916
- source id
- llmStatsArchive
- leaderboard name
- AIME 2024 · Raw
- dimension
- general
- metric
- AIME 2024 · Raw
- rank
- 2
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- AIME 2024
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:11.941652+00:00
- paper_link
- https://arxiv.org/html/2503.21380v2
- updated_at
- 2025-09-30T00:00:00.000000+00:00
- description
- American Invitational Mathematics Examination 2024, consisting of 30 challenging mathematical reasoning problems from AIME I and AIME II competitions. Each problem requires an integer answer between 0-999 and tests advanced mathematical reasoning across algebra, geometry, combinatorics, and number theory. Used as a benchmark for evaluating mathematical reasoning capabilities in large language models at Olympiad-level difficulty.
- benchmark_id
- aime-2024
- multilingual
- false
- analysis_method
- accuracy (no tools)
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- AIME 2024 · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/openai/models/o3-2025-04-16/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://openai.com/index/introducing-o3-and-o4-mini/
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- openai/o3-2025-04-16
- metric_population
- 2
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- o3-2025-04-16
Original result ↗Published prices
1–8 / 8
$0.50
- Inference provider
- openRouter
- Platform
- OpenRouter
- Unit
- cache_read_tokens_per_million
- Currency
- USD
$2.00
- Inference provider
- openRouter
- Platform
- OpenRouter
- Unit
- input_tokens_per_million
- Currency
- USD
$8.00
- Inference provider
- openRouter
- Platform
- OpenRouter
- Unit
- output_tokens_per_million
- Currency
- USD
$0.01
- Inference provider
- openRouter
- Platform
- OpenRouter
- Unit
- web_search
- Currency
- USD
$0.25
- Inference provider
- openRouter
- Platform
- OpenRouter
- Unit
- cache_read_tokens_per_million
- Currency
- USD
- Billing conditions
- mode
- batch
$1.00
- Inference provider
- openRouter
- Platform
- OpenRouter
- Unit
- input_tokens_per_million
- Currency
- USD
- Billing conditions
- mode
- batch
$4.00
- Inference provider
- openRouter
- Platform
- OpenRouter
- Unit
- output_tokens_per_million
- Currency
- USD
- Billing conditions
- mode
- batch
$0.01
- Inference provider
- openRouter
- Platform
- OpenRouter
- Unit
- web_search
- Currency
- USD
- Billing conditions
- mode
- batch
Provider specifications & access
1–1 / 1
- source model id
- azure-openai/o3@2025-04-16
- hosting
- foundry-managed-compute
- license
- custom
- context length
- 200000
- max output tokens
- 100000
Safety assessments
1–1 / 1
35 / 100
- source id
- stanfordFMTI
- canonical model id
- openai/o3-2025-04-16
- taxonomy
- transparency
- hazard category
- transparency
- hazard name
- Organization transparency
- policy area
- OpenAI
- assessment type
- organization-level disclosure
- metric
- disclosure_score
- result label
- 35 / 100
- sample size
- 100
- updated at
- 2026-11-09T18:08:05.227Z
Source specification observations
1–9 / 9
- field
- context_length
- value
- 200000
- normalized value
- 200000
- scope
- source-model
- source id
- azureFoundry
- confidence
- 1
- field
- context_length
- value
- 200000
- normalized value
- 200000
- scope
- source-model
- source id
- openRouter
- confidence
- 1
- field
- context_length
- value
- 200000
- normalized value
- 200000
- scope
- source-model
- source id
- openRouter
- confidence
- 1
- field
- knowledge_cutoff
- reported field
- knowledge_cutoff
- value
- 2024-06-30
- normalized value
- 2024-06-30
- precision
- day
- scope
- source-model
- source id
- openRouter
- confidence
- 1
- field
- knowledge_cutoff
- reported field
- knowledge_cutoff
- value
- 2024-06-30
- normalized value
- 2024-06-30
- precision
- day
- scope
- source-model
- source id
- openRouter
- confidence
- 1
- field
- max_output_tokens
- value
- 100000
- normalized value
- 100000
- scope
- source-model
- source id
- azureFoundry
- confidence
- 1
- field
- max_output_tokens
- value
- 100000
- normalized value
- 100000
- scope
- source-model
- source id
- openRouter
- confidence
- 1
- field
- max_output_tokens
- value
- 100000
- normalized value
- 100000
- scope
- source-model
- source id
- openRouter
- confidence
- 1
- field
- knowledge_cutoff
- reported field
- training_data_cutoff
- value
- May 2024
- normalized value
- 2024-05
- precision
- month
- scope
- source-model
- source id
- azureFoundry
- confidence
- 1
Explore more models