OpenAI: o1

The latest and strongest model family from OpenAI, o1 is designed to spend more time thinking before responding. The o1 model series is trained with large-scale reinforcement learning to reason...

openai/o1-2024-12-17

The latest and strongest model family from OpenAI, o1 is designed to spend more time thinking before responding. The o1 model series is trained with large-scale reinforcement learning to reason...

Specifications & capabilities

Input
text, image, file
Output
text
Context window · tokens
200000
Knowledge cutoff
2023-09
Capabilities
multimodal, reasoning, toolCalling, visionInput

Evaluations

21–40 / 200

LLM Stats · GitHub archive

0.893

source id
llmStatsArchive
leaderboard name
MGSM · Normalized (reported)
dimension
general
metric
MGSM · Normalized (reported)
rank
2
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
MGSM
language
en
modality
text
verified
false
max_score
1
categories
  • math
  • reasoning
created_at
2025-07-19T19:56:13.669061+00:00
paper_link
https://arxiv.org/abs/2210.03057
updated_at
2025-07-19T19:56:13.669061+00:00
description
MGSM (Multilingual Grade School Math) is a benchmark of grade-school math problems. Contains 250 grade-school math problems manually translated from the GSM8K dataset into ten typologically diverse languages: Spanish, French, German, Russian, Chinese, Japanese, Thai, Swahili, Bengali, and Telugu. Evaluates multilingual mathematical reasoning capabilities.
benchmark_id
mgsm
multilingual
true
analysis_method
pass@1
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
MGSM · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/o1-2024-12-17/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/o1-and-new-tools-for-developers/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/o1-2024-12-17
metric_population
2
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
o1-2024-12-17
Original result ↗

LLM Stats · GitHub archive

0.893

source id
llmStatsArchive
leaderboard name
MGSM · Raw
dimension
general
metric
MGSM · Raw
rank
2
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
MGSM
language
en
modality
text
verified
false
max_score
1
categories
  • math
  • reasoning
created_at
2025-07-19T19:56:13.669061+00:00
paper_link
https://arxiv.org/abs/2210.03057
updated_at
2025-07-19T19:56:13.669061+00:00
description
MGSM (Multilingual Grade School Math) is a benchmark of grade-school math problems. Contains 250 grade-school math problems manually translated from the GSM8K dataset into ten typologically diverse languages: Spanish, French, German, Russian, Chinese, Japanese, Thai, Swahili, Bengali, and Telugu. Evaluates multilingual mathematical reasoning capabilities.
benchmark_id
mgsm
multilingual
true
analysis_method
pass@1
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
MGSM · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/o1-2024-12-17/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/o1-and-new-tools-for-developers/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/o1-2024-12-17
metric_population
2
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
o1-2024-12-17
Original result ↗

LLM Stats · GitHub archive

0.47

source id
llmStatsArchive
leaderboard name
SimpleQA · Normalized (reported)
dimension
general
metric
SimpleQA · Normalized (reported)
rank
3
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
SimpleQA
language
en
modality
text
verified
false
max_score
1
categories
  • general
  • reasoning
created_at
2025-09-05T00:00:00.000000+00:00
paper_link
https://arxiv.org/abs/2411.04368
updated_at
2025-09-05T00:00:00.000000+00:00
description
SimpleQA is a factuality benchmark developed by OpenAI that measures the short-form factual accuracy of large language models. The benchmark contains 4,326 short, fact-seeking questions that are adversarially collected and designed to have single, indisputable answers. Questions cover diverse topics from science and technology to entertainment, and the benchmark also measures model calibration by evaluating whether models know what they know.
benchmark_id
simpleqa
multilingual
false
analysis_method
accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
SimpleQA · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/o1-2024-12-17/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/introducing-gpt-4-5/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/o1-2024-12-17
metric_population
5
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
o1-2024-12-17
Original result ↗

LLM Stats · GitHub archive

0.47

source id
llmStatsArchive
leaderboard name
SimpleQA · Raw
dimension
general
metric
SimpleQA · Raw
rank
3
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
SimpleQA
language
en
modality
text
verified
false
max_score
1
categories
  • general
  • reasoning
created_at
2025-09-05T00:00:00.000000+00:00
paper_link
https://arxiv.org/abs/2411.04368
updated_at
2025-09-05T00:00:00.000000+00:00
description
SimpleQA is a factuality benchmark developed by OpenAI that measures the short-form factual accuracy of large language models. The benchmark contains 4,326 short, fact-seeking questions that are adversarially collected and designed to have single, indisputable answers. Questions cover diverse topics from science and technology to entertainment, and the benchmark also measures model calibration by evaluating whether models know what they know.
benchmark_id
simpleqa
multilingual
false
analysis_method
accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
SimpleQA · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/o1-2024-12-17/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/introducing-gpt-4-5/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/o1-2024-12-17
metric_population
5
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
o1-2024-12-17
Original result ↗

LLM Stats · GitHub archive

0.881

source id
llmStatsArchive
leaderboard name
HumanEval · Raw
dimension
general
metric
HumanEval · Raw
rank
4
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
HumanEval
language
en
modality
text
verified
false
max_score
1
categories
  • reasoning
  • code
created_at
2025-07-19T19:56:12.595263+00:00
paper_link
https://arxiv.org/abs/2107.03374
updated_at
2025-07-19T19:56:12.595263+00:00
description
A benchmark that measures functional correctness for synthesizing programs from docstrings, consisting of 164 original programming problems assessing language comprehension, algorithms, and simple mathematics
benchmark_id
humaneval
multilingual
false
analysis_method
pass@1
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
HumanEval · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/o1-2024-12-17/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/learning-to-reason-with-llms/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/o1-2024-12-17
metric_population
10
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
o1-2024-12-17
Original result ↗

LLM Stats · GitHub archive

0.881

source id
llmStatsArchive
leaderboard name
HumanEval · Normalized (reported)
dimension
general
metric
HumanEval · Normalized (reported)
rank
4
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
HumanEval
language
en
modality
text
verified
false
max_score
1
categories
  • reasoning
  • code
created_at
2025-07-19T19:56:12.595263+00:00
paper_link
https://arxiv.org/abs/2107.03374
updated_at
2025-07-19T19:56:12.595263+00:00
description
A benchmark that measures functional correctness for synthesizing programs from docstrings, consisting of 164 original programming problems assessing language comprehension, algorithms, and simple mathematics
benchmark_id
humaneval
multilingual
false
analysis_method
pass@1
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
HumanEval · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/o1-2024-12-17/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/learning-to-reason-with-llms/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/o1-2024-12-17
metric_population
10
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
o1-2024-12-17
Original result ↗

LLM Stats · GitHub archive

0.743

source id
llmStatsArchive
leaderboard name
AIME 2024 · Normalized (reported)
dimension
general
metric
AIME 2024 · Normalized (reported)
rank
4
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
AIME 2024
language
en
modality
text
verified
false
max_score
1
categories
  • math
  • reasoning
created_at
2025-07-19T19:56:11.941652+00:00
paper_link
https://arxiv.org/html/2503.21380v2
updated_at
2025-09-30T00:00:00.000000+00:00
description
American Invitational Mathematics Examination 2024, consisting of 30 challenging mathematical reasoning problems from AIME I and AIME II competitions. Each problem requires an integer answer between 0-999 and tests advanced mathematical reasoning across algebra, geometry, combinatorics, and number theory. Used as a benchmark for evaluating mathematical reasoning capabilities in large language models at Olympiad-level difficulty.
benchmark_id
aime-2024
multilingual
false
analysis_method
accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
AIME 2024 · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/o1-2024-12-17/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/gpt-4-1/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/o1-2024-12-17
metric_population
5
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
o1-2024-12-17
Original result ↗

LLM Stats · GitHub archive

0.743

source id
llmStatsArchive
leaderboard name
AIME 2024 · Raw
dimension
general
metric
AIME 2024 · Raw
rank
4
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
AIME 2024
language
en
modality
text
verified
false
max_score
1
categories
  • math
  • reasoning
created_at
2025-07-19T19:56:11.941652+00:00
paper_link
https://arxiv.org/html/2503.21380v2
updated_at
2025-09-30T00:00:00.000000+00:00
description
American Invitational Mathematics Examination 2024, consisting of 30 challenging mathematical reasoning problems from AIME I and AIME II competitions. Each problem requires an integer answer between 0-999 and tests advanced mathematical reasoning across algebra, geometry, combinatorics, and number theory. Used as a benchmark for evaluating mathematical reasoning capabilities in large language models at Olympiad-level difficulty.
benchmark_id
aime-2024
multilingual
false
analysis_method
accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
AIME 2024 · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/openai/models/o1-2024-12-17/benchmarks.json
archive_kind
llmStats
evidence_url
https://openai.com/index/gpt-4-1/
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
openai/o1-2024-12-17
metric_population
5
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
o1-2024-12-17
Original result ↗

Kaggle Benchmarks

66.4%

source id
kaggleBenchmarks
leaderboard name
MultiLoKo English · EM English
dimension
reasoning
metric
MultiLoKo English · EM English
rank
4
aggregation scope
kaggle
true
status
published
license
Apache-2.0
direction
higherIsBetter
conditions
task
id
27
name
EM English
version
1
sort_order
DESCENDING
display_type
percentages
benchmark
id
52
name
MultiLoKo English
type
INDIVIDUAL
owner
metaresearch
version
1
categories
  • multilingual and cross-lingual capabilities
  • general knowledge and reasoning
version_id
38
maintainer_tier
Kaggle-maintained
maintenance_level
KAGGLE_MAINTAINED
candidate
candidate_class
modelVersion
record_kind
aggregate
attribution
Kaggle · metaresearch · MultiLoKo English
license_url
https://www.apache.org/licenses/LICENSE-2.0
metric_kind
MultiLoKo English · EM English
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
38/leaderboard.json
source_page
https://www.kaggle.com/benchmarks/metaresearch/multiloko-english/versions/1
measurements
  • name
    EM English
    unit
    percent
    scale
    1
    metric
    MultiLoKo English · EM English
    result
    resultCase
    numericResult
    booleanResult
    false
    numericResult
    value
    0.664
    standardDeviation
    0
    confidenceInterval
    0.0585506559735121
    hasStandardDeviation
    false
    hasConfidenceInterval
    true
    confidenceIntervalNullable
    0.0585506559735121
    hasUnevenConfidenceInterval
    false
    taskVersionId
    0
    evaluationDate
    2025-06-16T00:00:00Z
    hasBooleanResult
    false
    hasNumericResult
    true
    hasTaskVersionId
    false
    hasEvaluationDate
    true
    numericResultNullable
    value
    0.664
    standardDeviation
    0
    confidenceInterval
    0.0585506559735121
    hasStandardDeviation
    false
    hasConfidenceInterval
    true
    confidenceIntervalNullable
    0.0585506559735121
    hasUnevenConfidenceInterval
    false
    hasNumericResultPublic
    false
    customAdditionalResults
      hasNumericResultPrivate
      false
      task_id
      27
      sort_order
      DESCENDING
      source_name
      EM English
      display_type
      percentages
      task_version
      1
      task_display_type
      PERCENTAGES
      normalization_status
      normalized
    published_at
    2025-07-15T11:49:31.790Z
    candidate_class
    modelVersion
    source_revision
    b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
    publisher_anchor
    openai/o1-2024-12-17
    metric_population
    51
    benchmark_evaluation
    true
    task_metadata_source
    benchmarkVersion
    candidate_metadata_source
    benchmarkVersionMappings
    evaluated at
    2025-06-16T00:00:00.000Z
    source model id
    o1
    confidence lower
    0.605449344026488
    confidence upper
    0.7225506559735121
    Original result ↗

    LiveBench

    64.755%

    source id
    liveBench
    leaderboard name
    LiveBench Language
    dimension
    intelligence
    metric
    Average score
    rank
    4
    sample size
    140
    evaluated at
    2025-03-04T16:25:32.323Z
    source model id
    o1-2024-12-17-medium
    Original result ↗

    Kaggle Benchmarks

    59.2%

    source id
    kaggleBenchmarks
    leaderboard name
    MultiLoKo Portuguese · EM Portuguese
    dimension
    reasoning
    metric
    MultiLoKo Portuguese · EM Portuguese
    rank
    4
    aggregation scope
    kaggle
    true
    status
    published
    license
    Apache-2.0
    direction
    higherIsBetter
    conditions
    task
    id
    48
    name
    EM Portuguese
    version
    1
    sort_order
    DESCENDING
    display_type
    percentages
    benchmark
    id
    71
    name
    MultiLoKo Portuguese
    type
    INDIVIDUAL
    owner
    metaresearch
    version
    1
    categories
    • multilingual and cross-lingual capabilities
    • general knowledge and reasoning
    version_id
    57
    maintainer_tier
    Kaggle-maintained
    maintenance_level
    KAGGLE_MAINTAINED
    candidate
    candidate_class
    modelVersion
    record_kind
    aggregate
    attribution
    Kaggle · metaresearch · MultiLoKo Portuguese
    license_url
    https://www.apache.org/licenses/LICENSE-2.0
    metric_kind
    MultiLoKo Portuguese · EM Portuguese
    rank_method
    competition rank among reported evaluations in identical conditions, including unlinked observations
    source_file
    57/leaderboard.json
    source_page
    https://www.kaggle.com/benchmarks/metaresearch/multiloko-portuguese/versions/1
    measurements
    • name
      EM Portuguese
      unit
      percent
      scale
      1
      metric
      MultiLoKo Portuguese · EM Portuguese
      result
      resultCase
      numericResult
      booleanResult
      false
      numericResult
      value
      0.592
      standardDeviation
      0
      confidenceInterval
      0.0609212802792317
      hasStandardDeviation
      false
      hasConfidenceInterval
      true
      confidenceIntervalNullable
      0.0609212802792317
      hasUnevenConfidenceInterval
      false
      taskVersionId
      0
      evaluationDate
      2025-06-16T00:00:00Z
      hasBooleanResult
      false
      hasNumericResult
      true
      hasTaskVersionId
      false
      hasEvaluationDate
      true
      numericResultNullable
      value
      0.592
      standardDeviation
      0
      confidenceInterval
      0.0609212802792317
      hasStandardDeviation
      false
      hasConfidenceInterval
      true
      confidenceIntervalNullable
      0.0609212802792317
      hasUnevenConfidenceInterval
      false
      hasNumericResultPublic
      false
      customAdditionalResults
        hasNumericResultPrivate
        false
        task_id
        48
        sort_order
        DESCENDING
        source_name
        EM Portuguese
        display_type
        percentages
        task_version
        1
        task_display_type
        PERCENTAGES
        normalization_status
        normalized
      published_at
      2025-07-15T11:49:31.790Z
      candidate_class
      modelVersion
      source_revision
      b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
      publisher_anchor
      openai/o1-2024-12-17
      metric_population
      51
      benchmark_evaluation
      true
      task_metadata_source
      benchmarkVersion
      candidate_metadata_source
      benchmarkVersionMappings
      evaluated at
      2025-06-16T00:00:00.000Z
      source model id
      o1
      confidence lower
      0.5310787197207683
      confidence upper
      0.6529212802792317
      Original result ↗

      Kaggle Benchmarks

      54.8%

      source id
      kaggleBenchmarks
      leaderboard name
      MultiLoKo Spanish · EM Spanish
      dimension
      reasoning
      metric
      MultiLoKo Spanish · EM Spanish
      rank
      4
      aggregation scope
      kaggle
      true
      status
      published
      license
      Apache-2.0
      direction
      higherIsBetter
      conditions
      task
      id
      52
      name
      EM Spanish
      version
      1
      sort_order
      DESCENDING
      display_type
      percentages
      benchmark
      id
      75
      name
      MultiLoKo Spanish
      type
      INDIVIDUAL
      owner
      metaresearch
      version
      1
      categories
      • multilingual and cross-lingual capabilities
      • general knowledge and reasoning
      version_id
      61
      maintainer_tier
      Kaggle-maintained
      maintenance_level
      KAGGLE_MAINTAINED
      candidate
      candidate_class
      modelVersion
      record_kind
      aggregate
      attribution
      Kaggle · metaresearch · MultiLoKo Spanish
      license_url
      https://www.apache.org/licenses/LICENSE-2.0
      metric_kind
      MultiLoKo Spanish · EM Spanish
      rank_method
      competition rank among reported evaluations in identical conditions, including unlinked observations
      source_file
      61/leaderboard.json
      source_page
      https://www.kaggle.com/benchmarks/metaresearch/multiloko-spanish/versions/1
      measurements
      • name
        EM Spanish
        unit
        percent
        scale
        1
        metric
        MultiLoKo Spanish · EM Spanish
        result
        resultCase
        numericResult
        booleanResult
        false
        numericResult
        value
        0.548
        standardDeviation
        0
        confidenceInterval
        0.0616932406038344
        hasStandardDeviation
        false
        hasConfidenceInterval
        true
        confidenceIntervalNullable
        0.0616932406038344
        hasUnevenConfidenceInterval
        false
        taskVersionId
        0
        evaluationDate
        2025-06-16T00:00:00Z
        hasBooleanResult
        false
        hasNumericResult
        true
        hasTaskVersionId
        false
        hasEvaluationDate
        true
        numericResultNullable
        value
        0.548
        standardDeviation
        0
        confidenceInterval
        0.0616932406038344
        hasStandardDeviation
        false
        hasConfidenceInterval
        true
        confidenceIntervalNullable
        0.0616932406038344
        hasUnevenConfidenceInterval
        false
        hasNumericResultPublic
        false
        customAdditionalResults
          hasNumericResultPrivate
          false
          task_id
          52
          sort_order
          DESCENDING
          source_name
          EM Spanish
          display_type
          percentages
          task_version
          1
          task_display_type
          PERCENTAGES
          normalization_status
          normalized
        published_at
        2025-07-15T11:49:31.790Z
        candidate_class
        modelVersion
        source_revision
        b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
        publisher_anchor
        openai/o1-2024-12-17
        metric_population
        51
        benchmark_evaluation
        true
        task_metadata_source
        benchmarkVersion
        candidate_metadata_source
        benchmarkVersionMappings
        evaluated at
        2025-06-16T00:00:00.000Z
        source model id
        o1
        confidence lower
        0.48630675939616563
        confidence upper
        0.6096932406038345
        Original result ↗

        Kaggle Benchmarks

        47.2%

        source id
        kaggleBenchmarks
        leaderboard name
        MultiLoKo Romanian · EM Romanian
        dimension
        reasoning
        metric
        MultiLoKo Romanian · EM Romanian
        rank
        4
        aggregation scope
        kaggle
        true
        status
        published
        license
        Apache-2.0
        direction
        higherIsBetter
        conditions
        task
        id
        49
        name
        EM Romanian
        version
        1
        sort_order
        DESCENDING
        display_type
        percentages
        benchmark
        id
        72
        name
        MultiLoKo Romanian
        type
        INDIVIDUAL
        owner
        metaresearch
        version
        1
        categories
        • multilingual and cross-lingual capabilities
        • general knowledge and reasoning
        version_id
        58
        maintainer_tier
        Kaggle-maintained
        maintenance_level
        KAGGLE_MAINTAINED
        candidate
        candidate_class
        modelVersion
        record_kind
        aggregate
        attribution
        Kaggle · metaresearch · MultiLoKo Romanian
        license_url
        https://www.apache.org/licenses/LICENSE-2.0
        metric_kind
        MultiLoKo Romanian · EM Romanian
        rank_method
        competition rank among reported evaluations in identical conditions, including unlinked observations
        source_file
        58/leaderboard.json
        source_page
        https://www.kaggle.com/benchmarks/metaresearch/multiloko-romanian/versions/1
        measurements
        • name
          EM Romanian
          unit
          percent
          scale
          1
          metric
          MultiLoKo Romanian · EM Romanian
          result
          resultCase
          numericResult
          booleanResult
          false
          numericResult
          value
          0.472
          standardDeviation
          0
          confidenceInterval
          0.0618822430575397
          hasStandardDeviation
          false
          hasConfidenceInterval
          true
          confidenceIntervalNullable
          0.0618822430575397
          hasUnevenConfidenceInterval
          false
          taskVersionId
          0
          evaluationDate
          2025-06-16T00:00:00Z
          hasBooleanResult
          false
          hasNumericResult
          true
          hasTaskVersionId
          false
          hasEvaluationDate
          true
          numericResultNullable
          value
          0.472
          standardDeviation
          0
          confidenceInterval
          0.0618822430575397
          hasStandardDeviation
          false
          hasConfidenceInterval
          true
          confidenceIntervalNullable
          0.0618822430575397
          hasUnevenConfidenceInterval
          false
          hasNumericResultPublic
          false
          customAdditionalResults
            hasNumericResultPrivate
            false
            task_id
            49
            sort_order
            DESCENDING
            source_name
            EM Romanian
            display_type
            percentages
            task_version
            1
            task_display_type
            PERCENTAGES
            normalization_status
            normalized
          published_at
          2025-07-15T11:49:31.790Z
          candidate_class
          modelVersion
          source_revision
          b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
          publisher_anchor
          openai/o1-2024-12-17
          metric_population
          51
          benchmark_evaluation
          true
          task_metadata_source
          benchmarkVersion
          candidate_metadata_source
          benchmarkVersionMappings
          evaluated at
          2025-06-16T00:00:00.000Z
          source model id
          o1
          confidence lower
          0.4101177569424603
          confidence upper
          0.5338822430575396
          Original result ↗

          Kaggle Benchmarks

          44.8%

          source id
          kaggleBenchmarks
          leaderboard name
          MultiLoKo Swedish · EM Swedish
          dimension
          reasoning
          metric
          MultiLoKo Swedish · EM Swedish
          rank
          4
          aggregation scope
          kaggle
          true
          status
          published
          license
          Apache-2.0
          direction
          higherIsBetter
          conditions
          task
          id
          53
          name
          EM Swedish
          version
          1
          sort_order
          DESCENDING
          display_type
          percentages
          benchmark
          id
          76
          name
          MultiLoKo Swedish
          type
          INDIVIDUAL
          owner
          metaresearch
          version
          1
          categories
          • multilingual and cross-lingual capabilities
          • general knowledge and reasoning
          version_id
          62
          maintainer_tier
          Kaggle-maintained
          maintenance_level
          KAGGLE_MAINTAINED
          candidate
          candidate_class
          modelVersion
          record_kind
          aggregate
          attribution
          Kaggle · metaresearch · MultiLoKo Swedish
          license_url
          https://www.apache.org/licenses/LICENSE-2.0
          metric_kind
          MultiLoKo Swedish · EM Swedish
          rank_method
          competition rank among reported evaluations in identical conditions, including unlinked observations
          source_file
          62/leaderboard.json
          source_page
          https://www.kaggle.com/benchmarks/metaresearch/multiloko-swedish/versions/1
          measurements
          • name
            EM Swedish
            unit
            percent
            scale
            1
            metric
            MultiLoKo Swedish · EM Swedish
            result
            resultCase
            numericResult
            booleanResult
            false
            numericResult
            value
            0.448
            standardDeviation
            0
            confidenceInterval
            0.0616434068014536
            hasStandardDeviation
            false
            hasConfidenceInterval
            true
            confidenceIntervalNullable
            0.0616434068014536
            hasUnevenConfidenceInterval
            false
            taskVersionId
            0
            evaluationDate
            2025-06-16T00:00:00Z
            hasBooleanResult
            false
            hasNumericResult
            true
            hasTaskVersionId
            false
            hasEvaluationDate
            true
            numericResultNullable
            value
            0.448
            standardDeviation
            0
            confidenceInterval
            0.0616434068014536
            hasStandardDeviation
            false
            hasConfidenceInterval
            true
            confidenceIntervalNullable
            0.0616434068014536
            hasUnevenConfidenceInterval
            false
            hasNumericResultPublic
            false
            customAdditionalResults
              hasNumericResultPrivate
              false
              task_id
              53
              sort_order
              DESCENDING
              source_name
              EM Swedish
              display_type
              percentages
              task_version
              1
              task_display_type
              PERCENTAGES
              normalization_status
              normalized
            published_at
            2025-07-15T11:49:31.790Z
            candidate_class
            modelVersion
            source_revision
            b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
            publisher_anchor
            openai/o1-2024-12-17
            metric_population
            51
            benchmark_evaluation
            true
            task_metadata_source
            benchmarkVersion
            candidate_metadata_source
            benchmarkVersionMappings
            evaluated at
            2025-06-16T00:00:00.000Z
            source model id
            o1
            confidence lower
            0.3863565931985464
            confidence upper
            0.5096434068014536
            Original result ↗

            LiveBench

            67.451%

            source id
            liveBench
            leaderboard name
            LiveBench Overall
            dimension
            general
            metric
            Average score
            rank
            5
            sample size
            418
            evaluated at
            2025-02-06T20:21:56.570Z
            source model id
            o1-2024-12-17-high
            Original result ↗

            LiveBench

            63.539%

            source id
            liveBench
            leaderboard name
            LiveBench Language
            dimension
            intelligence
            metric
            Average score
            rank
            5
            sample size
            240
            evaluated at
            2025-02-06T20:21:56.570Z
            source model id
            o1-2024-12-17-high
            Original result ↗

            Kaggle Benchmarks

            49.2%

            source id
            kaggleBenchmarks
            leaderboard name
            MultiLoKo Czech · EM Czech
            dimension
            reasoning
            metric
            MultiLoKo Czech · EM Czech
            rank
            5
            aggregation scope
            kaggle
            true
            status
            published
            license
            Apache-2.0
            direction
            higherIsBetter
            conditions
            task
            id
            33
            name
            EM Czech
            version
            1
            sort_order
            DESCENDING
            display_type
            percentages
            benchmark
            id
            56
            name
            MultiLoKo Czech
            type
            INDIVIDUAL
            owner
            metaresearch
            version
            1
            categories
            • multilingual and cross-lingual capabilities
            • general knowledge and reasoning
            version_id
            42
            maintainer_tier
            Kaggle-maintained
            maintenance_level
            KAGGLE_MAINTAINED
            candidate
            candidate_class
            modelVersion
            record_kind
            aggregate
            attribution
            Kaggle · metaresearch · MultiLoKo Czech
            license_url
            https://www.apache.org/licenses/LICENSE-2.0
            metric_kind
            MultiLoKo Czech · EM Czech
            rank_method
            competition rank among reported evaluations in identical conditions, including unlinked observations
            source_file
            42/leaderboard.json
            source_page
            https://www.kaggle.com/benchmarks/metaresearch/multiloko-czech/versions/1
            measurements
            • name
              EM Czech
              unit
              percent
              scale
              1
              metric
              MultiLoKo Czech · EM Czech
              result
              resultCase
              numericResult
              booleanResult
              false
              numericResult
              value
              0.492
              standardDeviation
              0
              confidenceInterval
              0.0619715693462415
              hasStandardDeviation
              false
              hasConfidenceInterval
              true
              confidenceIntervalNullable
              0.0619715693462415
              hasUnevenConfidenceInterval
              false
              taskVersionId
              0
              evaluationDate
              2025-06-16T00:00:00Z
              hasBooleanResult
              false
              hasNumericResult
              true
              hasTaskVersionId
              false
              hasEvaluationDate
              true
              numericResultNullable
              value
              0.492
              standardDeviation
              0
              confidenceInterval
              0.0619715693462415
              hasStandardDeviation
              false
              hasConfidenceInterval
              true
              confidenceIntervalNullable
              0.0619715693462415
              hasUnevenConfidenceInterval
              false
              hasNumericResultPublic
              false
              customAdditionalResults
                hasNumericResultPrivate
                false
                task_id
                33
                sort_order
                DESCENDING
                source_name
                EM Czech
                display_type
                percentages
                task_version
                1
                task_display_type
                PERCENTAGES
                normalization_status
                normalized
              published_at
              2025-07-15T11:49:31.790Z
              candidate_class
              modelVersion
              source_revision
              b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
              publisher_anchor
              openai/o1-2024-12-17
              metric_population
              51
              benchmark_evaluation
              true
              task_metadata_source
              benchmarkVersion
              candidate_metadata_source
              benchmarkVersionMappings
              evaluated at
              2025-06-16T00:00:00.000Z
              source model id
              o1
              confidence lower
              0.4300284306537585
              confidence upper
              0.5539715693462415
              Original result ↗

              Kaggle Benchmarks

              48%

              source id
              kaggleBenchmarks
              leaderboard name
              MultiLoKo Russian · EM Russian
              dimension
              reasoning
              metric
              MultiLoKo Russian · EM Russian
              rank
              5
              aggregation scope
              kaggle
              true
              status
              published
              license
              Apache-2.0
              direction
              higherIsBetter
              conditions
              task
              id
              50
              name
              EM Russian
              version
              1
              sort_order
              DESCENDING
              display_type
              percentages
              benchmark
              id
              73
              name
              MultiLoKo Russian
              type
              INDIVIDUAL
              owner
              metaresearch
              version
              1
              categories
              • multilingual and cross-lingual capabilities
              • general knowledge and reasoning
              version_id
              59
              maintainer_tier
              Kaggle-maintained
              maintenance_level
              KAGGLE_MAINTAINED
              candidate
              candidate_class
              modelVersion
              record_kind
              aggregate
              attribution
              Kaggle · metaresearch · MultiLoKo Russian
              license_url
              https://www.apache.org/licenses/LICENSE-2.0
              metric_kind
              MultiLoKo Russian · EM Russian
              rank_method
              competition rank among reported evaluations in identical conditions, including unlinked observations
              source_file
              59/leaderboard.json
              source_page
              https://www.kaggle.com/benchmarks/metaresearch/multiloko-russian/versions/1
              measurements
              • name
                EM Russian
                unit
                percent
                scale
                1
                metric
                MultiLoKo Russian · EM Russian
                result
                resultCase
                numericResult
                booleanResult
                false
                numericResult
                value
                0.48
                standardDeviation
                0
                confidenceInterval
                0.0619298997785481
                hasStandardDeviation
                false
                hasConfidenceInterval
                true
                confidenceIntervalNullable
                0.0619298997785481
                hasUnevenConfidenceInterval
                false
                taskVersionId
                0
                evaluationDate
                2025-06-16T00:00:00Z
                hasBooleanResult
                false
                hasNumericResult
                true
                hasTaskVersionId
                false
                hasEvaluationDate
                true
                numericResultNullable
                value
                0.48
                standardDeviation
                0
                confidenceInterval
                0.0619298997785481
                hasStandardDeviation
                false
                hasConfidenceInterval
                true
                confidenceIntervalNullable
                0.0619298997785481
                hasUnevenConfidenceInterval
                false
                hasNumericResultPublic
                false
                customAdditionalResults
                  hasNumericResultPrivate
                  false
                  task_id
                  50
                  sort_order
                  DESCENDING
                  source_name
                  EM Russian
                  display_type
                  percentages
                  task_version
                  1
                  task_display_type
                  PERCENTAGES
                  normalization_status
                  normalized
                published_at
                2025-07-15T11:49:31.790Z
                candidate_class
                modelVersion
                source_revision
                b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
                publisher_anchor
                openai/o1-2024-12-17
                metric_population
                51
                benchmark_evaluation
                true
                task_metadata_source
                benchmarkVersion
                candidate_metadata_source
                benchmarkVersionMappings
                evaluated at
                2025-06-16T00:00:00.000Z
                source model id
                o1
                confidence lower
                0.41807010022145186
                confidence upper
                0.5419298997785481
                Original result ↗

                Kaggle Benchmarks

                47.6%

                source id
                kaggleBenchmarks
                leaderboard name
                MultiLoKo Turkish · EM Turkish
                dimension
                reasoning
                metric
                MultiLoKo Turkish · EM Turkish
                rank
                5
                aggregation scope
                kaggle
                true
                status
                published
                license
                Apache-2.0
                direction
                higherIsBetter
                conditions
                task
                id
                57
                name
                EM Turkish
                version
                1
                sort_order
                DESCENDING
                display_type
                percentages
                benchmark
                id
                80
                name
                MultiLoKo Turkish
                type
                INDIVIDUAL
                owner
                metaresearch
                version
                1
                categories
                • multilingual and cross-lingual capabilities
                • general knowledge and reasoning
                version_id
                66
                maintainer_tier
                Kaggle-maintained
                maintenance_level
                KAGGLE_MAINTAINED
                candidate
                candidate_class
                modelVersion
                record_kind
                aggregate
                attribution
                Kaggle · metaresearch · MultiLoKo Turkish
                license_url
                https://www.apache.org/licenses/LICENSE-2.0
                metric_kind
                MultiLoKo Turkish · EM Turkish
                rank_method
                competition rank among reported evaluations in identical conditions, including unlinked observations
                source_file
                66/leaderboard.json
                source_page
                https://www.kaggle.com/benchmarks/metaresearch/multiloko-turkish/versions/1
                measurements
                • name
                  EM Turkish
                  unit
                  percent
                  scale
                  1
                  metric
                  MultiLoKo Turkish · EM Turkish
                  result
                  resultCase
                  numericResult
                  booleanResult
                  false
                  numericResult
                  value
                  0.476
                  standardDeviation
                  0
                  confidenceInterval
                  0.0619080616686651
                  hasStandardDeviation
                  false
                  hasConfidenceInterval
                  true
                  confidenceIntervalNullable
                  0.0619080616686651
                  hasUnevenConfidenceInterval
                  false
                  taskVersionId
                  0
                  evaluationDate
                  2025-06-16T00:00:00Z
                  hasBooleanResult
                  false
                  hasNumericResult
                  true
                  hasTaskVersionId
                  false
                  hasEvaluationDate
                  true
                  numericResultNullable
                  value
                  0.476
                  standardDeviation
                  0
                  confidenceInterval
                  0.0619080616686651
                  hasStandardDeviation
                  false
                  hasConfidenceInterval
                  true
                  confidenceIntervalNullable
                  0.0619080616686651
                  hasUnevenConfidenceInterval
                  false
                  hasNumericResultPublic
                  false
                  customAdditionalResults
                    hasNumericResultPrivate
                    false
                    task_id
                    57
                    sort_order
                    DESCENDING
                    source_name
                    EM Turkish
                    display_type
                    percentages
                    task_version
                    1
                    task_display_type
                    PERCENTAGES
                    normalization_status
                    normalized
                  published_at
                  2025-07-15T11:49:31.790Z
                  candidate_class
                  modelVersion
                  source_revision
                  b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
                  publisher_anchor
                  openai/o1-2024-12-17
                  metric_population
                  51
                  benchmark_evaluation
                  true
                  task_metadata_source
                  benchmarkVersion
                  candidate_metadata_source
                  benchmarkVersionMappings
                  evaluated at
                  2025-06-16T00:00:00.000Z
                  source model id
                  o1
                  confidence lower
                  0.4140919383313349
                  confidence upper
                  0.5379080616686651
                  Original result ↗

                  Kaggle Benchmarks

                  40.8%

                  source id
                  kaggleBenchmarks
                  leaderboard name
                  MultiLoKo Hebrew · EM Hebrew
                  dimension
                  reasoning
                  metric
                  MultiLoKo Hebrew · EM Hebrew
                  rank
                  5
                  aggregation scope
                  kaggle
                  true
                  status
                  published
                  license
                  Apache-2.0
                  direction
                  higherIsBetter
                  conditions
                  task
                  id
                  38
                  name
                  EM Hebrew
                  version
                  1
                  sort_order
                  DESCENDING
                  display_type
                  percentages
                  benchmark
                  id
                  61
                  name
                  MultiLoKo Hebrew
                  type
                  INDIVIDUAL
                  owner
                  metaresearch
                  version
                  1
                  categories
                  • multilingual and cross-lingual capabilities
                  • general knowledge and reasoning
                  version_id
                  47
                  maintainer_tier
                  Kaggle-maintained
                  maintenance_level
                  KAGGLE_MAINTAINED
                  candidate
                  candidate_class
                  modelVersion
                  record_kind
                  aggregate
                  attribution
                  Kaggle · metaresearch · MultiLoKo Hebrew
                  license_url
                  https://www.apache.org/licenses/LICENSE-2.0
                  metric_kind
                  MultiLoKo Hebrew · EM Hebrew
                  rank_method
                  competition rank among reported evaluations in identical conditions, including unlinked observations
                  source_file
                  47/leaderboard.json
                  source_page
                  https://www.kaggle.com/benchmarks/metaresearch/multiloko-hebrew/versions/1
                  measurements
                  • name
                    EM Hebrew
                    unit
                    percent
                    scale
                    1
                    metric
                    MultiLoKo Hebrew · EM Hebrew
                    result
                    resultCase
                    numericResult
                    booleanResult
                    false
                    numericResult
                    value
                    0.408
                    standardDeviation
                    0
                    confidenceInterval
                    0.0609212802792317
                    hasStandardDeviation
                    false
                    hasConfidenceInterval
                    true
                    confidenceIntervalNullable
                    0.0609212802792317
                    hasUnevenConfidenceInterval
                    false
                    taskVersionId
                    0
                    evaluationDate
                    2025-06-16T00:00:00Z
                    hasBooleanResult
                    false
                    hasNumericResult
                    true
                    hasTaskVersionId
                    false
                    hasEvaluationDate
                    true
                    numericResultNullable
                    value
                    0.408
                    standardDeviation
                    0
                    confidenceInterval
                    0.0609212802792317
                    hasStandardDeviation
                    false
                    hasConfidenceInterval
                    true
                    confidenceIntervalNullable
                    0.0609212802792317
                    hasUnevenConfidenceInterval
                    false
                    hasNumericResultPublic
                    false
                    customAdditionalResults
                      hasNumericResultPrivate
                      false
                      task_id
                      38
                      sort_order
                      DESCENDING
                      source_name
                      EM Hebrew
                      display_type
                      percentages
                      task_version
                      1
                      task_display_type
                      PERCENTAGES
                      normalization_status
                      normalized
                    published_at
                    2025-07-15T11:49:31.790Z
                    candidate_class
                    modelVersion
                    source_revision
                    b32d007df5acbddafca23c7dd2beba8221c519cf7dd3445a1d7bc28740ffd0d8
                    publisher_anchor
                    openai/o1-2024-12-17
                    metric_population
                    51
                    benchmark_evaluation
                    true
                    task_metadata_source
                    benchmarkVersion
                    candidate_metadata_source
                    benchmarkVersionMappings
                    evaluated at
                    2025-06-16T00:00:00.000Z
                    source model id
                    o1
                    confidence lower
                    0.34707871972076826
                    confidence upper
                    0.4689212802792317
                    Original result ↗

                    Provider specifications & access

                    1–1 / 1

                    OpenAI o1 · azure-openai v2024-12-17 ↗

                    source model id
                    azure-openai/o1@2024-12-17
                    hosting
                    foundry-managed-compute
                    platforms
                    • Microsoft Foundry
                    license
                    custom
                    context length
                    200000
                    max output tokens
                    100000
                    architecture
                    input_modalities
                    • text
                    • image
                    output_modalities
                    • text
                    capabilities
                    • reasoning
                    • toolCalling

                    Safety assessments

                    1–1 / 1

                    Foundation Model Transparency Index 2025 ↗

                    35 / 100

                    source id
                    stanfordFMTI
                    canonical model id
                    openai/o1-2024-12-17
                    taxonomy
                    transparency
                    hazard category
                    transparency
                    hazard name
                    Organization transparency
                    policy area
                    OpenAI
                    assessment type
                    organization-level disclosure
                    metric
                    disclosure_score
                    result label
                    35 / 100
                    sample size
                    100
                    updated at
                    2026-11-09T18:08:05.227Z

                    Source specification observations

                    1–3 / 3

                    azureFoundry ↗

                    field
                    context_length
                    value
                    200000
                    normalized value
                    200000
                    scope
                    source-model
                    source id
                    azureFoundry
                    confidence
                    1

                    azureFoundry ↗

                    field
                    max_output_tokens
                    value
                    100000
                    normalized value
                    100000
                    scope
                    source-model
                    source id
                    azureFoundry
                    confidence
                    1

                    azureFoundry ↗

                    field
                    knowledge_cutoff
                    reported field
                    training_data_cutoff
                    value
                    September 2023
                    normalized value
                    2023-09
                    precision
                    month
                    scope
                    source-model
                    source id
                    azureFoundry
                    confidence
                    1

                    Original sources

                    1–8 / 8

                    Microsoft Foundry · Model page ↗

                    Microsoft Foundry · Model documentation ↗

                    GitHub · Catalog ↗

                    GitHub · Catalog ↗

                    Kaggle · Catalog ↗

                    LMArena · Catalog ↗

                    Microsoft Foundry · Catalog ↗

                    livebench.ai · Catalog ↗

                    Source links

                    1–7 / 7

                    Explore more models

                    Models

                    Loading
                    EVIDENCE / COMPARISON

                    Benchmarks

                    Explore every benchmark

                    Loading

                    These charts contain model-linked results. The charts above also include unlinked source profiles. Compare scores within each evaluation. Model quality, usage, transparency and system performance measure different things.

                    Install TensorTrace