DeepSeek: DeepSeek V3
DeepSeek-V3 is the latest model from the DeepSeek team, building upon the instruction following and coding abilities of the previous versions. Pre-trained on nearly 15 trillion tokens, the reported e…
deepseek/deepseek-chat-v3
DeepSeek-V3 is the latest model from the DeepSeek team, building upon the instruction following and coding abilities of the previous versions. Pre-trained on nearly 15 trillion tokens, the reported evaluations reveal that the model outperforms other open-source models and rivals leading closed-source models.
For model details, please visit [the DeepSeek-V3 repo](https://github.com/deepseek-ai/DeepSeek-V3) for more information, or see the [launch announcement](https://api-docs.deepseek.com/news/news1226).
Specifications & capabilities
- Input
- text
- Output
- text
- Context window · tokens
- 163840
- Knowledge cutoff
- 2024-07-31
- Published model ID
- DeepSeek-V3
- Capabilities
- structuredOutputs, toolCalling
Description source ↗Evaluations
1–20 / 55
0.916
- source id
- llmStatsArchive
- leaderboard name
- DROP · Raw
- dimension
- general
- metric
- DROP · Raw
- rank
- 1
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- DROP
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:12.981569+00:00
- paper_link
- https://arxiv.org/abs/1903.00161
- updated_at
- 2025-07-19T19:56:12.981569+00:00
- description
- DROP (Discrete Reasoning Over Paragraphs) is a reading comprehension benchmark requiring discrete reasoning over paragraph content. It contains crowdsourced, adversarially-created questions that require resolving references and performing discrete operations like addition, counting, or sorting, demanding comprehensive paragraph understanding beyond paraphrase-and-entity-typing shortcuts.
- benchmark_id
- drop
- multilingual
- false
- analysis_method
- 3-shot F1
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- DROP · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/deepseek/models/deepseek-v3/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://github.com/deepseek-ai/DeepSeek-V3
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- deepseek/deepseek-v3
- metric_population
- 1
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- deepseek-v3
Original result ↗0.916
- source id
- llmStatsArchive
- leaderboard name
- DROP · Normalized (reported)
- dimension
- general
- metric
- DROP · Normalized (reported)
- rank
- 1
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- DROP
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:12.981569+00:00
- paper_link
- https://arxiv.org/abs/1903.00161
- updated_at
- 2025-07-19T19:56:12.981569+00:00
- description
- DROP (Discrete Reasoning Over Paragraphs) is a reading comprehension benchmark requiring discrete reasoning over paragraph content. It contains crowdsourced, adversarially-created questions that require resolving references and performing discrete operations like addition, counting, or sorting, demanding comprehensive paragraph understanding beyond paraphrase-and-entity-typing shortcuts.
- benchmark_id
- drop
- multilingual
- false
- analysis_method
- 3-shot F1
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- DROP · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/deepseek/models/deepseek-v3/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://github.com/deepseek-ai/DeepSeek-V3
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- deepseek/deepseek-v3
- metric_population
- 1
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- deepseek-v3
Original result ↗0.891
- source id
- llmStatsArchive
- leaderboard name
- MMLU-Redux · Raw
- dimension
- general
- metric
- MMLU-Redux · Raw
- rank
- 1
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- MMLU-Redux
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- categories
- language
- reasoning
- math
- general
- created_at
- 2025-09-05T00:00:00.000000+00:00
- paper_link
- https://arxiv.org/abs/2406.04127
- updated_at
- 2025-09-05T00:00:00.000000+00:00
- description
- An improved version of the MMLU benchmark featuring manually re-annotated questions to identify and correct errors in the original dataset. Provides more reliable evaluation metrics for language models by addressing dataset quality issues found in the original MMLU.
- benchmark_id
- mmlu-redux
- multilingual
- false
- analysis_method
- Exact Match
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- MMLU-Redux · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/deepseek/models/deepseek-v3/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://github.com/deepseek-ai/DeepSeek-V3
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- deepseek/deepseek-v3
- metric_population
- 1
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- deepseek-v3
Original result ↗0.891
- source id
- llmStatsArchive
- leaderboard name
- MMLU-Redux · Normalized (reported)
- dimension
- general
- metric
- MMLU-Redux · Normalized (reported)
- rank
- 1
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- MMLU-Redux
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- categories
- language
- reasoning
- math
- general
- created_at
- 2025-09-05T00:00:00.000000+00:00
- paper_link
- https://arxiv.org/abs/2406.04127
- updated_at
- 2025-09-05T00:00:00.000000+00:00
- description
- An improved version of the MMLU benchmark featuring manually re-annotated questions to identify and correct errors in the original dataset. Provides more reliable evaluation metrics for language models by addressing dataset quality issues found in the original MMLU.
- benchmark_id
- mmlu-redux
- multilingual
- false
- analysis_method
- Exact Match
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- MMLU-Redux · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/deepseek/models/deepseek-v3/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://github.com/deepseek-ai/DeepSeek-V3
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- deepseek/deepseek-v3
- metric_population
- 1
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- deepseek-v3
Original result ↗0.885
- source id
- llmStatsArchive
- leaderboard name
- MMLU · Normalized (reported)
- dimension
- general
- metric
- MMLU · Normalized (reported)
- rank
- 1
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- MMLU
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- categories
- general
- reasoning
- language
- math
- created_at
- 2025-07-19T19:56:11.200416+00:00
- paper_link
- https://arxiv.org/abs/2009.03300
- updated_at
- 2025-07-19T19:56:11.200416+00:00
- description
- Massive Multitask Language Understanding benchmark testing knowledge across 57 diverse subjects including STEM, humanities, social sciences, and professional domains
- benchmark_id
- mmlu
- multilingual
- false
- analysis_method
- Exact Match
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- MMLU · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/deepseek/models/deepseek-v3/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://github.com/deepseek-ai/DeepSeek-V3
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- deepseek/deepseek-v3
- metric_population
- 2
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- deepseek-v3
Original result ↗0.885
- source id
- llmStatsArchive
- leaderboard name
- MMLU · Raw
- dimension
- general
- metric
- MMLU · Raw
- rank
- 1
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- MMLU
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- categories
- general
- reasoning
- language
- math
- created_at
- 2025-07-19T19:56:11.200416+00:00
- paper_link
- https://arxiv.org/abs/2009.03300
- updated_at
- 2025-07-19T19:56:11.200416+00:00
- description
- Massive Multitask Language Understanding benchmark testing knowledge across 57 diverse subjects including STEM, humanities, social sciences, and professional domains
- benchmark_id
- mmlu
- multilingual
- false
- analysis_method
- Exact Match
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- MMLU · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/deepseek/models/deepseek-v3/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://github.com/deepseek-ai/DeepSeek-V3
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- deepseek/deepseek-v3
- metric_population
- 2
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- deepseek-v3
Original result ↗0.826
- source id
- llmStatsArchive
- leaderboard name
- HumanEval-Mul · Raw
- dimension
- general
- metric
- HumanEval-Mul · Raw
- rank
- 1
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- HumanEval-Mul
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:15.032472+00:00
- paper_link
- https://arxiv.org/abs/2107.03374
- updated_at
- 2025-07-19T19:56:15.032472+00:00
- description
- A multilingual variant of the HumanEval benchmark that measures functional correctness for synthesizing programs from docstrings, consisting of 164 original programming problems assessing language comprehension, algorithms, and simple mathematics
- benchmark_id
- humaneval-mul
- multilingual
- true
- analysis_method
- Pass@1
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- HumanEval-Mul · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/deepseek/models/deepseek-v3/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://github.com/deepseek-ai/DeepSeek-V3
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- deepseek/deepseek-v3
- metric_population
- 2
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- deepseek-v3
Original result ↗0.826
- source id
- llmStatsArchive
- leaderboard name
- HumanEval-Mul · Normalized (reported)
- dimension
- general
- metric
- HumanEval-Mul · Normalized (reported)
- rank
- 1
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- HumanEval-Mul
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:15.032472+00:00
- paper_link
- https://arxiv.org/abs/2107.03374
- updated_at
- 2025-07-19T19:56:15.032472+00:00
- description
- A multilingual variant of the HumanEval benchmark that measures functional correctness for synthesizing programs from docstrings, consisting of 164 original programming problems assessing language comprehension, algorithms, and simple mathematics
- benchmark_id
- humaneval-mul
- multilingual
- true
- analysis_method
- Pass@1
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- HumanEval-Mul · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/deepseek/models/deepseek-v3/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://github.com/deepseek-ai/DeepSeek-V3
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- deepseek/deepseek-v3
- metric_population
- 2
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- deepseek-v3
Original result ↗0.797
- source id
- llmStatsArchive
- leaderboard name
- Aider-Polyglot Edit · Normalized (reported)
- dimension
- general
- metric
- Aider-Polyglot Edit · Normalized (reported)
- rank
- 1
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- Aider-Polyglot Edit
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:13.789839+00:00
- updated_at
- 2025-09-30T00:00:00.000000+00:00
- description
- A challenging multi-language coding benchmark that evaluates models' code editing abilities across C++, Go, Java, JavaScript, Python, and Rust. Contains 225 of Exercism's most difficult programming problems, selected as problems that were solved by 3 or fewer out of 7 top coding models. The benchmark focuses on code editing tasks and measures both correctness of solutions and proper edit format usage. Designed to re-calibrate evaluation scales so top models score between 5-50%.
- benchmark_id
- aider-polyglot-edit
- multilingual
- false
- implementation_link
- https://github.com/Aider-AI/polyglot-benchmark
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- Aider-Polyglot Edit · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/deepseek/models/deepseek-v3/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://github.com/deepseek-ai/DeepSeek-V3
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- deepseek/deepseek-v3
- metric_population
- 3
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- deepseek-v3
Original result ↗0.797
- source id
- llmStatsArchive
- leaderboard name
- Aider-Polyglot Edit · Raw
- dimension
- general
- metric
- Aider-Polyglot Edit · Raw
- rank
- 1
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- Aider-Polyglot Edit
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:13.789839+00:00
- updated_at
- 2025-09-30T00:00:00.000000+00:00
- description
- A challenging multi-language coding benchmark that evaluates models' code editing abilities across C++, Go, Java, JavaScript, Python, and Rust. Contains 225 of Exercism's most difficult programming problems, selected as problems that were solved by 3 or fewer out of 7 top coding models. The benchmark focuses on code editing tasks and measures both correctness of solutions and proper edit format usage. Designed to re-calibrate evaluation scales so top models score between 5-50%.
- benchmark_id
- aider-polyglot-edit
- multilingual
- false
- implementation_link
- https://github.com/Aider-AI/polyglot-benchmark
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- Aider-Polyglot Edit · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/deepseek/models/deepseek-v3/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://github.com/deepseek-ai/DeepSeek-V3
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- deepseek/deepseek-v3
- metric_population
- 3
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- deepseek-v3
Original result ↗0.733
- source id
- llmStatsArchive
- leaderboard name
- FRAMES · Normalized (reported)
- dimension
- general
- metric
- FRAMES · Normalized (reported)
- rank
- 1
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- FRAMES
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:14.954436+00:00
- paper_link
- https://arxiv.org/abs/2409.12941
- updated_at
- 2025-07-19T19:56:14.954436+00:00
- description
- Factuality, Retrieval, And reasoning MEasurement Set - a unified evaluation dataset of 824 challenging multi-hop questions for testing retrieval-augmented generation systems across factuality, retrieval accuracy, and reasoning capabilities, requiring integration of 2-15 Wikipedia articles per question
- benchmark_id
- frames
- multilingual
- false
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- FRAMES · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/deepseek/models/deepseek-v3/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://github.com/deepseek-ai/DeepSeek-V3
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- deepseek/deepseek-v3
- metric_population
- 1
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- deepseek-v3
Original result ↗0.733
- source id
- llmStatsArchive
- leaderboard name
- FRAMES · Raw
- dimension
- general
- metric
- FRAMES · Raw
- rank
- 1
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- FRAMES
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:14.954436+00:00
- paper_link
- https://arxiv.org/abs/2409.12941
- updated_at
- 2025-07-19T19:56:14.954436+00:00
- description
- Factuality, Retrieval, And reasoning MEasurement Set - a unified evaluation dataset of 824 challenging multi-hop questions for testing retrieval-augmented generation systems across factuality, retrieval accuracy, and reasoning capabilities, requiring integration of 2-15 Wikipedia articles per question
- benchmark_id
- frames
- multilingual
- false
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- FRAMES · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/deepseek/models/deepseek-v3/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://github.com/deepseek-ai/DeepSeek-V3
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- deepseek/deepseek-v3
- metric_population
- 1
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- deepseek-v3
Original result ↗0.487
- source id
- llmStatsArchive
- leaderboard name
- LongBench v2 · Normalized (reported)
- dimension
- general
- metric
- LongBench v2 · Normalized (reported)
- rank
- 1
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- LongBench v2
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- categories
- long_context
- reasoning
- general
- created_at
- 2025-07-19T19:56:15.029281+00:00
- paper_link
- https://arxiv.org/abs/2412.15204
- updated_at
- 2025-07-19T19:56:15.029281+00:00
- description
- LongBench v2 is a benchmark designed to assess the ability of LLMs to handle long-context problems requiring deep understanding and reasoning across real-world multitasks. It consists of 503 challenging multiple-choice questions with contexts ranging from 8k to 2M words across six major task categories: single-document QA, multi-document QA, long in-context learning, long-dialogue history understanding, code repository understanding, and long structured data understanding.
- benchmark_id
- longbench-v2
- multilingual
- true
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- LongBench v2 · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/deepseek/models/deepseek-v3/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://github.com/deepseek-ai/DeepSeek-V3
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- deepseek/deepseek-v3
- metric_population
- 1
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- deepseek-v3
Original result ↗0.487
- source id
- llmStatsArchive
- leaderboard name
- LongBench v2 · Raw
- dimension
- general
- metric
- LongBench v2 · Raw
- rank
- 1
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- LongBench v2
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- categories
- long_context
- reasoning
- general
- created_at
- 2025-07-19T19:56:15.029281+00:00
- paper_link
- https://arxiv.org/abs/2412.15204
- updated_at
- 2025-07-19T19:56:15.029281+00:00
- description
- LongBench v2 is a benchmark designed to assess the ability of LLMs to handle long-context problems requiring deep understanding and reasoning across real-world multitasks. It consists of 503 challenging multiple-choice questions with contexts ranging from 8k to 2M words across six major task categories: single-document QA, multi-document QA, long in-context learning, long-dialogue history understanding, code repository understanding, and long structured data understanding.
- benchmark_id
- longbench-v2
- multilingual
- true
- analysis_method
- Accuracy
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- LongBench v2 · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/deepseek/models/deepseek-v3/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://github.com/deepseek-ai/DeepSeek-V3
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- deepseek/deepseek-v3
- metric_population
- 1
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- deepseek-v3
Original result ↗0.432
- source id
- llmStatsArchive
- leaderboard name
- CNMO 2024 · Normalized (reported)
- dimension
- general
- metric
- CNMO 2024 · Normalized (reported)
- rank
- 1
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- CNMO 2024
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-09-05T00:00:00.000000+00:00
- updated_at
- 2025-09-05T00:00:00.000000+00:00
- description
- China Mathematical Olympiad 2024 - A challenging mathematics competition.
- benchmark_id
- cnmo-2024
- multilingual
- false
- analysis_method
- Pass@1
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- CNMO 2024 · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/deepseek/models/deepseek-v3/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://github.com/deepseek-ai/DeepSeek-V3
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- deepseek/deepseek-v3
- metric_population
- 1
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- deepseek-v3
Original result ↗0.432
- source id
- llmStatsArchive
- leaderboard name
- CNMO 2024 · Raw
- dimension
- general
- metric
- CNMO 2024 · Raw
- rank
- 1
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- CNMO 2024
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-09-05T00:00:00.000000+00:00
- updated_at
- 2025-09-05T00:00:00.000000+00:00
- description
- China Mathematical Olympiad 2024 - A challenging mathematics competition.
- benchmark_id
- cnmo-2024
- multilingual
- false
- analysis_method
- Pass@1
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- CNMO 2024 · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/deepseek/models/deepseek-v3/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://github.com/deepseek-ai/DeepSeek-V3
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- deepseek/deepseek-v3
- metric_population
- 1
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- deepseek-v3
Original result ↗0.909
- source id
- llmStatsArchive
- leaderboard name
- CLUEWSC · Normalized (reported)
- dimension
- general
- metric
- CLUEWSC · Normalized (reported)
- rank
- 2
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- CLUEWSC
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:12.233189+00:00
- paper_link
- https://arxiv.org/abs/2004.05986
- updated_at
- 2025-07-19T19:56:12.233189+00:00
- description
- CLUEWSC2020 is the Chinese version of the Winograd Schema Challenge, part of the CLUE benchmark. It focuses on pronoun disambiguation and coreference resolution, requiring models to determine which noun a pronoun refers to in a sentence. The dataset contains 1,244 training samples and 304 development samples extracted from contemporary Chinese literature.
- benchmark_id
- cluewsc
- multilingual
- true
- analysis_method
- Exact Match
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- CLUEWSC · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/deepseek/models/deepseek-v3/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://github.com/deepseek-ai/DeepSeek-V3
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- deepseek/deepseek-v3
- metric_population
- 2
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- deepseek-v3
Original result ↗0.909
- source id
- llmStatsArchive
- leaderboard name
- CLUEWSC · Raw
- dimension
- general
- metric
- CLUEWSC · Raw
- rank
- 2
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- CLUEWSC
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:12.233189+00:00
- paper_link
- https://arxiv.org/abs/2004.05986
- updated_at
- 2025-07-19T19:56:12.233189+00:00
- description
- CLUEWSC2020 is the Chinese version of the Winograd Schema Challenge, part of the CLUE benchmark. It focuses on pronoun disambiguation and coreference resolution, requiring models to determine which noun a pronoun refers to in a sentence. The dataset contains 1,244 training samples and 304 development samples extracted from contemporary Chinese literature.
- benchmark_id
- cluewsc
- multilingual
- true
- analysis_method
- Exact Match
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- CLUEWSC · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/deepseek/models/deepseek-v3/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://github.com/deepseek-ai/DeepSeek-V3
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- deepseek/deepseek-v3
- metric_population
- 2
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- deepseek-v3
Original result ↗0.902
- source id
- llmStatsArchive
- leaderboard name
- MATH-500 · Normalized (reported)
- dimension
- general
- metric
- MATH-500 · Normalized (reported)
- rank
- 2
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- MATH-500
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:12.027850+00:00
- paper_link
- https://arxiv.org/abs/2103.03874
- updated_at
- 2025-07-19T19:56:12.027850+00:00
- description
- MATH-500 is a subset of the MATH dataset containing 500 challenging competition mathematics problems from AMC 10, AMC 12, AIME, and other mathematics competitions. Each problem includes full step-by-step solutions and spans multiple difficulty levels across seven mathematical subjects including Prealgebra, Algebra, Number Theory, Counting and Probability, Geometry, Intermediate Algebra, and Precalculus.
- benchmark_id
- math-500
- multilingual
- false
- analysis_method
- Exact Match
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- MATH-500 · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/deepseek/models/deepseek-v3/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://github.com/deepseek-ai/DeepSeek-V3
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- deepseek/deepseek-v3
- metric_population
- 2
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- deepseek-v3
Original result ↗0.902
- source id
- llmStatsArchive
- leaderboard name
- MATH-500 · Raw
- dimension
- general
- metric
- MATH-500 · Raw
- rank
- 2
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- MATH-500
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:12.027850+00:00
- paper_link
- https://arxiv.org/abs/2103.03874
- updated_at
- 2025-07-19T19:56:12.027850+00:00
- description
- MATH-500 is a subset of the MATH dataset containing 500 challenging competition mathematics problems from AMC 10, AMC 12, AIME, and other mathematics competitions. Each problem includes full step-by-step solutions and spans multiple difficulty levels across seven mathematical subjects including Prealgebra, Algebra, Number Theory, Counting and Probability, Geometry, Intermediate Algebra, and Precalculus.
- benchmark_id
- math-500
- multilingual
- false
- analysis_method
- Exact Match
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- MATH-500 · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/deepseek/models/deepseek-v3/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://github.com/deepseek-ai/DeepSeek-V3
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- deepseek/deepseek-v3
- metric_population
- 2
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- deepseek-v3
Original result ↗Published prices
1–4 / 4
$0.26
- Inference provider
- openRouter
- Platform
- OpenRouter
- Unit
- input_tokens_per_million
- Currency
- USD
$1.03
- Inference provider
- openRouter
- Platform
- OpenRouter
- Unit
- output_tokens_per_million
- Currency
- USD
$0.25
- Inference provider
- siliconFlow
- Platform
- SiliconFlow
- Unit
- input_tokens_per_million
- Currency
- USD
- Billing label
- Input tokens (per 1M)
$1.00
- Inference provider
- siliconFlow
- Platform
- SiliconFlow
- Unit
- output_tokens_per_million
- Currency
- USD
- Billing label
- Output tokens (per 1M)
Provider specifications & access
1–3 / 3
- source model id
- deepseek-ai/DeepSeek-V3
- hosting
- siliconflow-hosted
- max output tokens
- 163840
- capabilities
- structuredOutputs
- toolCalling
- source model id
- publishers/deepseek-ai/models/deepseek-v3@deepseek-v3
- hosting
- vertex-self-deployed
- source model id
- publishers/hf-deepseek-ai/models/deepseek-v3@001
- hosting
- vertex-self-deployed
Safety assessments
1–1 / 1
32 / 100
- source id
- stanfordFMTI
- canonical model id
- deepseek/deepseek-chat-v3
- taxonomy
- transparency
- hazard category
- transparency
- hazard name
- Organization transparency
- policy area
- DeepSeek
- assessment type
- organization-level disclosure
- metric
- disclosure_score
- result label
- 32 / 100
- sample size
- 100
- updated at
- 2026-11-09T18:08:05.227Z
Source specification observations
1–5 / 5
- field
- context_length
- value
- 163840
- normalized value
- 163840
- scope
- source-model
- source id
- openRouter
- confidence
- 1
- field
- context_length
- value
- 164K
- scope
- source-model
- source id
- siliconFlow
- confidence
- 1
- field
- knowledge_cutoff
- reported field
- knowledge_cutoff
- value
- 2024-07-31
- normalized value
- 2024-07-31
- precision
- day
- scope
- source-model
- source id
- openRouter
- confidence
- 1
- field
- max_output_tokens
- value
- 16000
- normalized value
- 16000
- scope
- source-model
- source id
- openRouter
- confidence
- 1
- field
- max_output_tokens
- value
- 164K
- scope
- source-model
- source id
- siliconFlow
- confidence
- 1
Hugging Face model card
1–1 / 1
- downloads
- 1402684
- likes
- 4463
- pipeline tag
- text-generation
- library name
- transformers
- License
Explore more models