DeepSeek: DeepSeek V3

DeepSeek-V3 is the latest model from the DeepSeek team, building upon the instruction following and coding abilities of the previous versions. Pre-trained on nearly 15 trillion tokens, the reported e…

deepseek/deepseek-chat-v3

DeepSeek-V3 is the latest model from the DeepSeek team, building upon the instruction following and coding abilities of the previous versions. Pre-trained on nearly 15 trillion tokens, the reported evaluations reveal that the model outperforms other open-source models and rivals leading closed-source models.

For model details, please visit [the DeepSeek-V3 repo](https://github.com/deepseek-ai/DeepSeek-V3) for more information, or see the [launch announcement](https://api-docs.deepseek.com/news/news1226).

Specifications & capabilities

Input
text
Output
text
Context window · tokens
163840
Knowledge cutoff
2024-07-31
Published model ID
DeepSeek-V3
Capabilities
structuredOutputs, toolCalling
Description source ↗

Evaluations

1–20 / 55

LLM Stats · GitHub archive

0.916

source id
llmStatsArchive
leaderboard name
DROP · Raw
dimension
general
metric
DROP · Raw
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
DROP
language
en
modality
text
verified
false
max_score
1
categories
  • reasoning
  • math
created_at
2025-07-19T19:56:12.981569+00:00
paper_link
https://arxiv.org/abs/1903.00161
updated_at
2025-07-19T19:56:12.981569+00:00
description
DROP (Discrete Reasoning Over Paragraphs) is a reading comprehension benchmark requiring discrete reasoning over paragraph content. It contains crowdsourced, adversarially-created questions that require resolving references and performing discrete operations like addition, counting, or sorting, demanding comprehensive paragraph understanding beyond paraphrase-and-entity-typing shortcuts.
benchmark_id
drop
multilingual
false
analysis_method
3-shot F1
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
DROP · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/deepseek/models/deepseek-v3/benchmarks.json
archive_kind
llmStats
evidence_url
https://github.com/deepseek-ai/DeepSeek-V3
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
deepseek/deepseek-v3
metric_population
1
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
deepseek-v3
Original result ↗

LLM Stats · GitHub archive

0.916

source id
llmStatsArchive
leaderboard name
DROP · Normalized (reported)
dimension
general
metric
DROP · Normalized (reported)
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
DROP
language
en
modality
text
verified
false
max_score
1
categories
  • reasoning
  • math
created_at
2025-07-19T19:56:12.981569+00:00
paper_link
https://arxiv.org/abs/1903.00161
updated_at
2025-07-19T19:56:12.981569+00:00
description
DROP (Discrete Reasoning Over Paragraphs) is a reading comprehension benchmark requiring discrete reasoning over paragraph content. It contains crowdsourced, adversarially-created questions that require resolving references and performing discrete operations like addition, counting, or sorting, demanding comprehensive paragraph understanding beyond paraphrase-and-entity-typing shortcuts.
benchmark_id
drop
multilingual
false
analysis_method
3-shot F1
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
DROP · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/deepseek/models/deepseek-v3/benchmarks.json
archive_kind
llmStats
evidence_url
https://github.com/deepseek-ai/DeepSeek-V3
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
deepseek/deepseek-v3
metric_population
1
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
deepseek-v3
Original result ↗

LLM Stats · GitHub archive

0.891

source id
llmStatsArchive
leaderboard name
MMLU-Redux · Raw
dimension
general
metric
MMLU-Redux · Raw
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
MMLU-Redux
language
en
modality
text
verified
false
max_score
1
categories
  • language
  • reasoning
  • math
  • general
created_at
2025-09-05T00:00:00.000000+00:00
paper_link
https://arxiv.org/abs/2406.04127
updated_at
2025-09-05T00:00:00.000000+00:00
description
An improved version of the MMLU benchmark featuring manually re-annotated questions to identify and correct errors in the original dataset. Provides more reliable evaluation metrics for language models by addressing dataset quality issues found in the original MMLU.
benchmark_id
mmlu-redux
multilingual
false
analysis_method
Exact Match
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
MMLU-Redux · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/deepseek/models/deepseek-v3/benchmarks.json
archive_kind
llmStats
evidence_url
https://github.com/deepseek-ai/DeepSeek-V3
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
deepseek/deepseek-v3
metric_population
1
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
deepseek-v3
Original result ↗

LLM Stats · GitHub archive

0.891

source id
llmStatsArchive
leaderboard name
MMLU-Redux · Normalized (reported)
dimension
general
metric
MMLU-Redux · Normalized (reported)
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
MMLU-Redux
language
en
modality
text
verified
false
max_score
1
categories
  • language
  • reasoning
  • math
  • general
created_at
2025-09-05T00:00:00.000000+00:00
paper_link
https://arxiv.org/abs/2406.04127
updated_at
2025-09-05T00:00:00.000000+00:00
description
An improved version of the MMLU benchmark featuring manually re-annotated questions to identify and correct errors in the original dataset. Provides more reliable evaluation metrics for language models by addressing dataset quality issues found in the original MMLU.
benchmark_id
mmlu-redux
multilingual
false
analysis_method
Exact Match
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
MMLU-Redux · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/deepseek/models/deepseek-v3/benchmarks.json
archive_kind
llmStats
evidence_url
https://github.com/deepseek-ai/DeepSeek-V3
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
deepseek/deepseek-v3
metric_population
1
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
deepseek-v3
Original result ↗

LLM Stats · GitHub archive

0.885

source id
llmStatsArchive
leaderboard name
MMLU · Normalized (reported)
dimension
general
metric
MMLU · Normalized (reported)
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
MMLU
language
en
modality
text
verified
false
max_score
1
categories
  • general
  • reasoning
  • language
  • math
created_at
2025-07-19T19:56:11.200416+00:00
paper_link
https://arxiv.org/abs/2009.03300
updated_at
2025-07-19T19:56:11.200416+00:00
description
Massive Multitask Language Understanding benchmark testing knowledge across 57 diverse subjects including STEM, humanities, social sciences, and professional domains
benchmark_id
mmlu
multilingual
false
analysis_method
Exact Match
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
MMLU · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/deepseek/models/deepseek-v3/benchmarks.json
archive_kind
llmStats
evidence_url
https://github.com/deepseek-ai/DeepSeek-V3
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
deepseek/deepseek-v3
metric_population
2
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
deepseek-v3
Original result ↗

LLM Stats · GitHub archive

0.885

source id
llmStatsArchive
leaderboard name
MMLU · Raw
dimension
general
metric
MMLU · Raw
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
MMLU
language
en
modality
text
verified
false
max_score
1
categories
  • general
  • reasoning
  • language
  • math
created_at
2025-07-19T19:56:11.200416+00:00
paper_link
https://arxiv.org/abs/2009.03300
updated_at
2025-07-19T19:56:11.200416+00:00
description
Massive Multitask Language Understanding benchmark testing knowledge across 57 diverse subjects including STEM, humanities, social sciences, and professional domains
benchmark_id
mmlu
multilingual
false
analysis_method
Exact Match
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
MMLU · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/deepseek/models/deepseek-v3/benchmarks.json
archive_kind
llmStats
evidence_url
https://github.com/deepseek-ai/DeepSeek-V3
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
deepseek/deepseek-v3
metric_population
2
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
deepseek-v3
Original result ↗

LLM Stats · GitHub archive

0.826

source id
llmStatsArchive
leaderboard name
HumanEval-Mul · Raw
dimension
general
metric
HumanEval-Mul · Raw
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
HumanEval-Mul
language
en
modality
text
verified
false
max_score
1
categories
  • reasoning
created_at
2025-07-19T19:56:15.032472+00:00
paper_link
https://arxiv.org/abs/2107.03374
updated_at
2025-07-19T19:56:15.032472+00:00
description
A multilingual variant of the HumanEval benchmark that measures functional correctness for synthesizing programs from docstrings, consisting of 164 original programming problems assessing language comprehension, algorithms, and simple mathematics
benchmark_id
humaneval-mul
multilingual
true
analysis_method
Pass@1
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
HumanEval-Mul · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/deepseek/models/deepseek-v3/benchmarks.json
archive_kind
llmStats
evidence_url
https://github.com/deepseek-ai/DeepSeek-V3
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
deepseek/deepseek-v3
metric_population
2
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
deepseek-v3
Original result ↗

LLM Stats · GitHub archive

0.826

source id
llmStatsArchive
leaderboard name
HumanEval-Mul · Normalized (reported)
dimension
general
metric
HumanEval-Mul · Normalized (reported)
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
HumanEval-Mul
language
en
modality
text
verified
false
max_score
1
categories
  • reasoning
created_at
2025-07-19T19:56:15.032472+00:00
paper_link
https://arxiv.org/abs/2107.03374
updated_at
2025-07-19T19:56:15.032472+00:00
description
A multilingual variant of the HumanEval benchmark that measures functional correctness for synthesizing programs from docstrings, consisting of 164 original programming problems assessing language comprehension, algorithms, and simple mathematics
benchmark_id
humaneval-mul
multilingual
true
analysis_method
Pass@1
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
HumanEval-Mul · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/deepseek/models/deepseek-v3/benchmarks.json
archive_kind
llmStats
evidence_url
https://github.com/deepseek-ai/DeepSeek-V3
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
deepseek/deepseek-v3
metric_population
2
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
deepseek-v3
Original result ↗

LLM Stats · GitHub archive

0.797

source id
llmStatsArchive
leaderboard name
Aider-Polyglot Edit · Normalized (reported)
dimension
general
metric
Aider-Polyglot Edit · Normalized (reported)
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
Aider-Polyglot Edit
language
en
modality
text
verified
false
max_score
1
categories
  • general
  • code
created_at
2025-07-19T19:56:13.789839+00:00
updated_at
2025-09-30T00:00:00.000000+00:00
description
A challenging multi-language coding benchmark that evaluates models' code editing abilities across C++, Go, Java, JavaScript, Python, and Rust. Contains 225 of Exercism's most difficult programming problems, selected as problems that were solved by 3 or fewer out of 7 top coding models. The benchmark focuses on code editing tasks and measures both correctness of solutions and proper edit format usage. Designed to re-calibrate evaluation scales so top models score between 5-50%.
benchmark_id
aider-polyglot-edit
multilingual
false
implementation_link
https://github.com/Aider-AI/polyglot-benchmark
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
Aider-Polyglot Edit · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/deepseek/models/deepseek-v3/benchmarks.json
archive_kind
llmStats
evidence_url
https://github.com/deepseek-ai/DeepSeek-V3
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
deepseek/deepseek-v3
metric_population
3
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
deepseek-v3
Original result ↗

LLM Stats · GitHub archive

0.797

source id
llmStatsArchive
leaderboard name
Aider-Polyglot Edit · Raw
dimension
general
metric
Aider-Polyglot Edit · Raw
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
Aider-Polyglot Edit
language
en
modality
text
verified
false
max_score
1
categories
  • general
  • code
created_at
2025-07-19T19:56:13.789839+00:00
updated_at
2025-09-30T00:00:00.000000+00:00
description
A challenging multi-language coding benchmark that evaluates models' code editing abilities across C++, Go, Java, JavaScript, Python, and Rust. Contains 225 of Exercism's most difficult programming problems, selected as problems that were solved by 3 or fewer out of 7 top coding models. The benchmark focuses on code editing tasks and measures both correctness of solutions and proper edit format usage. Designed to re-calibrate evaluation scales so top models score between 5-50%.
benchmark_id
aider-polyglot-edit
multilingual
false
implementation_link
https://github.com/Aider-AI/polyglot-benchmark
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
Aider-Polyglot Edit · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/deepseek/models/deepseek-v3/benchmarks.json
archive_kind
llmStats
evidence_url
https://github.com/deepseek-ai/DeepSeek-V3
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
deepseek/deepseek-v3
metric_population
3
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
deepseek-v3
Original result ↗

LLM Stats · GitHub archive

0.733

source id
llmStatsArchive
leaderboard name
FRAMES · Normalized (reported)
dimension
general
metric
FRAMES · Normalized (reported)
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
FRAMES
language
en
modality
text
verified
false
max_score
1
categories
  • reasoning
  • search
created_at
2025-07-19T19:56:14.954436+00:00
paper_link
https://arxiv.org/abs/2409.12941
updated_at
2025-07-19T19:56:14.954436+00:00
description
Factuality, Retrieval, And reasoning MEasurement Set - a unified evaluation dataset of 824 challenging multi-hop questions for testing retrieval-augmented generation systems across factuality, retrieval accuracy, and reasoning capabilities, requiring integration of 2-15 Wikipedia articles per question
benchmark_id
frames
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
FRAMES · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/deepseek/models/deepseek-v3/benchmarks.json
archive_kind
llmStats
evidence_url
https://github.com/deepseek-ai/DeepSeek-V3
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
deepseek/deepseek-v3
metric_population
1
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
deepseek-v3
Original result ↗

LLM Stats · GitHub archive

0.733

source id
llmStatsArchive
leaderboard name
FRAMES · Raw
dimension
general
metric
FRAMES · Raw
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
FRAMES
language
en
modality
text
verified
false
max_score
1
categories
  • reasoning
  • search
created_at
2025-07-19T19:56:14.954436+00:00
paper_link
https://arxiv.org/abs/2409.12941
updated_at
2025-07-19T19:56:14.954436+00:00
description
Factuality, Retrieval, And reasoning MEasurement Set - a unified evaluation dataset of 824 challenging multi-hop questions for testing retrieval-augmented generation systems across factuality, retrieval accuracy, and reasoning capabilities, requiring integration of 2-15 Wikipedia articles per question
benchmark_id
frames
multilingual
false
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
FRAMES · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/deepseek/models/deepseek-v3/benchmarks.json
archive_kind
llmStats
evidence_url
https://github.com/deepseek-ai/DeepSeek-V3
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
deepseek/deepseek-v3
metric_population
1
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
deepseek-v3
Original result ↗

LLM Stats · GitHub archive

0.487

source id
llmStatsArchive
leaderboard name
LongBench v2 · Normalized (reported)
dimension
general
metric
LongBench v2 · Normalized (reported)
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
LongBench v2
language
en
modality
text
verified
false
max_score
1
categories
  • long_context
  • reasoning
  • general
created_at
2025-07-19T19:56:15.029281+00:00
paper_link
https://arxiv.org/abs/2412.15204
updated_at
2025-07-19T19:56:15.029281+00:00
description
LongBench v2 is a benchmark designed to assess the ability of LLMs to handle long-context problems requiring deep understanding and reasoning across real-world multitasks. It consists of 503 challenging multiple-choice questions with contexts ranging from 8k to 2M words across six major task categories: single-document QA, multi-document QA, long in-context learning, long-dialogue history understanding, code repository understanding, and long structured data understanding.
benchmark_id
longbench-v2
multilingual
true
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
LongBench v2 · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/deepseek/models/deepseek-v3/benchmarks.json
archive_kind
llmStats
evidence_url
https://github.com/deepseek-ai/DeepSeek-V3
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
deepseek/deepseek-v3
metric_population
1
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
deepseek-v3
Original result ↗

LLM Stats · GitHub archive

0.487

source id
llmStatsArchive
leaderboard name
LongBench v2 · Raw
dimension
general
metric
LongBench v2 · Raw
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
LongBench v2
language
en
modality
text
verified
false
max_score
1
categories
  • long_context
  • reasoning
  • general
created_at
2025-07-19T19:56:15.029281+00:00
paper_link
https://arxiv.org/abs/2412.15204
updated_at
2025-07-19T19:56:15.029281+00:00
description
LongBench v2 is a benchmark designed to assess the ability of LLMs to handle long-context problems requiring deep understanding and reasoning across real-world multitasks. It consists of 503 challenging multiple-choice questions with contexts ranging from 8k to 2M words across six major task categories: single-document QA, multi-document QA, long in-context learning, long-dialogue history understanding, code repository understanding, and long structured data understanding.
benchmark_id
longbench-v2
multilingual
true
analysis_method
Accuracy
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
LongBench v2 · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/deepseek/models/deepseek-v3/benchmarks.json
archive_kind
llmStats
evidence_url
https://github.com/deepseek-ai/DeepSeek-V3
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
deepseek/deepseek-v3
metric_population
1
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
deepseek-v3
Original result ↗

LLM Stats · GitHub archive

0.432

source id
llmStatsArchive
leaderboard name
CNMO 2024 · Normalized (reported)
dimension
general
metric
CNMO 2024 · Normalized (reported)
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
CNMO 2024
language
en
modality
text
verified
false
max_score
1
categories
  • math
created_at
2025-09-05T00:00:00.000000+00:00
updated_at
2025-09-05T00:00:00.000000+00:00
description
China Mathematical Olympiad 2024 - A challenging mathematics competition.
benchmark_id
cnmo-2024
multilingual
false
analysis_method
Pass@1
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
CNMO 2024 · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/deepseek/models/deepseek-v3/benchmarks.json
archive_kind
llmStats
evidence_url
https://github.com/deepseek-ai/DeepSeek-V3
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
deepseek/deepseek-v3
metric_population
1
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
deepseek-v3
Original result ↗

LLM Stats · GitHub archive

0.432

source id
llmStatsArchive
leaderboard name
CNMO 2024 · Raw
dimension
general
metric
CNMO 2024 · Raw
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
CNMO 2024
language
en
modality
text
verified
false
max_score
1
categories
  • math
created_at
2025-09-05T00:00:00.000000+00:00
updated_at
2025-09-05T00:00:00.000000+00:00
description
China Mathematical Olympiad 2024 - A challenging mathematics competition.
benchmark_id
cnmo-2024
multilingual
false
analysis_method
Pass@1
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
CNMO 2024 · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/deepseek/models/deepseek-v3/benchmarks.json
archive_kind
llmStats
evidence_url
https://github.com/deepseek-ai/DeepSeek-V3
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
deepseek/deepseek-v3
metric_population
1
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
deepseek-v3
Original result ↗

LLM Stats · GitHub archive

0.909

source id
llmStatsArchive
leaderboard name
CLUEWSC · Normalized (reported)
dimension
general
metric
CLUEWSC · Normalized (reported)
rank
2
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
CLUEWSC
language
en
modality
text
verified
false
max_score
1
categories
  • language
  • reasoning
created_at
2025-07-19T19:56:12.233189+00:00
paper_link
https://arxiv.org/abs/2004.05986
updated_at
2025-07-19T19:56:12.233189+00:00
description
CLUEWSC2020 is the Chinese version of the Winograd Schema Challenge, part of the CLUE benchmark. It focuses on pronoun disambiguation and coreference resolution, requiring models to determine which noun a pronoun refers to in a sentence. The dataset contains 1,244 training samples and 304 development samples extracted from contemporary Chinese literature.
benchmark_id
cluewsc
multilingual
true
analysis_method
Exact Match
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
CLUEWSC · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/deepseek/models/deepseek-v3/benchmarks.json
archive_kind
llmStats
evidence_url
https://github.com/deepseek-ai/DeepSeek-V3
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
deepseek/deepseek-v3
metric_population
2
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
deepseek-v3
Original result ↗

LLM Stats · GitHub archive

0.909

source id
llmStatsArchive
leaderboard name
CLUEWSC · Raw
dimension
general
metric
CLUEWSC · Raw
rank
2
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
CLUEWSC
language
en
modality
text
verified
false
max_score
1
categories
  • language
  • reasoning
created_at
2025-07-19T19:56:12.233189+00:00
paper_link
https://arxiv.org/abs/2004.05986
updated_at
2025-07-19T19:56:12.233189+00:00
description
CLUEWSC2020 is the Chinese version of the Winograd Schema Challenge, part of the CLUE benchmark. It focuses on pronoun disambiguation and coreference resolution, requiring models to determine which noun a pronoun refers to in a sentence. The dataset contains 1,244 training samples and 304 development samples extracted from contemporary Chinese literature.
benchmark_id
cluewsc
multilingual
true
analysis_method
Exact Match
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
CLUEWSC · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/deepseek/models/deepseek-v3/benchmarks.json
archive_kind
llmStats
evidence_url
https://github.com/deepseek-ai/DeepSeek-V3
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
deepseek/deepseek-v3
metric_population
2
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
deepseek-v3
Original result ↗

LLM Stats · GitHub archive

0.902

source id
llmStatsArchive
leaderboard name
MATH-500 · Normalized (reported)
dimension
general
metric
MATH-500 · Normalized (reported)
rank
2
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
MATH-500
language
en
modality
text
verified
false
max_score
1
categories
  • math
  • reasoning
created_at
2025-07-19T19:56:12.027850+00:00
paper_link
https://arxiv.org/abs/2103.03874
updated_at
2025-07-19T19:56:12.027850+00:00
description
MATH-500 is a subset of the MATH dataset containing 500 challenging competition mathematics problems from AMC 10, AMC 12, AIME, and other mathematics competitions. Each problem includes full step-by-step solutions and spans multiple difficulty levels across seven mathematical subjects including Prealgebra, Algebra, Number Theory, Counting and Probability, Geometry, Intermediate Algebra, and Precalculus.
benchmark_id
math-500
multilingual
false
analysis_method
Exact Match
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
MATH-500 · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/deepseek/models/deepseek-v3/benchmarks.json
archive_kind
llmStats
evidence_url
https://github.com/deepseek-ai/DeepSeek-V3
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
deepseek/deepseek-v3
metric_population
2
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
deepseek-v3
Original result ↗

LLM Stats · GitHub archive

0.902

source id
llmStatsArchive
leaderboard name
MATH-500 · Raw
dimension
general
metric
MATH-500 · Raw
rank
2
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
MATH-500
language
en
modality
text
verified
false
max_score
1
categories
  • math
  • reasoning
created_at
2025-07-19T19:56:12.027850+00:00
paper_link
https://arxiv.org/abs/2103.03874
updated_at
2025-07-19T19:56:12.027850+00:00
description
MATH-500 is a subset of the MATH dataset containing 500 challenging competition mathematics problems from AMC 10, AMC 12, AIME, and other mathematics competitions. Each problem includes full step-by-step solutions and spans multiple difficulty levels across seven mathematical subjects including Prealgebra, Algebra, Number Theory, Counting and Probability, Geometry, Intermediate Algebra, and Precalculus.
benchmark_id
math-500
multilingual
false
analysis_method
Exact Match
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
MATH-500 · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/deepseek/models/deepseek-v3/benchmarks.json
archive_kind
llmStats
evidence_url
https://github.com/deepseek-ai/DeepSeek-V3
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
deepseek/deepseek-v3
metric_population
2
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
deepseek-v3
Original result ↗

Published prices

1–4 / 4

deepseek/deepseek-chat ↗

$0.26

Inference provider
openRouter
Platform
OpenRouter
Unit
input_tokens_per_million
Currency
USD
Billing conditions

deepseek/deepseek-chat ↗

$1.03

Inference provider
openRouter
Platform
OpenRouter
Unit
output_tokens_per_million
Currency
USD
Billing conditions

deepseek-ai/DeepSeek-V3 ↗

$0.25

Inference provider
siliconFlow
Platform
SiliconFlow
Unit
input_tokens_per_million
Currency
USD
Billing conditions
Billing label
Input tokens (per 1M)

deepseek-ai/DeepSeek-V3 ↗

$1.00

Inference provider
siliconFlow
Platform
SiliconFlow
Unit
output_tokens_per_million
Currency
USD
Billing conditions
Billing label
Output tokens (per 1M)

Provider specifications & access

1–3 / 3

DeepSeek-V3 ↗

source model id
deepseek-ai/DeepSeek-V3
hosting
siliconflow-hosted
platforms
  • SiliconFlow
max output tokens
163840
architecture
input_modalities
  • text
output_modalities
  • text
capabilities
  • structuredOutputs
  • toolCalling

deepseek-ai/DeepSeek-V3 ↗

source model id
publishers/deepseek-ai/models/deepseek-v3@deepseek-v3
hosting
vertex-self-deployed
platforms
  • Google Cloud Vertex AI
architecture
input_modalities
  • text
output_modalities
  • text
capabilities

    deepseek-ai/DeepSeek-V3 ↗

    source model id
    publishers/hf-deepseek-ai/models/deepseek-v3@001
    hosting
    vertex-self-deployed
    platforms
    • Google Cloud Vertex AI
    architecture
    input_modalities
    • text
    output_modalities
    • text
    capabilities

      Safety assessments

      1–1 / 1

      Foundation Model Transparency Index 2025 ↗

      32 / 100

      source id
      stanfordFMTI
      canonical model id
      deepseek/deepseek-chat-v3
      taxonomy
      transparency
      hazard category
      transparency
      hazard name
      Organization transparency
      policy area
      DeepSeek
      assessment type
      organization-level disclosure
      metric
      disclosure_score
      result label
      32 / 100
      sample size
      100
      updated at
      2026-11-09T18:08:05.227Z

      Source specification observations

      1–5 / 5

      openRouter ↗

      field
      context_length
      value
      163840
      normalized value
      163840
      scope
      source-model
      source id
      openRouter
      confidence
      1

      siliconFlow ↗

      field
      context_length
      value
      164K
      scope
      source-model
      source id
      siliconFlow
      confidence
      1

      openRouter ↗

      field
      knowledge_cutoff
      reported field
      knowledge_cutoff
      value
      2024-07-31
      normalized value
      2024-07-31
      precision
      day
      scope
      source-model
      source id
      openRouter
      confidence
      1

      openRouter ↗

      field
      max_output_tokens
      value
      16000
      normalized value
      16000
      scope
      source-model
      source id
      openRouter
      confidence
      1

      siliconFlow ↗

      field
      max_output_tokens
      value
      164K
      scope
      source-model
      source id
      siliconFlow
      confidence
      1

      Original sources

      1–20 / 20

      SiliconFlow · Model page ↗

      Hugging Face · Model repository ↗

      Google Cloud · Model page ↗

      Google Cloud · Model page ↗

      OpenRouter · Model page ↗

      SiliconFlow · Playground ↗

      Google Cloud · Model documentation ↗

      SiliconFlow · API documentation ↗

      arXiv · Model reference ↗

      Epoch AI · Catalog ↗

      GitHub · Catalog ↗

      GitHub · Catalog ↗

      Google Cloud · Catalog ↗

      Hugging Face · Catalog ↗

      Hugging Face · Catalog ↗

      OpenRouter · Catalog ↗

      OpenRouter · Catalog ↗

      SiliconFlow · Catalog ↗

      crfm.stanford.edu · Catalog ↗

      livebench.ai · Catalog ↗

      Source links

      1–11 / 11

      Hugging Face model card

      1–1 / 1

      Explore more models

      Models

      Loading
      EVIDENCE / COMPARISON

      Benchmarks

      Explore every benchmark

      Loading

      These charts contain model-linked results. The charts above also include unlinked source profiles. Compare scores within each evaluation. Model quality, usage, transparency and system performance measure different things.

      Install TensorTrace