DeepSeek: DeepSeek V3 0324

DeepSeek V3, a 685B-parameter, mixture-of-experts model, is the latest iteration of the flagship chat model family from the DeepSeek team. It succeeds the [DeepSeek V3](/deepseek/deepseek-chat-v3) mo…

deepseek/deepseek-chat-v3-0324

DeepSeek V3, a 685B-parameter, mixture-of-experts model, is the latest iteration of the flagship chat model family from the DeepSeek team. It succeeds the [DeepSeek V3](/deepseek/deepseek-chat-v3) model and performs really well...

Specifications & capabilities

Input
text
Output
text
Context window · tokens
163840
Knowledge cutoff
2024-07-31
Published model ID
DeepSeek-V3-0324
Capabilities
promptCaching, structuredOutputs, toolCalling

Evaluations

1–20 / 84

LLM Stats · GitHub archive

0.812

source id
llmStatsArchive
leaderboard name
MMLU-Pro · Raw
dimension
general
metric
MMLU-Pro · Raw
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
MMLU-Pro
language
en
modality
text
verified
false
max_score
1
categories
  • language
  • reasoning
  • math
  • general
created_at
2025-07-19T19:56:11.408351+00:00
paper_link
https://arxiv.org/abs/2406.01574
updated_at
2025-07-19T19:56:11.408351+00:00
description
A more robust and challenging multi-task language understanding benchmark that extends MMLU by expanding multiple-choice options from 4 to 10, eliminating trivial questions, and focusing on reasoning-intensive tasks. Features over 12,000 curated questions across 14 domains and causes a 16-33% accuracy drop compared to original MMLU.
benchmark_id
mmlu-pro
multilingual
false
analysis_method
Exact Match
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
MMLU-Pro · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/deepseek/models/deepseek-v3-0324/benchmarks.json
archive_kind
llmStats
evidence_url
https://api-docs.deepseek.com/news/news250325
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
deepseek/deepseek-v3-0324
metric_population
2
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
deepseek-v3-0324
Original result ↗

LLM Stats · GitHub archive

0.812

source id
llmStatsArchive
leaderboard name
MMLU-Pro · Normalized (reported)
dimension
general
metric
MMLU-Pro · Normalized (reported)
rank
1
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
MMLU-Pro
language
en
modality
text
verified
false
max_score
1
categories
  • language
  • reasoning
  • math
  • general
created_at
2025-07-19T19:56:11.408351+00:00
paper_link
https://arxiv.org/abs/2406.01574
updated_at
2025-07-19T19:56:11.408351+00:00
description
A more robust and challenging multi-task language understanding benchmark that extends MMLU by expanding multiple-choice options from 4 to 10, eliminating trivial questions, and focusing on reasoning-intensive tasks. Features over 12,000 curated questions across 14 domains and causes a 16-33% accuracy drop compared to original MMLU.
benchmark_id
mmlu-pro
multilingual
false
analysis_method
Exact Match
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
MMLU-Pro · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/deepseek/models/deepseek-v3-0324/benchmarks.json
archive_kind
llmStats
evidence_url
https://api-docs.deepseek.com/news/news250325
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
deepseek/deepseek-v3-0324
metric_population
2
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
deepseek-v3-0324
Original result ↗

LLM Stats · GitHub archive

0.684

source id
llmStatsArchive
leaderboard name
GPQA · Raw
dimension
general
metric
GPQA · Raw
rank
3
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
GPQA
language
en
modality
text
verified
false
max_score
1
categories
  • reasoning
  • general
created_at
2025-07-19T19:56:11.588605+00:00
paper_link
https://arxiv.org/abs/2311.12022
updated_at
2025-07-19T19:56:11.588605+00:00
description
A challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry. Questions are Google-proof and extremely difficult, with PhD experts reaching 65% accuracy.
benchmark_id
gpqa
multilingual
false
analysis_method
Pass@1
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
GPQA · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/deepseek/models/deepseek-v3-0324/benchmarks.json
archive_kind
llmStats
evidence_url
https://api-docs.deepseek.com/news/news250325
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
deepseek/deepseek-v3-0324
metric_population
5
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
deepseek-v3-0324
Original result ↗

LLM Stats · GitHub archive

0.684

source id
llmStatsArchive
leaderboard name
GPQA · Normalized (reported)
dimension
general
metric
GPQA · Normalized (reported)
rank
3
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
GPQA
language
en
modality
text
verified
false
max_score
1
categories
  • reasoning
  • general
created_at
2025-07-19T19:56:11.588605+00:00
paper_link
https://arxiv.org/abs/2311.12022
updated_at
2025-07-19T19:56:11.588605+00:00
description
A challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry. Questions are Google-proof and extremely difficult, with PhD experts reaching 65% accuracy.
benchmark_id
gpqa
multilingual
false
analysis_method
Pass@1
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
GPQA · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/deepseek/models/deepseek-v3-0324/benchmarks.json
archive_kind
llmStats
evidence_url
https://api-docs.deepseek.com/news/news250325
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
deepseek/deepseek-v3-0324
metric_population
5
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
deepseek-v3-0324
Original result ↗

LLM Stats · GitHub archive

0.94

source id
llmStatsArchive
leaderboard name
MATH-500 · Raw
dimension
general
metric
MATH-500 · Raw
rank
4
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
MATH-500
language
en
modality
text
verified
false
max_score
1
categories
  • math
  • reasoning
created_at
2025-07-19T19:56:12.027850+00:00
paper_link
https://arxiv.org/abs/2103.03874
updated_at
2025-07-19T19:56:12.027850+00:00
description
MATH-500 is a subset of the MATH dataset containing 500 challenging competition mathematics problems from AMC 10, AMC 12, AIME, and other mathematics competitions. Each problem includes full step-by-step solutions and spans multiple difficulty levels across seven mathematical subjects including Prealgebra, Algebra, Number Theory, Counting and Probability, Geometry, Intermediate Algebra, and Precalculus.
benchmark_id
math-500
multilingual
false
analysis_method
Pass@1
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
MATH-500 · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/deepseek/models/deepseek-v3-0324/benchmarks.json
archive_kind
llmStats
evidence_url
https://api-docs.deepseek.com/news/news250325
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
deepseek/deepseek-v3-0324
metric_population
8
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
deepseek-v3-0324
Original result ↗

LLM Stats · GitHub archive

0.94

source id
llmStatsArchive
leaderboard name
MATH-500 · Normalized (reported)
dimension
general
metric
MATH-500 · Normalized (reported)
rank
4
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
MATH-500
language
en
modality
text
verified
false
max_score
1
categories
  • math
  • reasoning
created_at
2025-07-19T19:56:12.027850+00:00
paper_link
https://arxiv.org/abs/2103.03874
updated_at
2025-07-19T19:56:12.027850+00:00
description
MATH-500 is a subset of the MATH dataset containing 500 challenging competition mathematics problems from AMC 10, AMC 12, AIME, and other mathematics competitions. Each problem includes full step-by-step solutions and spans multiple difficulty levels across seven mathematical subjects including Prealgebra, Algebra, Number Theory, Counting and Probability, Geometry, Intermediate Algebra, and Precalculus.
benchmark_id
math-500
multilingual
false
analysis_method
Pass@1
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
MATH-500 · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/deepseek/models/deepseek-v3-0324/benchmarks.json
archive_kind
llmStats
evidence_url
https://api-docs.deepseek.com/news/news250325
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
deepseek/deepseek-v3-0324
metric_population
8
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
deepseek-v3-0324
Original result ↗

LiveBench

86.783%

source id
liveBench
leaderboard name
LiveBench Instruction Following
dimension
general
metric
Average score
rank
4
sample size
50
evaluated at
2025-03-24T21:27:03.706Z
source model id
deepseek-v3-0324
Original result ↗

LLM Stats · GitHub archive

0.594

source id
llmStatsArchive
leaderboard name
AIME 2024 · Normalized (reported)
dimension
general
metric
AIME 2024 · Normalized (reported)
rank
4
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
AIME 2024
language
en
modality
text
verified
false
max_score
1
categories
  • math
  • reasoning
created_at
2025-07-19T19:56:11.941652+00:00
paper_link
https://arxiv.org/html/2503.21380v2
updated_at
2025-09-30T00:00:00.000000+00:00
description
American Invitational Mathematics Examination 2024, consisting of 30 challenging mathematical reasoning problems from AIME I and AIME II competitions. Each problem requires an integer answer between 0-999 and tests advanced mathematical reasoning across algebra, geometry, combinatorics, and number theory. Used as a benchmark for evaluating mathematical reasoning capabilities in large language models at Olympiad-level difficulty.
benchmark_id
aime-2024
multilingual
false
analysis_method
Pass@1
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
AIME 2024 · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/deepseek/models/deepseek-v3-0324/benchmarks.json
archive_kind
llmStats
evidence_url
https://api-docs.deepseek.com/news/news250325
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
deepseek/deepseek-v3-0324
metric_population
5
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
deepseek-v3-0324
Original result ↗

LLM Stats · GitHub archive

0.594

source id
llmStatsArchive
leaderboard name
AIME 2024 · Raw
dimension
general
metric
AIME 2024 · Raw
rank
4
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
AIME 2024
language
en
modality
text
verified
false
max_score
1
categories
  • math
  • reasoning
created_at
2025-07-19T19:56:11.941652+00:00
paper_link
https://arxiv.org/html/2503.21380v2
updated_at
2025-09-30T00:00:00.000000+00:00
description
American Invitational Mathematics Examination 2024, consisting of 30 challenging mathematical reasoning problems from AIME I and AIME II competitions. Each problem requires an integer answer between 0-999 and tests advanced mathematical reasoning across algebra, geometry, combinatorics, and number theory. Used as a benchmark for evaluating mathematical reasoning capabilities in large language models at Olympiad-level difficulty.
benchmark_id
aime-2024
multilingual
false
analysis_method
Pass@1
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
AIME 2024 · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/deepseek/models/deepseek-v3-0324/benchmarks.json
archive_kind
llmStats
evidence_url
https://api-docs.deepseek.com/news/news250325
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
deepseek/deepseek-v3-0324
metric_population
5
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
deepseek-v3-0324
Original result ↗

LLM Stats · GitHub archive

0.492

source id
llmStatsArchive
leaderboard name
LiveCodeBench · Normalized (reported)
dimension
general
metric
LiveCodeBench · Normalized (reported)
rank
5
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
LiveCodeBench
language
en
modality
text
verified
false
max_score
1
categories
  • reasoning
  • general
  • code
created_at
2025-07-19T19:56:13.292229+00:00
paper_link
https://arxiv.org/abs/2403.07974
updated_at
2025-07-19T19:56:13.292229+00:00
description
LiveCodeBench is a holistic and contamination-free evaluation benchmark for large language models for code. It continuously collects new problems from programming contests (LeetCode, AtCoder, CodeForces) and evaluates four different scenarios: code generation, self-repair, code execution, and test output prediction. Problems are annotated with release dates to enable evaluation on unseen problems released after a model's training cutoff.
benchmark_id
livecodebench
multilingual
false
analysis_method
Pass@1
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
LiveCodeBench · Normalized (reported)
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/deepseek/models/deepseek-v3-0324/benchmarks.json
archive_kind
llmStats
evidence_url
https://api-docs.deepseek.com/news/news250325
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
deepseek/deepseek-v3-0324
metric_population
9
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
deepseek-v3-0324
Original result ↗

LLM Stats · GitHub archive

0.492

source id
llmStatsArchive
leaderboard name
LiveCodeBench · Raw
dimension
general
metric
LiveCodeBench · Raw
rank
5
aggregation scope
status
archived
license
CC-BY-4.0
direction
higherIsBetter
conditions
language
en
benchmark
name
LiveCodeBench
language
en
modality
text
verified
false
max_score
1
categories
  • reasoning
  • general
  • code
created_at
2025-07-19T19:56:13.292229+00:00
paper_link
https://arxiv.org/abs/2403.07974
updated_at
2025-07-19T19:56:13.292229+00:00
description
LiveCodeBench is a holistic and contamination-free evaluation benchmark for large language models for code. It continuously collects new problems from programming contests (LeetCode, AtCoder, CodeForces) and evaluates four different scenarios: code generation, self-repair, code execution, and test output prediction. Problems are annotated with release dates to enable evaluation on unseen problems released after a model's training cutoff.
benchmark_id
livecodebench
multilingual
false
analysis_method
Pass@1
is_self_reported
true
verified_by_llmstats
false
attribution
Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
license_url
https://creativecommons.org/licenses/by/4.0/
metric_kind
LiveCodeBench · Raw
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
data/organizations/deepseek/models/deepseek-v3-0324/benchmarks.json
archive_kind
llmStats
evidence_url
https://api-docs.deepseek.com/news/news250325
git_revision
872b75f63b8d853c46ed17856424029a0c47b1d2
published_at
2025-10-24T17:47:59Z
snapshot_date
2025-10-24T17:47:59Z
source_revision
6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
publisher_anchor
deepseek/deepseek-v3-0324
metric_population
9
current_catalog_url
https://llm-stats.com/
benchmark_evaluation
true
source model id
deepseek-v3-0324
Original result ↗

LiveBench

71.094%

source id
liveBench
leaderboard name
LiveBench Coding
dimension
coding
metric
Average score
rank
9
sample size
128
evaluated at
2025-03-24T21:27:01.956Z
source model id
deepseek-v3-0324
Original result ↗

OpenRouter Benchmarks

64.436%

source id
openRouterBenchmarks
leaderboard name
GPQA Diamond · Accuracy · Crusoe
dimension
knowledge
metric
GPQA Diamond · Accuracy
rank
13
sample size
396
aggregation scope
status
published
harness
url
https://github.com/OpenRouterTeam/benchmark-harness
name
OpenRouter benchmark harness
direction
higherIsBetter
run_count
2
conditions
window
newest 90 days; successful visible nonzero runs
dataset
benchmark-leaderboard
benchmark
id
gpqa_diamond
name
GPQA Diamond
record_kind
aggregate
provider_name
Crusoe
attribution
Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
last_run_at
2026-08-29T16:10:45.295Z
license_url
https://creativecommons.org/licenses/by/4.0/
methodology
Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
metric_kind
GPQA Diamond · Accuracy
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
sample_size
396
source_file
benchmark-leaderboard.json
source_page
https://openrouter.ai/benchmarks/gpqa-diamond
measurements
accuracy
0.6443605
accuracy_std_dev
0.007996499999999962
cost_per_task_usd
0.0038927146464646464
output_tokens_per_task
2359.770202020202
latency_seconds_per_task
63.45661868686869
published_at
2026-11-16T01:46:02.653Z
snapshot_date
2026-11-16
source_revision
7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
transformations
TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
metric_population
15
benchmark_evaluation
true
openrouter_permaslug
deepseek/deepseek-chat-v3-0324
source model id
deepseek/deepseek-chat-v3-0324
Original result ↗

LiveBench

60.106%

source id
liveBench
leaderboard name
LiveBench Overall
dimension
general
metric
Average score
rank
19
sample size
418
evaluated at
2025-04-01T22:00:47.885Z
source model id
deepseek-v3-0324
Original result ↗

OpenRouter Benchmarks

57.099%

source id
openRouterBenchmarks
leaderboard name
GPQA Diamond · Accuracy · GMICloud
dimension
knowledge
metric
GPQA Diamond · Accuracy
rank
26
sample size
986
aggregation scope
status
published
harness
url
https://github.com/OpenRouterTeam/benchmark-harness
name
OpenRouter benchmark harness
direction
higherIsBetter
run_count
5
conditions
window
newest 90 days; successful visible nonzero runs
dataset
benchmark-leaderboard
benchmark
id
gpqa_diamond
name
GPQA Diamond
record_kind
aggregate
provider_name
GMICloud
attribution
Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
last_run_at
2026-11-10T06:14:22.398Z
license_url
https://creativecommons.org/licenses/by/4.0/
methodology
Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
metric_kind
GPQA Diamond · Accuracy
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
sample_size
986
source_file
benchmark-leaderboard.json
source_page
https://openrouter.ai/benchmarks/gpqa-diamond
measurements
accuracy
0.5709940405679513
accuracy_std_dev
0.154429513944712
cost_per_task_usd
0.0030763367139959434
output_tokens_per_task
2517.3924949290063
latency_seconds_per_task
63.55328093306288
published_at
2026-11-16T01:46:02.653Z
snapshot_date
2026-11-16
source_revision
7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
transformations
TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
metric_population
26
benchmark_evaluation
true
openrouter_permaslug
deepseek/deepseek-chat-v3-0324
source model id
deepseek/deepseek-chat-v3-0324
Original result ↗

OpenRouter Benchmarks

45%

source id
openRouterBenchmarks
leaderboard name
tau2-bench Verified Airline · Accuracy · SiliconFlow
dimension
agentic
metric
tau2-bench Verified Airline · Accuracy
rank
26
sample size
100
aggregation scope
status
published
harness
url
https://github.com/OpenRouterTeam/benchmark-harness
name
OpenRouter benchmark harness
direction
higherIsBetter
run_count
2
conditions
window
newest 90 days; successful visible nonzero runs
dataset
benchmark-leaderboard
benchmark
id
tau_bench_verified_airline
name
tau2-bench Verified Airline
record_kind
aggregate
provider_name
SiliconFlow
attribution
Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
last_run_at
2026-08-29T16:10:09.206Z
license_url
https://creativecommons.org/licenses/by/4.0/
methodology
Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
metric_kind
tau2-bench Verified Airline · Accuracy
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
sample_size
100
source_file
benchmark-leaderboard.json
source_page
https://openrouter.ai/benchmarks/tau2-bench-airline
measurements
accuracy
0.45
accuracy_std_dev
0.010000000000000009
cost_per_task_usd
0.09892811
output_tokens_per_task
23498.89
latency_seconds_per_task
969.0375600000001
published_at
2026-11-16T01:46:02.653Z
snapshot_date
2026-11-16
source_revision
7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
transformations
TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
metric_population
28
benchmark_evaluation
true
openrouter_permaslug
deepseek/deepseek-chat-v3-0324
source model id
deepseek/deepseek-chat-v3-0324
Original result ↗

OpenRouter Benchmarks

66.919%

source id
openRouterBenchmarks
leaderboard name
GPQA Diamond · Accuracy · SiliconFlow
dimension
knowledge
metric
GPQA Diamond · Accuracy
rank
29
sample size
396
aggregation scope
status
published
harness
url
https://github.com/OpenRouterTeam/benchmark-harness
name
OpenRouter benchmark harness
direction
higherIsBetter
run_count
2
conditions
window
newest 90 days; successful visible nonzero runs
dataset
benchmark-leaderboard
benchmark
id
gpqa_diamond
name
GPQA Diamond
record_kind
aggregate
provider_name
SiliconFlow
attribution
Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
last_run_at
2026-08-29T20:54:16.286Z
license_url
https://creativecommons.org/licenses/by/4.0/
methodology
Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
metric_kind
GPQA Diamond · Accuracy
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
sample_size
396
source_file
benchmark-leaderboard.json
source_page
https://openrouter.ai/benchmarks/gpqa-diamond
measurements
accuracy
0.6691920000000001
accuracy_std_dev
0.007576000000000027
cost_per_task_usd
0.002490050505050505
output_tokens_per_task
2307.4141414141413
latency_seconds_per_task
88.30796464646465
published_at
2026-11-16T01:46:02.653Z
snapshot_date
2026-11-16
source_revision
7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
transformations
TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
metric_population
34
benchmark_evaluation
true
openrouter_permaslug
deepseek/deepseek-chat-v3-0324
source model id
deepseek/deepseek-chat-v3-0324
Original result ↗

LiveBench

48.688%

source id
liveBench
leaderboard name
LiveBench Language
dimension
intelligence
metric
Average score
rank
29
sample size
240
evaluated at
2025-04-01T22:00:47.885Z
source model id
deepseek-v3-0324
Original result ↗

OpenRouter Benchmarks

39.333%

source id
openRouterBenchmarks
leaderboard name
tau2-bench Verified Airline · Accuracy · DeepInfra
dimension
agentic
metric
tau2-bench Verified Airline · Accuracy
rank
58
sample size
50
aggregation scope
status
published
harness
url
https://github.com/OpenRouterTeam/benchmark-harness
name
OpenRouter benchmark harness
direction
higherIsBetter
run_count
1
conditions
window
newest 90 days; successful visible nonzero runs
dataset
benchmark-leaderboard
benchmark
id
tau_bench_verified_airline
name
tau2-bench Verified Airline
record_kind
aggregate
provider_name
DeepInfra
attribution
Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
last_run_at
2026-08-22T16:13:43.431Z
license_url
https://creativecommons.org/licenses/by/4.0/
methodology
Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
metric_kind
tau2-bench Verified Airline · Accuracy
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
sample_size
50
source_file
benchmark-leaderboard.json
source_page
https://openrouter.ai/benchmarks/tau2-bench-airline
measurements
accuracy
0.393333
cost_per_task_usd
0.03258216
output_tokens_per_task
4792.38
latency_seconds_per_task
687.2739200000001
published_at
2026-11-16T01:46:02.653Z
snapshot_date
2026-11-16
source_revision
7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
transformations
TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
metric_population
63
benchmark_evaluation
true
openrouter_permaslug
deepseek/deepseek-chat-v3-0324
source model id
deepseek/deepseek-chat-v3-0324
Original result ↗

OpenRouter Benchmarks

11.532%

source id
openRouterBenchmarks
leaderboard name
GPQA Diamond · Accuracy · Novita
dimension
knowledge
metric
GPQA Diamond · Accuracy
rank
59
sample size
396
aggregation scope
status
published
harness
url
https://github.com/OpenRouterTeam/benchmark-harness
name
OpenRouter benchmark harness
direction
higherIsBetter
run_count
2
conditions
window
newest 90 days; successful visible nonzero runs
dataset
benchmark-leaderboard
benchmark
id
gpqa_diamond
name
GPQA Diamond
record_kind
aggregate
provider_name
Novita
attribution
Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
last_run_at
2026-08-29T18:43:09.114Z
license_url
https://creativecommons.org/licenses/by/4.0/
methodology
Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
metric_kind
GPQA Diamond · Accuracy
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
sample_size
396
source_file
benchmark-leaderboard.json
source_page
https://openrouter.ai/benchmarks/gpqa-diamond
measurements
accuracy
0.11531999999999999
accuracy_std_dev
0.027778000000000004
cost_per_task_usd
0.0013887626262626262
output_tokens_per_task
1211.2525252525252
latency_seconds_per_task
56.176383838383835
published_at
2026-11-16T01:46:02.653Z
snapshot_date
2026-11-16
source_revision
7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
transformations
TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
metric_population
59
benchmark_evaluation
true
openrouter_permaslug
deepseek/deepseek-chat-v3-0324
source model id
deepseek/deepseek-chat-v3-0324
Original result ↗

Published prices

1–3 / 3

Provider specifications & access

1–2 / 2

deepseek-ai/DeepSeek-V3-0324 ↗

source model id
publishers/deepseek-ai/models/deepseek-v3@deepseek-v3-0324
hosting
vertex-self-deployed
platforms
  • Google Cloud Vertex AI
architecture
input_modalities
  • text
output_modalities
  • text
capabilities

    deepseek-ai/DeepSeek-V3-0324 ↗

    source model id
    publishers/hf-deepseek-ai/models/deepseek-v3-0324@001
    hosting
    vertex-self-deployed
    platforms
    • Google Cloud Vertex AI
    architecture
    input_modalities
    • text
    output_modalities
    • text
    capabilities

      Safety assessments

      1–1 / 1

      Foundation Model Transparency Index 2025 ↗

      32 / 100

      source id
      stanfordFMTI
      canonical model id
      deepseek/deepseek-chat-v3-0324
      taxonomy
      transparency
      hazard category
      transparency
      hazard name
      Organization transparency
      policy area
      DeepSeek
      assessment type
      organization-level disclosure
      metric
      disclosure_score
      result label
      32 / 100
      sample size
      100
      updated at
      2026-11-09T18:08:05.227Z

      Source specification observations

      1–3 / 3

      openRouter ↗

      field
      context_length
      value
      163840
      normalized value
      163840
      scope
      source-model
      source id
      openRouter
      confidence
      1

      openRouter ↗

      field
      knowledge_cutoff
      reported field
      knowledge_cutoff
      value
      2024-07-31
      normalized value
      2024-07-31
      precision
      day
      scope
      source-model
      source id
      openRouter
      confidence
      1

      openRouter ↗

      field
      max_output_tokens
      value
      115200
      normalized value
      115200
      scope
      source-model
      source id
      openRouter
      confidence
      1

      Original sources

      1–15 / 15

      Hugging Face · deepseek-ai/DeepSeek-V3-0324 ↗

      Google Cloud · Model page ↗

      Google Cloud · Model page ↗

      OpenRouter · Model page ↗

      Google Cloud · Model documentation ↗

      arXiv · Paper · 2412.19437 ↗

      GitHub · Catalog ↗

      GitHub · Catalog ↗

      Google Cloud · Catalog ↗

      Hugging Face · Catalog ↗

      Hugging Face · Catalog ↗

      LMArena · Catalog ↗

      OpenRouter · Catalog ↗

      OpenRouter · Catalog ↗

      livebench.ai · Catalog ↗

      Source links

      1–10 / 10

      Hugging Face model card

      1–1 / 1

      Explore more models

      Models

      Loading
      EVIDENCE / COMPARISON

      Benchmarks

      Explore every benchmark

      Loading

      These charts contain model-linked results. The charts above also include unlinked source profiles. Compare scores within each evaluation. Model quality, usage, transparency and system performance measure different things.

      Install TensorTrace