DeepSeek: DeepSeek V3 0324
DeepSeek V3, a 685B-parameter, mixture-of-experts model, is the latest iteration of the flagship chat model family from the DeepSeek team. It succeeds the [DeepSeek V3](/deepseek/deepseek-chat-v3) mo…
deepseek/deepseek-chat-v3-0324
DeepSeek V3, a 685B-parameter, mixture-of-experts model, is the latest iteration of the flagship chat model family from the DeepSeek team. It succeeds the [DeepSeek V3](/deepseek/deepseek-chat-v3) model and performs really well...
Specifications & capabilities
- Input
- text
- Output
- text
- Context window · tokens
- 163840
- Knowledge cutoff
- 2024-07-31
- Published model ID
- DeepSeek-V3-0324
- Capabilities
- promptCaching, structuredOutputs, toolCalling
Evaluations
1–20 / 84
0.812
- source id
- llmStatsArchive
- leaderboard name
- MMLU-Pro · Raw
- dimension
- general
- metric
- MMLU-Pro · Raw
- rank
- 1
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- MMLU-Pro
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- categories
- language
- reasoning
- math
- general
- created_at
- 2025-07-19T19:56:11.408351+00:00
- paper_link
- https://arxiv.org/abs/2406.01574
- updated_at
- 2025-07-19T19:56:11.408351+00:00
- description
- A more robust and challenging multi-task language understanding benchmark that extends MMLU by expanding multiple-choice options from 4 to 10, eliminating trivial questions, and focusing on reasoning-intensive tasks. Features over 12,000 curated questions across 14 domains and causes a 16-33% accuracy drop compared to original MMLU.
- benchmark_id
- mmlu-pro
- multilingual
- false
- analysis_method
- Exact Match
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- MMLU-Pro · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/deepseek/models/deepseek-v3-0324/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://api-docs.deepseek.com/news/news250325
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- deepseek/deepseek-v3-0324
- metric_population
- 2
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- deepseek-v3-0324
Original result ↗0.812
- source id
- llmStatsArchive
- leaderboard name
- MMLU-Pro · Normalized (reported)
- dimension
- general
- metric
- MMLU-Pro · Normalized (reported)
- rank
- 1
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- MMLU-Pro
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- categories
- language
- reasoning
- math
- general
- created_at
- 2025-07-19T19:56:11.408351+00:00
- paper_link
- https://arxiv.org/abs/2406.01574
- updated_at
- 2025-07-19T19:56:11.408351+00:00
- description
- A more robust and challenging multi-task language understanding benchmark that extends MMLU by expanding multiple-choice options from 4 to 10, eliminating trivial questions, and focusing on reasoning-intensive tasks. Features over 12,000 curated questions across 14 domains and causes a 16-33% accuracy drop compared to original MMLU.
- benchmark_id
- mmlu-pro
- multilingual
- false
- analysis_method
- Exact Match
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- MMLU-Pro · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/deepseek/models/deepseek-v3-0324/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://api-docs.deepseek.com/news/news250325
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- deepseek/deepseek-v3-0324
- metric_population
- 2
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- deepseek-v3-0324
Original result ↗0.684
- source id
- llmStatsArchive
- leaderboard name
- GPQA · Raw
- dimension
- general
- metric
- GPQA · Raw
- rank
- 3
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- GPQA
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:11.588605+00:00
- paper_link
- https://arxiv.org/abs/2311.12022
- updated_at
- 2025-07-19T19:56:11.588605+00:00
- description
- A challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry. Questions are Google-proof and extremely difficult, with PhD experts reaching 65% accuracy.
- benchmark_id
- gpqa
- multilingual
- false
- analysis_method
- Pass@1
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- GPQA · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/deepseek/models/deepseek-v3-0324/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://api-docs.deepseek.com/news/news250325
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- deepseek/deepseek-v3-0324
- metric_population
- 5
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- deepseek-v3-0324
Original result ↗0.684
- source id
- llmStatsArchive
- leaderboard name
- GPQA · Normalized (reported)
- dimension
- general
- metric
- GPQA · Normalized (reported)
- rank
- 3
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- GPQA
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:11.588605+00:00
- paper_link
- https://arxiv.org/abs/2311.12022
- updated_at
- 2025-07-19T19:56:11.588605+00:00
- description
- A challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry. Questions are Google-proof and extremely difficult, with PhD experts reaching 65% accuracy.
- benchmark_id
- gpqa
- multilingual
- false
- analysis_method
- Pass@1
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- GPQA · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/deepseek/models/deepseek-v3-0324/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://api-docs.deepseek.com/news/news250325
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- deepseek/deepseek-v3-0324
- metric_population
- 5
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- deepseek-v3-0324
Original result ↗0.94
- source id
- llmStatsArchive
- leaderboard name
- MATH-500 · Raw
- dimension
- general
- metric
- MATH-500 · Raw
- rank
- 4
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- MATH-500
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:12.027850+00:00
- paper_link
- https://arxiv.org/abs/2103.03874
- updated_at
- 2025-07-19T19:56:12.027850+00:00
- description
- MATH-500 is a subset of the MATH dataset containing 500 challenging competition mathematics problems from AMC 10, AMC 12, AIME, and other mathematics competitions. Each problem includes full step-by-step solutions and spans multiple difficulty levels across seven mathematical subjects including Prealgebra, Algebra, Number Theory, Counting and Probability, Geometry, Intermediate Algebra, and Precalculus.
- benchmark_id
- math-500
- multilingual
- false
- analysis_method
- Pass@1
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- MATH-500 · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/deepseek/models/deepseek-v3-0324/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://api-docs.deepseek.com/news/news250325
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- deepseek/deepseek-v3-0324
- metric_population
- 8
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- deepseek-v3-0324
Original result ↗0.94
- source id
- llmStatsArchive
- leaderboard name
- MATH-500 · Normalized (reported)
- dimension
- general
- metric
- MATH-500 · Normalized (reported)
- rank
- 4
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- MATH-500
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:12.027850+00:00
- paper_link
- https://arxiv.org/abs/2103.03874
- updated_at
- 2025-07-19T19:56:12.027850+00:00
- description
- MATH-500 is a subset of the MATH dataset containing 500 challenging competition mathematics problems from AMC 10, AMC 12, AIME, and other mathematics competitions. Each problem includes full step-by-step solutions and spans multiple difficulty levels across seven mathematical subjects including Prealgebra, Algebra, Number Theory, Counting and Probability, Geometry, Intermediate Algebra, and Precalculus.
- benchmark_id
- math-500
- multilingual
- false
- analysis_method
- Pass@1
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- MATH-500 · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/deepseek/models/deepseek-v3-0324/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://api-docs.deepseek.com/news/news250325
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- deepseek/deepseek-v3-0324
- metric_population
- 8
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- deepseek-v3-0324
Original result ↗86.783%
- source id
- liveBench
- leaderboard name
- LiveBench Instruction Following
- dimension
- general
- metric
- Average score
- rank
- 4
- sample size
- 50
- evaluated at
- 2025-03-24T21:27:03.706Z
- source model id
- deepseek-v3-0324
Original result ↗0.594
- source id
- llmStatsArchive
- leaderboard name
- AIME 2024 · Normalized (reported)
- dimension
- general
- metric
- AIME 2024 · Normalized (reported)
- rank
- 4
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- AIME 2024
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:11.941652+00:00
- paper_link
- https://arxiv.org/html/2503.21380v2
- updated_at
- 2025-09-30T00:00:00.000000+00:00
- description
- American Invitational Mathematics Examination 2024, consisting of 30 challenging mathematical reasoning problems from AIME I and AIME II competitions. Each problem requires an integer answer between 0-999 and tests advanced mathematical reasoning across algebra, geometry, combinatorics, and number theory. Used as a benchmark for evaluating mathematical reasoning capabilities in large language models at Olympiad-level difficulty.
- benchmark_id
- aime-2024
- multilingual
- false
- analysis_method
- Pass@1
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- AIME 2024 · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/deepseek/models/deepseek-v3-0324/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://api-docs.deepseek.com/news/news250325
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- deepseek/deepseek-v3-0324
- metric_population
- 5
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- deepseek-v3-0324
Original result ↗0.594
- source id
- llmStatsArchive
- leaderboard name
- AIME 2024 · Raw
- dimension
- general
- metric
- AIME 2024 · Raw
- rank
- 4
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- AIME 2024
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:11.941652+00:00
- paper_link
- https://arxiv.org/html/2503.21380v2
- updated_at
- 2025-09-30T00:00:00.000000+00:00
- description
- American Invitational Mathematics Examination 2024, consisting of 30 challenging mathematical reasoning problems from AIME I and AIME II competitions. Each problem requires an integer answer between 0-999 and tests advanced mathematical reasoning across algebra, geometry, combinatorics, and number theory. Used as a benchmark for evaluating mathematical reasoning capabilities in large language models at Olympiad-level difficulty.
- benchmark_id
- aime-2024
- multilingual
- false
- analysis_method
- Pass@1
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- AIME 2024 · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/deepseek/models/deepseek-v3-0324/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://api-docs.deepseek.com/news/news250325
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- deepseek/deepseek-v3-0324
- metric_population
- 5
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- deepseek-v3-0324
Original result ↗0.492
- source id
- llmStatsArchive
- leaderboard name
- LiveCodeBench · Normalized (reported)
- dimension
- general
- metric
- LiveCodeBench · Normalized (reported)
- rank
- 5
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- LiveCodeBench
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:13.292229+00:00
- paper_link
- https://arxiv.org/abs/2403.07974
- updated_at
- 2025-07-19T19:56:13.292229+00:00
- description
- LiveCodeBench is a holistic and contamination-free evaluation benchmark for large language models for code. It continuously collects new problems from programming contests (LeetCode, AtCoder, CodeForces) and evaluates four different scenarios: code generation, self-repair, code execution, and test output prediction. Problems are annotated with release dates to enable evaluation on unseen problems released after a model's training cutoff.
- benchmark_id
- livecodebench
- multilingual
- false
- analysis_method
- Pass@1
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- LiveCodeBench · Normalized (reported)
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/deepseek/models/deepseek-v3-0324/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://api-docs.deepseek.com/news/news250325
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- deepseek/deepseek-v3-0324
- metric_population
- 9
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- deepseek-v3-0324
Original result ↗0.492
- source id
- llmStatsArchive
- leaderboard name
- LiveCodeBench · Raw
- dimension
- general
- metric
- LiveCodeBench · Raw
- rank
- 5
- aggregation scope
- status
- archived
- license
- CC-BY-4.0
- direction
- higherIsBetter
- conditions
- language
- en
- benchmark
- name
- LiveCodeBench
- language
- en
- modality
- text
- verified
- false
- max_score
- 1
- created_at
- 2025-07-19T19:56:13.292229+00:00
- paper_link
- https://arxiv.org/abs/2403.07974
- updated_at
- 2025-07-19T19:56:13.292229+00:00
- description
- LiveCodeBench is a holistic and contamination-free evaluation benchmark for large language models for code. It continuously collects new problems from programming contests (LeetCode, AtCoder, CodeForces) and evaluates four different scenarios: code generation, self-repair, code execution, and test output prediction. Problems are annotated with release dates to enable evaluation on unseen problems released after a model's training cutoff.
- benchmark_id
- livecodebench
- multilingual
- false
- analysis_method
- Pass@1
- is_self_reported
- true
- verified_by_llmstats
- false
- attribution
- Jonathan Chavez Tamales and LLM Stats contributors (copyright 2024 jc); TensorTrace structures original results and derives scoped ranks.
- license_url
- https://creativecommons.org/licenses/by/4.0/
- metric_kind
- LiveCodeBench · Raw
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- source_file
- data/organizations/deepseek/models/deepseek-v3-0324/benchmarks.json
- archive_kind
- llmStats
- evidence_url
- https://api-docs.deepseek.com/news/news250325
- git_revision
- 872b75f63b8d853c46ed17856424029a0c47b1d2
- published_at
- 2025-10-24T17:47:59Z
- snapshot_date
- 2025-10-24T17:47:59Z
- source_revision
- 6e27aa41f51fa8cb327602349240077d1c7acae9fead8ba2f07db0d549927ddc
- publisher_anchor
- deepseek/deepseek-v3-0324
- metric_population
- 9
- current_catalog_url
- https://llm-stats.com/
- benchmark_evaluation
- true
- source model id
- deepseek-v3-0324
Original result ↗71.094%
- source id
- liveBench
- leaderboard name
- LiveBench Coding
- dimension
- coding
- metric
- Average score
- rank
- 9
- sample size
- 128
- evaluated at
- 2025-03-24T21:27:01.956Z
- source model id
- deepseek-v3-0324
Original result ↗64.436%
- source id
- openRouterBenchmarks
- leaderboard name
- GPQA Diamond · Accuracy · Crusoe
- dimension
- knowledge
- metric
- GPQA Diamond · Accuracy
- rank
- 13
- sample size
- 396
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 2
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- gpqa_diamond
- name
- GPQA Diamond
- record_kind
- aggregate
- provider_name
- Crusoe
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-08-29T16:10:45.295Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- GPQA Diamond · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 396
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/gpqa-diamond
- measurements
- accuracy
- 0.6443605
- accuracy_std_dev
- 0.007996499999999962
- cost_per_task_usd
- 0.0038927146464646464
- output_tokens_per_task
- 2359.770202020202
- latency_seconds_per_task
- 63.45661868686869
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 15
- benchmark_evaluation
- true
- openrouter_permaslug
- deepseek/deepseek-chat-v3-0324
- source model id
- deepseek/deepseek-chat-v3-0324
Original result ↗60.106%
- source id
- liveBench
- leaderboard name
- LiveBench Overall
- dimension
- general
- metric
- Average score
- rank
- 19
- sample size
- 418
- evaluated at
- 2025-04-01T22:00:47.885Z
- source model id
- deepseek-v3-0324
Original result ↗57.099%
- source id
- openRouterBenchmarks
- leaderboard name
- GPQA Diamond · Accuracy · GMICloud
- dimension
- knowledge
- metric
- GPQA Diamond · Accuracy
- rank
- 26
- sample size
- 986
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 5
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- gpqa_diamond
- name
- GPQA Diamond
- record_kind
- aggregate
- provider_name
- GMICloud
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-11-10T06:14:22.398Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- GPQA Diamond · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 986
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/gpqa-diamond
- measurements
- accuracy
- 0.5709940405679513
- accuracy_std_dev
- 0.154429513944712
- cost_per_task_usd
- 0.0030763367139959434
- output_tokens_per_task
- 2517.3924949290063
- latency_seconds_per_task
- 63.55328093306288
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 26
- benchmark_evaluation
- true
- openrouter_permaslug
- deepseek/deepseek-chat-v3-0324
- source model id
- deepseek/deepseek-chat-v3-0324
Original result ↗45%
- source id
- openRouterBenchmarks
- leaderboard name
- tau2-bench Verified Airline · Accuracy · SiliconFlow
- dimension
- agentic
- metric
- tau2-bench Verified Airline · Accuracy
- rank
- 26
- sample size
- 100
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 2
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- tau_bench_verified_airline
- name
- tau2-bench Verified Airline
- record_kind
- aggregate
- provider_name
- SiliconFlow
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-08-29T16:10:09.206Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- tau2-bench Verified Airline · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 100
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/tau2-bench-airline
- measurements
- accuracy
- 0.45
- accuracy_std_dev
- 0.010000000000000009
- cost_per_task_usd
- 0.09892811
- output_tokens_per_task
- 23498.89
- latency_seconds_per_task
- 969.0375600000001
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 28
- benchmark_evaluation
- true
- openrouter_permaslug
- deepseek/deepseek-chat-v3-0324
- source model id
- deepseek/deepseek-chat-v3-0324
Original result ↗66.919%
- source id
- openRouterBenchmarks
- leaderboard name
- GPQA Diamond · Accuracy · SiliconFlow
- dimension
- knowledge
- metric
- GPQA Diamond · Accuracy
- rank
- 29
- sample size
- 396
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 2
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- gpqa_diamond
- name
- GPQA Diamond
- record_kind
- aggregate
- provider_name
- SiliconFlow
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-08-29T20:54:16.286Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- GPQA Diamond · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 396
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/gpqa-diamond
- measurements
- accuracy
- 0.6691920000000001
- accuracy_std_dev
- 0.007576000000000027
- cost_per_task_usd
- 0.002490050505050505
- output_tokens_per_task
- 2307.4141414141413
- latency_seconds_per_task
- 88.30796464646465
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 34
- benchmark_evaluation
- true
- openrouter_permaslug
- deepseek/deepseek-chat-v3-0324
- source model id
- deepseek/deepseek-chat-v3-0324
Original result ↗48.688%
- source id
- liveBench
- leaderboard name
- LiveBench Language
- dimension
- intelligence
- metric
- Average score
- rank
- 29
- sample size
- 240
- evaluated at
- 2025-04-01T22:00:47.885Z
- source model id
- deepseek-v3-0324
Original result ↗39.333%
- source id
- openRouterBenchmarks
- leaderboard name
- tau2-bench Verified Airline · Accuracy · DeepInfra
- dimension
- agentic
- metric
- tau2-bench Verified Airline · Accuracy
- rank
- 58
- sample size
- 50
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 1
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- tau_bench_verified_airline
- name
- tau2-bench Verified Airline
- record_kind
- aggregate
- provider_name
- DeepInfra
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-08-22T16:13:43.431Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- tau2-bench Verified Airline · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 50
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/tau2-bench-airline
- measurements
- accuracy
- 0.393333
- cost_per_task_usd
- 0.03258216
- output_tokens_per_task
- 4792.38
- latency_seconds_per_task
- 687.2739200000001
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 63
- benchmark_evaluation
- true
- openrouter_permaslug
- deepseek/deepseek-chat-v3-0324
- source model id
- deepseek/deepseek-chat-v3-0324
Original result ↗11.532%
- source id
- openRouterBenchmarks
- leaderboard name
- GPQA Diamond · Accuracy · Novita
- dimension
- knowledge
- metric
- GPQA Diamond · Accuracy
- rank
- 59
- sample size
- 396
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 2
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- gpqa_diamond
- name
- GPQA Diamond
- record_kind
- aggregate
- provider_name
- Novita
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-08-29T18:43:09.114Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- GPQA Diamond · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 396
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/gpqa-diamond
- measurements
- accuracy
- 0.11531999999999999
- accuracy_std_dev
- 0.027778000000000004
- cost_per_task_usd
- 0.0013887626262626262
- output_tokens_per_task
- 1211.2525252525252
- latency_seconds_per_task
- 56.176383838383835
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 59
- benchmark_evaluation
- true
- openrouter_permaslug
- deepseek/deepseek-chat-v3-0324
- source model id
- deepseek/deepseek-chat-v3-0324
Original result ↗Published prices
1–3 / 3
$0.11
- Inference provider
- openRouter
- Platform
- OpenRouter
- Unit
- cache_read_tokens_per_million
- Currency
- USD
$0.29
- Inference provider
- openRouter
- Platform
- OpenRouter
- Unit
- input_tokens_per_million
- Currency
- USD
$1.14
- Inference provider
- openRouter
- Platform
- OpenRouter
- Unit
- output_tokens_per_million
- Currency
- USD
Provider specifications & access
1–2 / 2
- source model id
- publishers/deepseek-ai/models/deepseek-v3@deepseek-v3-0324
- hosting
- vertex-self-deployed
- source model id
- publishers/hf-deepseek-ai/models/deepseek-v3-0324@001
- hosting
- vertex-self-deployed
Safety assessments
1–1 / 1
32 / 100
- source id
- stanfordFMTI
- canonical model id
- deepseek/deepseek-chat-v3-0324
- taxonomy
- transparency
- hazard category
- transparency
- hazard name
- Organization transparency
- policy area
- DeepSeek
- assessment type
- organization-level disclosure
- metric
- disclosure_score
- result label
- 32 / 100
- sample size
- 100
- updated at
- 2026-11-09T18:08:05.227Z
Source specification observations
1–3 / 3
- field
- context_length
- value
- 163840
- normalized value
- 163840
- scope
- source-model
- source id
- openRouter
- confidence
- 1
- field
- knowledge_cutoff
- reported field
- knowledge_cutoff
- value
- 2024-07-31
- normalized value
- 2024-07-31
- precision
- day
- scope
- source-model
- source id
- openRouter
- confidence
- 1
- field
- max_output_tokens
- value
- 115200
- normalized value
- 115200
- scope
- source-model
- source id
- openRouter
- confidence
- 1
Hugging Face model card
1–1 / 1
- downloads
- 829640
- likes
- 3222
- pipeline tag
- text-generation
- library name
- transformers
- License
- mit
Explore more models