NVIDIA: Nemotron 3.5 Lightning
NVIDIA Nemotron 3.5 Lightning is an open mixture-of-experts model from NVIDIA, with 3B active parameters out of 30B total. It is suited for high-throughput agentic workloads and specialized tasks tha…
nvidia/nemotron-3.5-lightning-20260807
NVIDIA Nemotron 3.5 Lightning is an open mixture-of-experts model from NVIDIA, with 3B active parameters out of 30B total. It is suited for high-throughput agentic workloads and specialized tasks that benefit from domain-specific customization.
Specifications & capabilities
- Input
- text
- Output
- text
- Context window · tokens
- 262144
- Published model ID
- NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16
- Capabilities
- promptCaching, reasoning, structuredOutputs, toolCalling
Description source ↗Evaluations
1–18 / 18
67.083%
- source id
- openRouterBenchmarks
- leaderboard name
- tau2-bench Verified Airline · Accuracy · Darkbloom
- dimension
- agentic
- metric
- tau2-bench Verified Airline · Accuracy
- rank
- 6
- sample size
- 200
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 4
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- tau_bench_verified_airline
- name
- tau2-bench Verified Airline
- record_kind
- aggregate
- provider_name
- Darkbloom
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-11-14T19:27:00.426Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- tau2-bench Verified Airline · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 200
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/tau2-bench-airline
- measurements
- accuracy
- 0.6708329999999999
- accuracy_std_dev
- 0.022776083947860723
- cost_per_task_usd
- 0.00710901
- output_tokens_per_task
- 20361.76
- latency_seconds_per_task
- 454.081565
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 8
- benchmark_evaluation
- true
- openrouter_permaslug
- nvidia/nemotron-3.5-lightning-20260807
- source model id
- nvidia/nemotron-3.5-lightning-20260807
Original result ↗65.124%
- source id
- openRouterBenchmarks
- leaderboard name
- GPQA Diamond · Accuracy · Darkbloom
- dimension
- knowledge
- metric
- GPQA Diamond · Accuracy
- rank
- 8
- sample size
- 2269
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 12
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- gpqa_diamond
- name
- GPQA Diamond
- record_kind
- aggregate
- provider_name
- Darkbloom
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-11-14T19:15:05.957Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- GPQA Diamond · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 2269
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/gpqa-diamond
- measurements
- accuracy
- 0.6512412503305419
- accuracy_std_dev
- 0.038999469240957485
- cost_per_task_usd
- 0.005451304539444689
- output_tokens_per_task
- 37464.12913177611
- latency_seconds_per_task
- 789.0263609519612
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 10
- benchmark_evaluation
- true
- openrouter_permaslug
- nvidia/nemotron-3.5-lightning-20260807
- source model id
- nvidia/nemotron-3.5-lightning-20260807
Original result ↗71.528%
- source id
- openRouterBenchmarks
- leaderboard name
- tau2-bench Verified Airline · Accuracy · Wafer
- dimension
- agentic
- metric
- tau2-bench Verified Airline · Accuracy
- rank
- 11
- sample size
- 48
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 1
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- tau_bench_verified_airline
- name
- tau2-bench Verified Airline
- record_kind
- aggregate
- provider_name
- Wafer
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-10-10T11:44:02.317Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- tau2-bench Verified Airline · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 48
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/tau2-bench-airline
- measurements
- accuracy
- 0.715278
- cost_per_task_usd
- 0.009582875
- output_tokens_per_task
- 18445.333333333332
- latency_seconds_per_task
- 164.1245
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 13
- benchmark_evaluation
- true
- openrouter_permaslug
- nvidia/nemotron-3.5-lightning-20260807
- source model id
- nvidia/nemotron-3.5-lightning-20260807
Original result ↗74.81%
- source id
- openRouterBenchmarks
- leaderboard name
- GPQA Diamond · Accuracy · Wafer
- dimension
- knowledge
- metric
- GPQA Diamond · Accuracy
- rank
- 13
- sample size
- 395
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 2
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- gpqa_diamond
- name
- GPQA Diamond
- record_kind
- aggregate
- provider_name
- Wafer
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-10-10T08:39:12.933Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- GPQA Diamond · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 395
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/gpqa-diamond
- measurements
- accuracy
- 0.7481013645569621
- accuracy_std_dev
- 0.004003499999999993
- cost_per_task_usd
- 0.010130237974683544
- output_tokens_per_task
- 60721.07341772152
- latency_seconds_per_task
- 298.77967341772154
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 13
- benchmark_evaluation
- true
- openrouter_permaslug
- nvidia/nemotron-3.5-lightning-20260807
- source model id
- nvidia/nemotron-3.5-lightning-20260807
Original result ↗74.018%
- source id
- openRouterBenchmarks
- leaderboard name
- GPQA Diamond · Accuracy · Io Net
- dimension
- knowledge
- metric
- GPQA Diamond · Accuracy
- rank
- 13
- sample size
- 2070
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 12
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- gpqa_diamond
- name
- GPQA Diamond
- record_kind
- aggregate
- provider_name
- Io Net
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-11-04T18:42:44.798Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- GPQA Diamond · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 2070
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/gpqa-diamond
- measurements
- accuracy
- 0.7401770526570047
- accuracy_std_dev
- 0.0865423410071625
- cost_per_task_usd
- 0.0039563893719806766
- output_tokens_per_task
- 24944.29468599034
- latency_seconds_per_task
- 552.5429028985508
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 15
- benchmark_evaluation
- true
- openrouter_permaslug
- nvidia/nemotron-3.5-lightning-20260807
- source model id
- nvidia/nemotron-3.5-lightning-20260807
Original result ↗65.354%
- source id
- openRouterBenchmarks
- leaderboard name
- tau2-bench Verified Airline · Accuracy · Io Net
- dimension
- agentic
- metric
- tau2-bench Verified Airline · Accuracy
- rank
- 14
- sample size
- 546
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 11
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- tau_bench_verified_airline
- name
- tau2-bench Verified Airline
- record_kind
- aggregate
- provider_name
- Io Net
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-11-04T19:03:50.422Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- tau2-bench Verified Airline · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 546
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/tau2-bench-airline
- measurements
- accuracy
- 0.6535409120879121
- accuracy_std_dev
- 0.03689102528724027
- cost_per_task_usd
- 0.012232007326007327
- output_tokens_per_task
- 22783.287545787545
- latency_seconds_per_task
- 1066.692706959707
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 15
- benchmark_evaluation
- true
- openrouter_permaslug
- nvidia/nemotron-3.5-lightning-20260807
- source model id
- nvidia/nemotron-3.5-lightning-20260807
Original result ↗68.125%
- source id
- openRouterBenchmarks
- leaderboard name
- tau2-bench Verified Airline · Accuracy · CoreWeave
- dimension
- agentic
- metric
- tau2-bench Verified Airline · Accuracy
- rank
- 17
- sample size
- 400
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 8
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- tau_bench_verified_airline
- name
- tau2-bench Verified Airline
- record_kind
- aggregate
- provider_name
- CoreWeave
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-10-31T11:46:38.147Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- tau2-bench Verified Airline · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 400
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/tau2-bench-airline
- measurements
- accuracy
- 0.68125
- accuracy_std_dev
- 0.0202715728731147
- cost_per_task_usd
- 0.0174918175
- output_tokens_per_task
- 21340.095
- latency_seconds_per_task
- 123.0915525
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 22
- benchmark_evaluation
- true
- openrouter_permaslug
- nvidia/nemotron-3.5-lightning-20260807
- source model id
- nvidia/nemotron-3.5-lightning-20260807
Original result ↗58.101%
- source id
- openRouterBenchmarks
- leaderboard name
- GPQA Diamond · Accuracy · Fireworks
- dimension
- knowledge
- metric
- GPQA Diamond · Accuracy
- rank
- 17
- sample size
- 395
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 2
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- gpqa_diamond
- name
- GPQA Diamond
- record_kind
- aggregate
- provider_name
- Fireworks
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-10-31T08:09:25.783Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- GPQA Diamond · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 395
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/gpqa-diamond
- measurements
- accuracy
- 0.5810130708860759
- accuracy_std_dev
- 0.012019500000000016
- cost_per_task_usd
- 0.01962629113924051
- output_tokens_per_task
- 97953.7417721519
- latency_seconds_per_task
- 376.7089949367089
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 17
- benchmark_evaluation
- true
- openrouter_permaslug
- nvidia/nemotron-3.5-lightning-20260807
- source model id
- nvidia/nemotron-3.5-lightning-20260807
Original result ↗73.336%
- source id
- openRouterBenchmarks
- leaderboard name
- GPQA Diamond · Accuracy · CoreWeave
- dimension
- knowledge
- metric
- GPQA Diamond · Accuracy
- rank
- 22
- sample size
- 2144
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 11
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- gpqa_diamond
- name
- GPQA Diamond
- record_kind
- aggregate
- provider_name
- CoreWeave
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-10-31T08:46:43.680Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- GPQA Diamond · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 2144
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/gpqa-diamond
- measurements
- accuracy
- 0.733364333488806
- accuracy_std_dev
- 0.03217972751863435
- cost_per_task_usd
- 0.010386112406716419
- output_tokens_per_task
- 48631.178638059704
- latency_seconds_per_task
- 251.82099860074626
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 27
- benchmark_evaluation
- true
- openrouter_permaslug
- nvidia/nemotron-3.5-lightning-20260807
- source model id
- nvidia/nemotron-3.5-lightning-20260807
Original result ↗72.718%
- source id
- openRouterBenchmarks
- leaderboard name
- GPQA Diamond · Accuracy · Phala
- dimension
- knowledge
- metric
- GPQA Diamond · Accuracy
- rank
- 22
- sample size
- 1154
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 6
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- gpqa_diamond
- name
- GPQA Diamond
- record_kind
- aggregate
- provider_name
- Phala
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-10-10T08:24:27.616Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- GPQA Diamond · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 1154
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/gpqa-diamond
- measurements
- accuracy
- 0.7271808509532062
- accuracy_std_dev
- 0.03540222199908109
- cost_per_task_usd
- 0.011152869150779895
- output_tokens_per_task
- 55506.62998266898
- latency_seconds_per_task
- 504.50158318890817
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 24
- benchmark_evaluation
- true
- openrouter_permaslug
- nvidia/nemotron-3.5-lightning-20260807
- source model id
- nvidia/nemotron-3.5-lightning-20260807
Original result ↗65.578%
- source id
- openRouterBenchmarks
- leaderboard name
- tau2-bench Verified Airline · Accuracy · Phala
- dimension
- agentic
- metric
- tau2-bench Verified Airline · Accuracy
- rank
- 22
- sample size
- 245
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 5
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- tau_bench_verified_airline
- name
- tau2-bench Verified Airline
- record_kind
- aggregate
- provider_name
- Phala
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-10-10T11:18:54.565Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- tau2-bench Verified Airline · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 245
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/tau2-bench-airline
- measurements
- accuracy
- 0.6557823102040816
- accuracy_std_dev
- 0.009318408855593308
- cost_per_task_usd
- 0.014797057142857142
- output_tokens_per_task
- 20245.579591836733
- latency_seconds_per_task
- 277.9072775510204
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 23
- benchmark_evaluation
- true
- openrouter_permaslug
- nvidia/nemotron-3.5-lightning-20260807
- source model id
- nvidia/nemotron-3.5-lightning-20260807
Original result ↗65.556%
- source id
- openRouterBenchmarks
- leaderboard name
- tau2-bench Verified Airline · Accuracy · DeepInfra
- dimension
- agentic
- metric
- tau2-bench Verified Airline · Accuracy
- rank
- 42
- sample size
- 450
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 9
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- tau_bench_verified_airline
- name
- tau2-bench Verified Airline
- record_kind
- aggregate
- provider_name
- DeepInfra
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-10-31T12:25:21.635Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- tau2-bench Verified Airline · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 450
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/tau2-bench-airline
- measurements
- accuracy
- 0.6555556666666666
- accuracy_std_dev
- 0.03807065880409451
- cost_per_task_usd
- 0.021039175555555557
- output_tokens_per_task
- 26269.908888888887
- latency_seconds_per_task
- 248.79159777777778
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 63
- benchmark_evaluation
- true
- openrouter_permaslug
- nvidia/nemotron-3.5-lightning-20260807
- source model id
- nvidia/nemotron-3.5-lightning-20260807
Original result ↗71.1%
- source id
- openRouterBenchmarks
- leaderboard name
- GPQA Diamond · Accuracy · DeepInfra
- dimension
- knowledge
- metric
- GPQA Diamond · Accuracy
- rank
- 48
- sample size
- 1966
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 10
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- gpqa_diamond
- name
- GPQA Diamond
- record_kind
- aggregate
- provider_name
- DeepInfra
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-10-31T08:15:24.314Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- GPQA Diamond · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 1966
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/gpqa-diamond
- measurements
- accuracy
- 0.7110037019328587
- accuracy_std_dev
- 0.031890632320636075
- cost_per_task_usd
- 0.0072013123092573746
- output_tokens_per_task
- 40164.048830111904
- latency_seconds_per_task
- 309.64626246185145
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 71
- benchmark_evaluation
- true
- openrouter_permaslug
- nvidia/nemotron-3.5-lightning-20260807
- source model id
- nvidia/nemotron-3.5-lightning-20260807
Original result ↗65.622%
- source id
- openRouterBenchmarks
- leaderboard name
- tau2-bench Verified Airline · Accuracy · auto-routing
- dimension
- agentic
- metric
- tau2-bench Verified Airline · Accuracy
- rank
- 92
- sample size
- 399
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 8
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- tau_bench_verified_airline
- name
- tau2-bench Verified Airline
- record_kind
- aggregate
- provider_name
- auto-routing
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-10-31T12:28:49.050Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- tau2-bench Verified Airline · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 399
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/tau2-bench-airline
- measurements
- accuracy
- 0.6562239298245615
- accuracy_std_dev
- 0.018946289616967236
- cost_per_task_usd
- 0.01692722807017544
- output_tokens_per_task
- 22811.754385964912
- latency_seconds_per_task
- 312.8047944862155
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 128
- benchmark_evaluation
- true
- openrouter_permaslug
- nvidia/nemotron-3.5-lightning-20260807
- source model id
- nvidia/nemotron-3.5-lightning-20260807
Original result ↗70.613%
- source id
- openRouterBenchmarks
- leaderboard name
- GPQA Diamond · Accuracy · auto-routing
- dimension
- knowledge
- metric
- GPQA Diamond · Accuracy
- rank
- 101
- sample size
- 1926
- aggregation scope
- status
- published
- harness
- url
- https://github.com/OpenRouterTeam/benchmark-harness
- name
- OpenRouter benchmark harness
- direction
- higherIsBetter
- run_count
- 10
- conditions
- window
- newest 90 days; successful visible nonzero runs
- dataset
- benchmark-leaderboard
- benchmark
- id
- gpqa_diamond
- name
- GPQA Diamond
- record_kind
- aggregate
- provider_name
- auto-routing
- attribution
- Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
- last_run_at
- 2026-10-31T08:03:09.796Z
- license_url
- https://creativecommons.org/licenses/by/4.0/
- methodology
- Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
- metric_kind
- GPQA Diamond · Accuracy
- rank_method
- competition rank among reported evaluations in identical conditions, including unlinked observations
- sample_size
- 1926
- source_file
- benchmark-leaderboard.json
- source_page
- https://openrouter.ai/benchmarks/gpqa-diamond
- measurements
- accuracy
- 0.7061267289719626
- accuracy_std_dev
- 0.026110787988875415
- cost_per_task_usd
- 0.00886934890965732
- output_tokens_per_task
- 45095.30893042575
- latency_seconds_per_task
- 374.4079574247145
- published_at
- 2026-11-16T01:46:02.653Z
- snapshot_date
- 2026-11-16
- source_revision
- 7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
- transformations
- TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
- metric_population
- 132
- benchmark_evaluation
- true
- openrouter_permaslug
- nvidia/nemotron-3.5-lightning-20260807
- source model id
- nvidia/nemotron-3.5-lightning-20260807
Original result ↗26.8
- source id
- artificialAnalysis
- dimension
- coding
- metric
- Coding Index
Original result ↗5.5
- source id
- artificialAnalysis
- dimension
- intelligence
- metric
- Intelligence Index
Original result ↗3.5
- source id
- artificialAnalysis
- dimension
- agentic
- metric
- Agentic Index
Original result ↗Published prices
1–5 / 5
$0.04
- Inference provider
- openRouter
- Platform
- OpenRouter
- Unit
- cache_read_tokens_per_million
- Currency
- USD
$0.07
- Inference provider
- openRouter
- Platform
- OpenRouter
- Unit
- input_tokens_per_million
- Currency
- USD
$0.20
- Inference provider
- openRouter
- Platform
- OpenRouter
- Unit
- output_tokens_per_million
- Currency
- USD
$0.00
- Inference provider
- openRouter
- Platform
- OpenRouter
- Unit
- input_tokens_per_million
- Currency
- USD
- Billing conditions
- tier
- free
$0.00
- Inference provider
- openRouter
- Platform
- OpenRouter
- Unit
- output_tokens_per_million
- Currency
- USD
- Billing conditions
- tier
- free
Provider specifications & access
1–6 / 6
- source model id
- azure-huggingface/nvidia--nvidia-nemotron-3.5-lightning-30b-a3b-bf16@1
- hosting
- foundry-managed-compute
- license
- other
- context length
- 262144
- source model id
- azure-huggingface/nvidia--nvidia-nemotron-3.5-lightning-30b-a3b-bf16@2
- hosting
- foundry-managed-compute
- license
- other
- context length
- 262144
- source model id
- azure-huggingface/nvidia--nvidia-nemotron-3.5-lightning-30b-a3b-bf16@3
- hosting
- foundry-managed-compute
- license
- other
- context length
- 262144
- source model id
- azure-huggingface/nvidia--nvidia-nemotron-3.5-lightning-30b-a3b-bf16@4
- hosting
- foundry-managed-compute
- license
- other
- context length
- 262144
- source model id
- azure-huggingface/nvidia--nvidia-nemotron-3.5-lightning-30b-a3b-bf16@5
- hosting
- foundry-managed-compute
- license
- other
- context length
- 262144
- source model id
- publishers/hf-nvidia/models/nvidia-nemotron-3.5-lightning-30b-a3b-bf16@001
- hosting
- vertex-self-deployed
Source specification observations
1–9 / 9
- field
- context_length
- value
- 262144
- normalized value
- 262144
- scope
- source-model
- source id
- azureFoundry
- confidence
- 1
- field
- context_length
- value
- 262144
- normalized value
- 262144
- scope
- source-model
- source id
- azureFoundry
- confidence
- 1
- field
- context_length
- value
- 262144
- normalized value
- 262144
- scope
- source-model
- source id
- azureFoundry
- confidence
- 1
- field
- context_length
- value
- 262144
- normalized value
- 262144
- scope
- source-model
- source id
- azureFoundry
- confidence
- 1
- field
- context_length
- value
- 262144
- normalized value
- 262144
- scope
- source-model
- source id
- azureFoundry
- confidence
- 1
- field
- context_length
- value
- 262144
- normalized value
- 262144
- scope
- source-model
- source id
- openRouter
- confidence
- 1
- field
- context_length
- value
- 1000000
- normalized value
- 1000000
- scope
- source-model
- source id
- openRouter
- confidence
- 1
- field
- max_output_tokens
- value
- 235929
- normalized value
- 235929
- scope
- source-model
- source id
- openRouter
- confidence
- 1
- field
- max_output_tokens
- value
- 65536
- normalized value
- 65536
- scope
- source-model
- source id
- openRouter
- confidence
- 1
Hugging Face model card
1–1 / 1
- downloads
- 1043242
- likes
- 272
- pipeline tag
- text-generation
- library name
- transformers
- License
- other
Explore more models