NVIDIA: Nemotron 3.5 Lightning

NVIDIA Nemotron 3.5 Lightning is an open mixture-of-experts model from NVIDIA, with 3B active parameters out of 30B total. It is suited for high-throughput agentic workloads and specialized tasks tha…

nvidia/nemotron-3.5-lightning-20260807

NVIDIA Nemotron 3.5 Lightning is an open mixture-of-experts model from NVIDIA, with 3B active parameters out of 30B total. It is suited for high-throughput agentic workloads and specialized tasks that benefit from domain-specific customization.

Specifications & capabilities

Input
text
Output
text
Context window · tokens
262144
Published model ID
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16
Capabilities
promptCaching, reasoning, structuredOutputs, toolCalling
Description source ↗

Evaluations

1–18 / 18

OpenRouter Benchmarks

67.083%

source id
openRouterBenchmarks
leaderboard name
tau2-bench Verified Airline · Accuracy · Darkbloom
dimension
agentic
metric
tau2-bench Verified Airline · Accuracy
rank
6
sample size
200
aggregation scope
status
published
harness
url
https://github.com/OpenRouterTeam/benchmark-harness
name
OpenRouter benchmark harness
direction
higherIsBetter
run_count
4
conditions
window
newest 90 days; successful visible nonzero runs
dataset
benchmark-leaderboard
benchmark
id
tau_bench_verified_airline
name
tau2-bench Verified Airline
record_kind
aggregate
provider_name
Darkbloom
attribution
Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
last_run_at
2026-11-14T19:27:00.426Z
license_url
https://creativecommons.org/licenses/by/4.0/
methodology
Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
metric_kind
tau2-bench Verified Airline · Accuracy
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
sample_size
200
source_file
benchmark-leaderboard.json
source_page
https://openrouter.ai/benchmarks/tau2-bench-airline
measurements
accuracy
0.6708329999999999
accuracy_std_dev
0.022776083947860723
cost_per_task_usd
0.00710901
output_tokens_per_task
20361.76
latency_seconds_per_task
454.081565
published_at
2026-11-16T01:46:02.653Z
snapshot_date
2026-11-16
source_revision
7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
transformations
TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
metric_population
8
benchmark_evaluation
true
openrouter_permaslug
nvidia/nemotron-3.5-lightning-20260807
source model id
nvidia/nemotron-3.5-lightning-20260807
Original result ↗

OpenRouter Benchmarks

65.124%

source id
openRouterBenchmarks
leaderboard name
GPQA Diamond · Accuracy · Darkbloom
dimension
knowledge
metric
GPQA Diamond · Accuracy
rank
8
sample size
2269
aggregation scope
status
published
harness
url
https://github.com/OpenRouterTeam/benchmark-harness
name
OpenRouter benchmark harness
direction
higherIsBetter
run_count
12
conditions
window
newest 90 days; successful visible nonzero runs
dataset
benchmark-leaderboard
benchmark
id
gpqa_diamond
name
GPQA Diamond
record_kind
aggregate
provider_name
Darkbloom
attribution
Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
last_run_at
2026-11-14T19:15:05.957Z
license_url
https://creativecommons.org/licenses/by/4.0/
methodology
Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
metric_kind
GPQA Diamond · Accuracy
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
sample_size
2269
source_file
benchmark-leaderboard.json
source_page
https://openrouter.ai/benchmarks/gpqa-diamond
measurements
accuracy
0.6512412503305419
accuracy_std_dev
0.038999469240957485
cost_per_task_usd
0.005451304539444689
output_tokens_per_task
37464.12913177611
latency_seconds_per_task
789.0263609519612
published_at
2026-11-16T01:46:02.653Z
snapshot_date
2026-11-16
source_revision
7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
transformations
TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
metric_population
10
benchmark_evaluation
true
openrouter_permaslug
nvidia/nemotron-3.5-lightning-20260807
source model id
nvidia/nemotron-3.5-lightning-20260807
Original result ↗

OpenRouter Benchmarks

71.528%

source id
openRouterBenchmarks
leaderboard name
tau2-bench Verified Airline · Accuracy · Wafer
dimension
agentic
metric
tau2-bench Verified Airline · Accuracy
rank
11
sample size
48
aggregation scope
status
published
harness
url
https://github.com/OpenRouterTeam/benchmark-harness
name
OpenRouter benchmark harness
direction
higherIsBetter
run_count
1
conditions
window
newest 90 days; successful visible nonzero runs
dataset
benchmark-leaderboard
benchmark
id
tau_bench_verified_airline
name
tau2-bench Verified Airline
record_kind
aggregate
provider_name
Wafer
attribution
Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
last_run_at
2026-10-10T11:44:02.317Z
license_url
https://creativecommons.org/licenses/by/4.0/
methodology
Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
metric_kind
tau2-bench Verified Airline · Accuracy
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
sample_size
48
source_file
benchmark-leaderboard.json
source_page
https://openrouter.ai/benchmarks/tau2-bench-airline
measurements
accuracy
0.715278
cost_per_task_usd
0.009582875
output_tokens_per_task
18445.333333333332
latency_seconds_per_task
164.1245
published_at
2026-11-16T01:46:02.653Z
snapshot_date
2026-11-16
source_revision
7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
transformations
TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
metric_population
13
benchmark_evaluation
true
openrouter_permaslug
nvidia/nemotron-3.5-lightning-20260807
source model id
nvidia/nemotron-3.5-lightning-20260807
Original result ↗

OpenRouter Benchmarks

74.81%

source id
openRouterBenchmarks
leaderboard name
GPQA Diamond · Accuracy · Wafer
dimension
knowledge
metric
GPQA Diamond · Accuracy
rank
13
sample size
395
aggregation scope
status
published
harness
url
https://github.com/OpenRouterTeam/benchmark-harness
name
OpenRouter benchmark harness
direction
higherIsBetter
run_count
2
conditions
window
newest 90 days; successful visible nonzero runs
dataset
benchmark-leaderboard
benchmark
id
gpqa_diamond
name
GPQA Diamond
record_kind
aggregate
provider_name
Wafer
attribution
Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
last_run_at
2026-10-10T08:39:12.933Z
license_url
https://creativecommons.org/licenses/by/4.0/
methodology
Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
metric_kind
GPQA Diamond · Accuracy
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
sample_size
395
source_file
benchmark-leaderboard.json
source_page
https://openrouter.ai/benchmarks/gpqa-diamond
measurements
accuracy
0.7481013645569621
accuracy_std_dev
0.004003499999999993
cost_per_task_usd
0.010130237974683544
output_tokens_per_task
60721.07341772152
latency_seconds_per_task
298.77967341772154
published_at
2026-11-16T01:46:02.653Z
snapshot_date
2026-11-16
source_revision
7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
transformations
TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
metric_population
13
benchmark_evaluation
true
openrouter_permaslug
nvidia/nemotron-3.5-lightning-20260807
source model id
nvidia/nemotron-3.5-lightning-20260807
Original result ↗

OpenRouter Benchmarks

74.018%

source id
openRouterBenchmarks
leaderboard name
GPQA Diamond · Accuracy · Io Net
dimension
knowledge
metric
GPQA Diamond · Accuracy
rank
13
sample size
2070
aggregation scope
status
published
harness
url
https://github.com/OpenRouterTeam/benchmark-harness
name
OpenRouter benchmark harness
direction
higherIsBetter
run_count
12
conditions
window
newest 90 days; successful visible nonzero runs
dataset
benchmark-leaderboard
benchmark
id
gpqa_diamond
name
GPQA Diamond
record_kind
aggregate
provider_name
Io Net
attribution
Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
last_run_at
2026-11-04T18:42:44.798Z
license_url
https://creativecommons.org/licenses/by/4.0/
methodology
Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
metric_kind
GPQA Diamond · Accuracy
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
sample_size
2070
source_file
benchmark-leaderboard.json
source_page
https://openrouter.ai/benchmarks/gpqa-diamond
measurements
accuracy
0.7401770526570047
accuracy_std_dev
0.0865423410071625
cost_per_task_usd
0.0039563893719806766
output_tokens_per_task
24944.29468599034
latency_seconds_per_task
552.5429028985508
published_at
2026-11-16T01:46:02.653Z
snapshot_date
2026-11-16
source_revision
7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
transformations
TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
metric_population
15
benchmark_evaluation
true
openrouter_permaslug
nvidia/nemotron-3.5-lightning-20260807
source model id
nvidia/nemotron-3.5-lightning-20260807
Original result ↗

OpenRouter Benchmarks

65.354%

source id
openRouterBenchmarks
leaderboard name
tau2-bench Verified Airline · Accuracy · Io Net
dimension
agentic
metric
tau2-bench Verified Airline · Accuracy
rank
14
sample size
546
aggregation scope
status
published
harness
url
https://github.com/OpenRouterTeam/benchmark-harness
name
OpenRouter benchmark harness
direction
higherIsBetter
run_count
11
conditions
window
newest 90 days; successful visible nonzero runs
dataset
benchmark-leaderboard
benchmark
id
tau_bench_verified_airline
name
tau2-bench Verified Airline
record_kind
aggregate
provider_name
Io Net
attribution
Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
last_run_at
2026-11-04T19:03:50.422Z
license_url
https://creativecommons.org/licenses/by/4.0/
methodology
Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
metric_kind
tau2-bench Verified Airline · Accuracy
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
sample_size
546
source_file
benchmark-leaderboard.json
source_page
https://openrouter.ai/benchmarks/tau2-bench-airline
measurements
accuracy
0.6535409120879121
accuracy_std_dev
0.03689102528724027
cost_per_task_usd
0.012232007326007327
output_tokens_per_task
22783.287545787545
latency_seconds_per_task
1066.692706959707
published_at
2026-11-16T01:46:02.653Z
snapshot_date
2026-11-16
source_revision
7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
transformations
TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
metric_population
15
benchmark_evaluation
true
openrouter_permaslug
nvidia/nemotron-3.5-lightning-20260807
source model id
nvidia/nemotron-3.5-lightning-20260807
Original result ↗

OpenRouter Benchmarks

68.125%

source id
openRouterBenchmarks
leaderboard name
tau2-bench Verified Airline · Accuracy · CoreWeave
dimension
agentic
metric
tau2-bench Verified Airline · Accuracy
rank
17
sample size
400
aggregation scope
status
published
harness
url
https://github.com/OpenRouterTeam/benchmark-harness
name
OpenRouter benchmark harness
direction
higherIsBetter
run_count
8
conditions
window
newest 90 days; successful visible nonzero runs
dataset
benchmark-leaderboard
benchmark
id
tau_bench_verified_airline
name
tau2-bench Verified Airline
record_kind
aggregate
provider_name
CoreWeave
attribution
Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
last_run_at
2026-10-31T11:46:38.147Z
license_url
https://creativecommons.org/licenses/by/4.0/
methodology
Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
metric_kind
tau2-bench Verified Airline · Accuracy
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
sample_size
400
source_file
benchmark-leaderboard.json
source_page
https://openrouter.ai/benchmarks/tau2-bench-airline
measurements
accuracy
0.68125
accuracy_std_dev
0.0202715728731147
cost_per_task_usd
0.0174918175
output_tokens_per_task
21340.095
latency_seconds_per_task
123.0915525
published_at
2026-11-16T01:46:02.653Z
snapshot_date
2026-11-16
source_revision
7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
transformations
TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
metric_population
22
benchmark_evaluation
true
openrouter_permaslug
nvidia/nemotron-3.5-lightning-20260807
source model id
nvidia/nemotron-3.5-lightning-20260807
Original result ↗

OpenRouter Benchmarks

58.101%

source id
openRouterBenchmarks
leaderboard name
GPQA Diamond · Accuracy · Fireworks
dimension
knowledge
metric
GPQA Diamond · Accuracy
rank
17
sample size
395
aggregation scope
status
published
harness
url
https://github.com/OpenRouterTeam/benchmark-harness
name
OpenRouter benchmark harness
direction
higherIsBetter
run_count
2
conditions
window
newest 90 days; successful visible nonzero runs
dataset
benchmark-leaderboard
benchmark
id
gpqa_diamond
name
GPQA Diamond
record_kind
aggregate
provider_name
Fireworks
attribution
Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
last_run_at
2026-10-31T08:09:25.783Z
license_url
https://creativecommons.org/licenses/by/4.0/
methodology
Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
metric_kind
GPQA Diamond · Accuracy
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
sample_size
395
source_file
benchmark-leaderboard.json
source_page
https://openrouter.ai/benchmarks/gpqa-diamond
measurements
accuracy
0.5810130708860759
accuracy_std_dev
0.012019500000000016
cost_per_task_usd
0.01962629113924051
output_tokens_per_task
97953.7417721519
latency_seconds_per_task
376.7089949367089
published_at
2026-11-16T01:46:02.653Z
snapshot_date
2026-11-16
source_revision
7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
transformations
TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
metric_population
17
benchmark_evaluation
true
openrouter_permaslug
nvidia/nemotron-3.5-lightning-20260807
source model id
nvidia/nemotron-3.5-lightning-20260807
Original result ↗

OpenRouter Benchmarks

73.336%

source id
openRouterBenchmarks
leaderboard name
GPQA Diamond · Accuracy · CoreWeave
dimension
knowledge
metric
GPQA Diamond · Accuracy
rank
22
sample size
2144
aggregation scope
status
published
harness
url
https://github.com/OpenRouterTeam/benchmark-harness
name
OpenRouter benchmark harness
direction
higherIsBetter
run_count
11
conditions
window
newest 90 days; successful visible nonzero runs
dataset
benchmark-leaderboard
benchmark
id
gpqa_diamond
name
GPQA Diamond
record_kind
aggregate
provider_name
CoreWeave
attribution
Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
last_run_at
2026-10-31T08:46:43.680Z
license_url
https://creativecommons.org/licenses/by/4.0/
methodology
Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
metric_kind
GPQA Diamond · Accuracy
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
sample_size
2144
source_file
benchmark-leaderboard.json
source_page
https://openrouter.ai/benchmarks/gpqa-diamond
measurements
accuracy
0.733364333488806
accuracy_std_dev
0.03217972751863435
cost_per_task_usd
0.010386112406716419
output_tokens_per_task
48631.178638059704
latency_seconds_per_task
251.82099860074626
published_at
2026-11-16T01:46:02.653Z
snapshot_date
2026-11-16
source_revision
7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
transformations
TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
metric_population
27
benchmark_evaluation
true
openrouter_permaslug
nvidia/nemotron-3.5-lightning-20260807
source model id
nvidia/nemotron-3.5-lightning-20260807
Original result ↗

OpenRouter Benchmarks

72.718%

source id
openRouterBenchmarks
leaderboard name
GPQA Diamond · Accuracy · Phala
dimension
knowledge
metric
GPQA Diamond · Accuracy
rank
22
sample size
1154
aggregation scope
status
published
harness
url
https://github.com/OpenRouterTeam/benchmark-harness
name
OpenRouter benchmark harness
direction
higherIsBetter
run_count
6
conditions
window
newest 90 days; successful visible nonzero runs
dataset
benchmark-leaderboard
benchmark
id
gpqa_diamond
name
GPQA Diamond
record_kind
aggregate
provider_name
Phala
attribution
Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
last_run_at
2026-10-10T08:24:27.616Z
license_url
https://creativecommons.org/licenses/by/4.0/
methodology
Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
metric_kind
GPQA Diamond · Accuracy
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
sample_size
1154
source_file
benchmark-leaderboard.json
source_page
https://openrouter.ai/benchmarks/gpqa-diamond
measurements
accuracy
0.7271808509532062
accuracy_std_dev
0.03540222199908109
cost_per_task_usd
0.011152869150779895
output_tokens_per_task
55506.62998266898
latency_seconds_per_task
504.50158318890817
published_at
2026-11-16T01:46:02.653Z
snapshot_date
2026-11-16
source_revision
7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
transformations
TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
metric_population
24
benchmark_evaluation
true
openrouter_permaslug
nvidia/nemotron-3.5-lightning-20260807
source model id
nvidia/nemotron-3.5-lightning-20260807
Original result ↗

OpenRouter Benchmarks

65.578%

source id
openRouterBenchmarks
leaderboard name
tau2-bench Verified Airline · Accuracy · Phala
dimension
agentic
metric
tau2-bench Verified Airline · Accuracy
rank
22
sample size
245
aggregation scope
status
published
harness
url
https://github.com/OpenRouterTeam/benchmark-harness
name
OpenRouter benchmark harness
direction
higherIsBetter
run_count
5
conditions
window
newest 90 days; successful visible nonzero runs
dataset
benchmark-leaderboard
benchmark
id
tau_bench_verified_airline
name
tau2-bench Verified Airline
record_kind
aggregate
provider_name
Phala
attribution
Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
last_run_at
2026-10-10T11:18:54.565Z
license_url
https://creativecommons.org/licenses/by/4.0/
methodology
Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
metric_kind
tau2-bench Verified Airline · Accuracy
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
sample_size
245
source_file
benchmark-leaderboard.json
source_page
https://openrouter.ai/benchmarks/tau2-bench-airline
measurements
accuracy
0.6557823102040816
accuracy_std_dev
0.009318408855593308
cost_per_task_usd
0.014797057142857142
output_tokens_per_task
20245.579591836733
latency_seconds_per_task
277.9072775510204
published_at
2026-11-16T01:46:02.653Z
snapshot_date
2026-11-16
source_revision
7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
transformations
TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
metric_population
23
benchmark_evaluation
true
openrouter_permaslug
nvidia/nemotron-3.5-lightning-20260807
source model id
nvidia/nemotron-3.5-lightning-20260807
Original result ↗

OpenRouter Benchmarks

65.556%

source id
openRouterBenchmarks
leaderboard name
tau2-bench Verified Airline · Accuracy · DeepInfra
dimension
agentic
metric
tau2-bench Verified Airline · Accuracy
rank
42
sample size
450
aggregation scope
status
published
harness
url
https://github.com/OpenRouterTeam/benchmark-harness
name
OpenRouter benchmark harness
direction
higherIsBetter
run_count
9
conditions
window
newest 90 days; successful visible nonzero runs
dataset
benchmark-leaderboard
benchmark
id
tau_bench_verified_airline
name
tau2-bench Verified Airline
record_kind
aggregate
provider_name
DeepInfra
attribution
Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
last_run_at
2026-10-31T12:25:21.635Z
license_url
https://creativecommons.org/licenses/by/4.0/
methodology
Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
metric_kind
tau2-bench Verified Airline · Accuracy
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
sample_size
450
source_file
benchmark-leaderboard.json
source_page
https://openrouter.ai/benchmarks/tau2-bench-airline
measurements
accuracy
0.6555556666666666
accuracy_std_dev
0.03807065880409451
cost_per_task_usd
0.021039175555555557
output_tokens_per_task
26269.908888888887
latency_seconds_per_task
248.79159777777778
published_at
2026-11-16T01:46:02.653Z
snapshot_date
2026-11-16
source_revision
7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
transformations
TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
metric_population
63
benchmark_evaluation
true
openrouter_permaslug
nvidia/nemotron-3.5-lightning-20260807
source model id
nvidia/nemotron-3.5-lightning-20260807
Original result ↗

OpenRouter Benchmarks

71.1%

source id
openRouterBenchmarks
leaderboard name
GPQA Diamond · Accuracy · DeepInfra
dimension
knowledge
metric
GPQA Diamond · Accuracy
rank
48
sample size
1966
aggregation scope
status
published
harness
url
https://github.com/OpenRouterTeam/benchmark-harness
name
OpenRouter benchmark harness
direction
higherIsBetter
run_count
10
conditions
window
newest 90 days; successful visible nonzero runs
dataset
benchmark-leaderboard
benchmark
id
gpqa_diamond
name
GPQA Diamond
record_kind
aggregate
provider_name
DeepInfra
attribution
Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
last_run_at
2026-10-31T08:15:24.314Z
license_url
https://creativecommons.org/licenses/by/4.0/
methodology
Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
metric_kind
GPQA Diamond · Accuracy
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
sample_size
1966
source_file
benchmark-leaderboard.json
source_page
https://openrouter.ai/benchmarks/gpqa-diamond
measurements
accuracy
0.7110037019328587
accuracy_std_dev
0.031890632320636075
cost_per_task_usd
0.0072013123092573746
output_tokens_per_task
40164.048830111904
latency_seconds_per_task
309.64626246185145
published_at
2026-11-16T01:46:02.653Z
snapshot_date
2026-11-16
source_revision
7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
transformations
TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
metric_population
71
benchmark_evaluation
true
openrouter_permaslug
nvidia/nemotron-3.5-lightning-20260807
source model id
nvidia/nemotron-3.5-lightning-20260807
Original result ↗

OpenRouter Benchmarks

65.622%

source id
openRouterBenchmarks
leaderboard name
tau2-bench Verified Airline · Accuracy · auto-routing
dimension
agentic
metric
tau2-bench Verified Airline · Accuracy
rank
92
sample size
399
aggregation scope
status
published
harness
url
https://github.com/OpenRouterTeam/benchmark-harness
name
OpenRouter benchmark harness
direction
higherIsBetter
run_count
8
conditions
window
newest 90 days; successful visible nonzero runs
dataset
benchmark-leaderboard
benchmark
id
tau_bench_verified_airline
name
tau2-bench Verified Airline
record_kind
aggregate
provider_name
auto-routing
attribution
Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
last_run_at
2026-10-31T12:28:49.050Z
license_url
https://creativecommons.org/licenses/by/4.0/
methodology
Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
metric_kind
tau2-bench Verified Airline · Accuracy
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
sample_size
399
source_file
benchmark-leaderboard.json
source_page
https://openrouter.ai/benchmarks/tau2-bench-airline
measurements
accuracy
0.6562239298245615
accuracy_std_dev
0.018946289616967236
cost_per_task_usd
0.01692722807017544
output_tokens_per_task
22811.754385964912
latency_seconds_per_task
312.8047944862155
published_at
2026-11-16T01:46:02.653Z
snapshot_date
2026-11-16
source_revision
7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
transformations
TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
metric_population
128
benchmark_evaluation
true
openrouter_permaslug
nvidia/nemotron-3.5-lightning-20260807
source model id
nvidia/nemotron-3.5-lightning-20260807
Original result ↗

OpenRouter Benchmarks

70.613%

source id
openRouterBenchmarks
leaderboard name
GPQA Diamond · Accuracy · auto-routing
dimension
knowledge
metric
GPQA Diamond · Accuracy
rank
101
sample size
1926
aggregation scope
status
published
harness
url
https://github.com/OpenRouterTeam/benchmark-harness
name
OpenRouter benchmark harness
direction
higherIsBetter
run_count
10
conditions
window
newest 90 days; successful visible nonzero runs
dataset
benchmark-leaderboard
benchmark
id
gpqa_diamond
name
GPQA Diamond
record_kind
aggregate
provider_name
auto-routing
attribution
Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
last_run_at
2026-10-31T08:03:09.796Z
license_url
https://creativecommons.org/licenses/by/4.0/
methodology
Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
metric_kind
GPQA Diamond · Accuracy
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
sample_size
1926
source_file
benchmark-leaderboard.json
source_page
https://openrouter.ai/benchmarks/gpqa-diamond
measurements
accuracy
0.7061267289719626
accuracy_std_dev
0.026110787988875415
cost_per_task_usd
0.00886934890965732
output_tokens_per_task
45095.30893042575
latency_seconds_per_task
374.4079574247145
published_at
2026-11-16T01:46:02.653Z
snapshot_date
2026-11-16
source_revision
7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
transformations
TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
metric_population
132
benchmark_evaluation
true
openrouter_permaslug
nvidia/nemotron-3.5-lightning-20260807
source model id
nvidia/nemotron-3.5-lightning-20260807
Original result ↗

Published prices

1–5 / 5

Provider specifications & access

1–6 / 6

nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 ↗

source model id
azure-huggingface/nvidia--nvidia-nemotron-3.5-lightning-30b-a3b-bf16@1
hosting
foundry-managed-compute
platforms
  • Microsoft Foundry
license
other
context length
262144
architecture
input_modalities
  • text
output_modalities
  • text
capabilities

    nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 ↗

    source model id
    azure-huggingface/nvidia--nvidia-nemotron-3.5-lightning-30b-a3b-bf16@2
    hosting
    foundry-managed-compute
    platforms
    • Microsoft Foundry
    license
    other
    context length
    262144
    architecture
    input_modalities
    • text
    output_modalities
    • text
    capabilities

      nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 ↗

      source model id
      azure-huggingface/nvidia--nvidia-nemotron-3.5-lightning-30b-a3b-bf16@3
      hosting
      foundry-managed-compute
      platforms
      • Microsoft Foundry
      license
      other
      context length
      262144
      architecture
      input_modalities
      • text
      output_modalities
      • text
      capabilities

        nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 ↗

        source model id
        azure-huggingface/nvidia--nvidia-nemotron-3.5-lightning-30b-a3b-bf16@4
        hosting
        foundry-managed-compute
        platforms
        • Microsoft Foundry
        license
        other
        context length
        262144
        architecture
        input_modalities
        • text
        output_modalities
        • text
        capabilities

          nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 ↗

          source model id
          azure-huggingface/nvidia--nvidia-nemotron-3.5-lightning-30b-a3b-bf16@5
          hosting
          foundry-managed-compute
          platforms
          • Microsoft Foundry
          license
          other
          context length
          262144
          architecture
          input_modalities
          • text
          output_modalities
          • text
          capabilities

            nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 ↗

            source model id
            publishers/hf-nvidia/models/nvidia-nemotron-3.5-lightning-30b-a3b-bf16@001
            hosting
            vertex-self-deployed
            platforms
            • Google Cloud Vertex AI
            architecture
            input_modalities
            • text
            output_modalities
            • text
            capabilities

              Source specification observations

              1–9 / 9

              azureFoundry ↗

              field
              context_length
              value
              262144
              normalized value
              262144
              scope
              source-model
              source id
              azureFoundry
              confidence
              1

              azureFoundry ↗

              field
              context_length
              value
              262144
              normalized value
              262144
              scope
              source-model
              source id
              azureFoundry
              confidence
              1

              azureFoundry ↗

              field
              context_length
              value
              262144
              normalized value
              262144
              scope
              source-model
              source id
              azureFoundry
              confidence
              1

              azureFoundry ↗

              field
              context_length
              value
              262144
              normalized value
              262144
              scope
              source-model
              source id
              azureFoundry
              confidence
              1

              azureFoundry ↗

              field
              context_length
              value
              262144
              normalized value
              262144
              scope
              source-model
              source id
              azureFoundry
              confidence
              1

              openRouter ↗

              field
              context_length
              value
              262144
              normalized value
              262144
              scope
              source-model
              source id
              openRouter
              confidence
              1

              openRouter ↗

              field
              context_length
              value
              1000000
              normalized value
              1000000
              scope
              source-model
              source id
              openRouter
              confidence
              1

              openRouter ↗

              field
              max_output_tokens
              value
              235929
              normalized value
              235929
              scope
              source-model
              source id
              openRouter
              confidence
              1

              openRouter ↗

              field
              max_output_tokens
              value
              65536
              normalized value
              65536
              scope
              source-model
              source id
              openRouter
              confidence
              1

              Original sources

              1–18 / 18

              NVIDIA · Model reference ↗

              Hugging Face · nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 ↗

              Google Cloud · Model page ↗

              Microsoft Foundry · Model page ↗

              Microsoft Foundry · Model page ↗

              Microsoft Foundry · Model page ↗

              Microsoft Foundry · Model page ↗

              Microsoft Foundry · Model page ↗

              OpenRouter · Model page ↗

              OpenRouter · Model page ↗

              Microsoft Foundry · Model documentation ↗

              Epoch AI · Catalog ↗

              Google Cloud · Catalog ↗

              Hugging Face · Catalog ↗

              Hugging Face · Catalog ↗

              Microsoft Foundry · Catalog ↗

              OpenRouter · Catalog ↗

              OpenRouter · Catalog ↗

              Source links

              1–8 / 8

              Hugging Face model card

              1–1 / 1

              Explore more models

              Models

              Loading
              EVIDENCE / COMPARISON

              Benchmarks

              Explore every benchmark

              Loading

              These charts contain model-linked results. The charts above also include unlinked source profiles. Compare scores within each evaluation. Model quality, usage, transparency and system performance measure different things.

              Install TensorTrace