Meta: Llama 3.3 70B Instruct

The Meta Llama 3.3 multilingual large language model (LLM) is a pretrained and instruction tuned generative model in 70B (text in/text out). The Llama 3.3 instruction tuned text only model is optimiz…

meta-llama/llama-3.3-70b-instruct

The Meta Llama 3.3 multilingual large language model (LLM) is a pretrained and instruction tuned generative model in 70B (text in/text out). The Llama 3.3 instruction tuned text only model is optimized for multilingual dialogue use cases and outperforms many of the available open source and closed chat models on common industry benchmarks.

Supported languages: English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai.

[Model Card](https://github.com/meta-llama/llama-models/blob/main/models/llama3_3/MODEL_CARD.md)

Specifications & capabilities

Input
text
Output
text
Context window · tokens
131072
Knowledge cutoff
2023-12-31
Published model ID
Llama-3.3-70B-Instruct
Capabilities
promptCaching, structuredOutputs, toolCalling
Description source ↗

Evaluations

1–20 / 41

Open LLM Leaderboard v2

89.976%

source id
openLLMLeaderboardV2
leaderboard name
Open LLM v2 IFEval · Raw
dimension
instructionFollowing
metric
Raw accuracy
rank
1
aggregation scope
suite
Open LLM v2
co2_kg
76.55907410745293
status
archived
precision
bfloat16
task_count
6
metric_kind
Raw accuracy
rank_method
Derived competition rank over published evaluations with reported scores, including variants and source flags
weight_type
Original
published_at
2025-03-20T12:17:27Z
chat_template
true
suite_version
2
source_flagged
false
missing_metrics
    dataset_revision
    9c09a7cae43334062a82cb164f2ef255013dafa2
    metric_population
    4576
    ranking_population
    4576
    source_flag_anomaly
    false
    source model id
    meta-llama/Llama-3.3-70B-Instruct
    Original result ↗

    Open LLM Leaderboard v2

    89.976%

    source id
    openLLMLeaderboardV2
    leaderboard name
    Open LLM v2 IFEval · Normalized
    dimension
    instructionFollowing
    metric
    Normalized score
    rank
    1
    aggregation scope
    suite
    Open LLM v2
    co2_kg
    76.55907410745293
    status
    archived
    precision
    bfloat16
    task_count
    6
    metric_kind
    Normalized score
    rank_method
    Derived competition rank over published evaluations with reported scores, including variants and source flags
    weight_type
    Original
    published_at
    2025-03-20T12:17:27Z
    chat_template
    true
    suite_version
    2
    source_flagged
    false
    missing_metrics
      dataset_revision
      9c09a7cae43334062a82cb164f2ef255013dafa2
      metric_population
      4576
      ranking_population
      4576
      source_flag_anomaly
      false
      source model id
      meta-llama/Llama-3.3-70B-Instruct
      Original result ↗

      Open Persian LLM Leaderboard

      71.4%

      source id
      openPersianLLM
      leaderboard name
      AUT Multiple Choice Persian
      dimension
      general
      metric
      AUT Multiple Choice Persian
      rank
      1
      aggregation scope
      status
      published
      direction
      higherIsBetter
      conditions
      suite
      Open Persian LLM
      language
      fa
      precision
      bfloat16
      metric_kind
      AUT Multiple Choice Persian
      rank_method
      competition rank among reported evaluations in identical conditions, including unlinked observations
      source_file
      PartAI/open-persian-llm-leaderboard/leaderboard_data.jsonl
      published_at
      2025-12-06T10:13:22Z
      source_revision
      92771e7359b0679b2f851b2a34e98c6ce2ccbf29bb5ae60b12bee0bd0319286e
      metric_population
      27
      benchmark_evaluation
      true
      source model id
      meta-llama/Llama-3.3-70B-Instruct
      Original result ↗

      Open Persian LLM Leaderboard

      51.42%

      source id
      openPersianLLM
      leaderboard name
      Part Multiple Choice
      dimension
      general
      metric
      Part Multiple Choice
      rank
      1
      aggregation scope
      status
      published
      direction
      higherIsBetter
      conditions
      suite
      Open Persian LLM
      language
      fa
      precision
      bfloat16
      metric_kind
      Part Multiple Choice
      rank_method
      competition rank among reported evaluations in identical conditions, including unlinked observations
      source_file
      PartAI/open-persian-llm-leaderboard/leaderboard_data.jsonl
      published_at
      2025-12-06T10:13:22Z
      source_revision
      92771e7359b0679b2f851b2a34e98c6ce2ccbf29bb5ae60b12bee0bd0319286e
      metric_population
      27
      benchmark_evaluation
      true
      source model id
      meta-llama/Llama-3.3-70B-Instruct
      Original result ↗

      Open Persian LLM Leaderboard

      43.67%

      source id
      openPersianLLM
      leaderboard name
      MMLU Pro
      dimension
      general
      metric
      MMLU Pro
      rank
      1
      aggregation scope
      status
      published
      direction
      higherIsBetter
      conditions
      suite
      Open Persian LLM
      language
      fa
      precision
      bfloat16
      metric_kind
      MMLU Pro
      rank_method
      competition rank among reported evaluations in identical conditions, including unlinked observations
      source_file
      PartAI/open-persian-llm-leaderboard/leaderboard_data.jsonl
      published_at
      2025-12-06T10:13:22Z
      source_revision
      92771e7359b0679b2f851b2a34e98c6ce2ccbf29bb5ae60b12bee0bd0319286e
      metric_population
      27
      benchmark_evaluation
      true
      source model id
      meta-llama/Llama-3.3-70B-Instruct
      Original result ↗

      Open Persian LLM Leaderboard

      89.93%

      source id
      openPersianLLM
      leaderboard name
      ARC Challenge
      dimension
      general
      metric
      ARC Challenge
      rank
      2
      aggregation scope
      status
      published
      direction
      higherIsBetter
      conditions
      suite
      Open Persian LLM
      language
      fa
      precision
      bfloat16
      metric_kind
      ARC Challenge
      rank_method
      competition rank among reported evaluations in identical conditions, including unlinked observations
      source_file
      PartAI/open-persian-llm-leaderboard/leaderboard_data.jsonl
      published_at
      2025-12-06T10:13:22Z
      source_revision
      92771e7359b0679b2f851b2a34e98c6ce2ccbf29bb5ae60b12bee0bd0319286e
      metric_population
      27
      benchmark_evaluation
      true
      source model id
      meta-llama/Llama-3.3-70B-Instruct
      Original result ↗

      Open Persian LLM Leaderboard

      95.4%

      source id
      openPersianLLM
      leaderboard name
      ARC Easy
      dimension
      general
      metric
      ARC Easy
      rank
      3
      aggregation scope
      status
      published
      direction
      higherIsBetter
      conditions
      suite
      Open Persian LLM
      language
      fa
      precision
      bfloat16
      metric_kind
      ARC Easy
      rank_method
      competition rank among reported evaluations in identical conditions, including unlinked observations
      source_file
      PartAI/open-persian-llm-leaderboard/leaderboard_data.jsonl
      published_at
      2025-12-06T10:13:22Z
      source_revision
      92771e7359b0679b2f851b2a34e98c6ce2ccbf29bb5ae60b12bee0bd0319286e
      metric_population
      27
      benchmark_evaluation
      true
      source model id
      meta-llama/Llama-3.3-70B-Instruct
      Original result ↗

      OpenRouter Benchmarks

      49.952%

      source id
      openRouterBenchmarks
      leaderboard name
      GPQA Diamond · Accuracy · Groq
      dimension
      knowledge
      metric
      GPQA Diamond · Accuracy
      rank
      5
      sample size
      1386
      aggregation scope
      status
      published
      harness
      url
      https://github.com/OpenRouterTeam/benchmark-harness
      name
      OpenRouter benchmark harness
      direction
      higherIsBetter
      run_count
      7
      conditions
      window
      newest 90 days; successful visible nonzero runs
      dataset
      benchmark-leaderboard
      benchmark
      id
      gpqa_diamond
      name
      GPQA Diamond
      record_kind
      aggregate
      provider_name
      Groq
      attribution
      Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
      last_run_at
      2026-10-31T16:04:29.124Z
      license_url
      https://creativecommons.org/licenses/by/4.0/
      methodology
      Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
      metric_kind
      GPQA Diamond · Accuracy
      rank_method
      competition rank among reported evaluations in identical conditions, including unlinked observations
      sample_size
      1386
      source_file
      benchmark-leaderboard.json
      source_page
      https://openrouter.ai/benchmarks/gpqa-diamond
      measurements
      accuracy
      0.49951899999999994
      accuracy_std_dev
      0.012685149528934545
      cost_per_task_usd
      0.0023819927849927846
      output_tokens_per_task
      2368.319624819625
      latency_seconds_per_task
      9.173090187590187
      published_at
      2026-11-16T01:46:02.653Z
      snapshot_date
      2026-11-16
      source_revision
      7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
      transformations
      TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
      metric_population
      6
      benchmark_evaluation
      true
      openrouter_permaslug
      meta-llama/llama-3.3-70b-instruct
      source model id
      meta-llama/llama-3.3-70b-instruct
      Original result ↗

      OpenRouter Benchmarks

      48.172%

      source id
      openRouterBenchmarks
      leaderboard name
      GPQA Diamond · Accuracy · SambaNova
      dimension
      knowledge
      metric
      GPQA Diamond · Accuracy
      rank
      8
      sample size
      1386
      aggregation scope
      status
      published
      harness
      url
      https://github.com/OpenRouterTeam/benchmark-harness
      name
      OpenRouter benchmark harness
      direction
      higherIsBetter
      run_count
      7
      conditions
      window
      newest 90 days; successful visible nonzero runs
      dataset
      benchmark-leaderboard
      benchmark
      id
      gpqa_diamond
      name
      GPQA Diamond
      record_kind
      aggregate
      provider_name
      SambaNova
      attribution
      Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
      last_run_at
      2026-10-31T16:05:14.701Z
      license_url
      https://creativecommons.org/licenses/by/4.0/
      methodology
      Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
      metric_kind
      GPQA Diamond · Accuracy
      rank_method
      competition rank among reported evaluations in identical conditions, including unlinked observations
      sample_size
      1386
      source_file
      benchmark-leaderboard.json
      source_page
      https://openrouter.ai/benchmarks/gpqa-diamond
      measurements
      accuracy
      0.48172171428571425
      accuracy_std_dev
      0.01107352294844762
      cost_per_task_usd
      0.0023519292929292932
      output_tokens_per_task
      2210.2922077922076
      latency_seconds_per_task
      16.333764790764793
      published_at
      2026-11-16T01:46:02.653Z
      snapshot_date
      2026-11-16
      source_revision
      7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
      transformations
      TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
      metric_population
      8
      benchmark_evaluation
      true
      openrouter_permaslug
      meta-llama/llama-3.3-70b-instruct
      source model id
      meta-llama/llama-3.3-70b-instruct
      Original result ↗

      OpenRouter Benchmarks

      48.363%

      source id
      openRouterBenchmarks
      leaderboard name
      GPQA Diamond · Accuracy · AkashML
      dimension
      knowledge
      metric
      GPQA Diamond · Accuracy
      rank
      10
      sample size
      2912
      aggregation scope
      status
      published
      harness
      url
      https://github.com/OpenRouterTeam/benchmark-harness
      name
      OpenRouter benchmark harness
      direction
      higherIsBetter
      run_count
      15
      conditions
      window
      newest 90 days; successful visible nonzero runs
      dataset
      benchmark-leaderboard
      benchmark
      id
      gpqa_diamond
      name
      GPQA Diamond
      record_kind
      aggregate
      provider_name
      AkashML
      attribution
      Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
      last_run_at
      2026-10-31T16:01:53.654Z
      license_url
      https://creativecommons.org/licenses/by/4.0/
      methodology
      Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
      metric_kind
      GPQA Diamond · Accuracy
      rank_method
      competition rank among reported evaluations in identical conditions, including unlinked observations
      sample_size
      2912
      source_file
      benchmark-leaderboard.json
      source_page
      https://openrouter.ai/benchmarks/gpqa-diamond
      measurements
      accuracy
      0.4836306971153846
      accuracy_std_dev
      0.010925716722180446
      cost_per_task_usd
      0.0013556023351648351
      output_tokens_per_task
      2363.5096153846152
      latency_seconds_per_task
      59.72106524725275
      published_at
      2026-11-16T01:46:02.653Z
      snapshot_date
      2026-11-16
      source_revision
      7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
      transformations
      TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
      metric_population
      10
      benchmark_evaluation
      true
      openrouter_permaslug
      meta-llama/llama-3.3-70b-instruct
      source model id
      meta-llama/llama-3.3-70b-instruct
      Original result ↗

      OpenRouter Benchmarks

      37.428%

      source id
      openRouterBenchmarks
      leaderboard name
      tau2-bench Verified Airline · Accuracy · AkashML
      dimension
      agentic
      metric
      tau2-bench Verified Airline · Accuracy
      rank
      10
      sample size
      749
      aggregation scope
      status
      published
      harness
      url
      https://github.com/OpenRouterTeam/benchmark-harness
      name
      OpenRouter benchmark harness
      direction
      higherIsBetter
      run_count
      15
      conditions
      window
      newest 90 days; successful visible nonzero runs
      dataset
      benchmark-leaderboard
      benchmark
      id
      tau_bench_verified_airline
      name
      tau2-bench Verified Airline
      record_kind
      aggregate
      provider_name
      AkashML
      attribution
      Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
      last_run_at
      2026-10-31T19:10:53.562Z
      license_url
      https://creativecommons.org/licenses/by/4.0/
      methodology
      Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
      metric_kind
      tau2-bench Verified Airline · Accuracy
      rank_method
      competition rank among reported evaluations in identical conditions, including unlinked observations
      sample_size
      749
      source_file
      benchmark-leaderboard.json
      source_page
      https://openrouter.ai/benchmarks/tau2-bench-airline
      measurements
      accuracy
      0.3742767863818424
      accuracy_std_dev
      0.02350411518067989
      cost_per_task_usd
      0.013144802403204274
      output_tokens_per_task
      781.1562082777036
      latency_seconds_per_task
      41.061738317757005
      published_at
      2026-11-16T01:46:02.653Z
      snapshot_date
      2026-11-16
      source_revision
      7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
      transformations
      TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
      metric_population
      10
      benchmark_evaluation
      true
      openrouter_permaslug
      meta-llama/llama-3.3-70b-instruct
      source model id
      meta-llama/llama-3.3-70b-instruct
      Original result ↗

      OpenRouter Benchmarks

      46.886%

      source id
      openRouterBenchmarks
      leaderboard name
      GPQA Diamond · Accuracy · Crusoe
      dimension
      knowledge
      metric
      GPQA Diamond · Accuracy
      rank
      15
      sample size
      396
      aggregation scope
      status
      published
      harness
      url
      https://github.com/OpenRouterTeam/benchmark-harness
      name
      OpenRouter benchmark harness
      direction
      higherIsBetter
      run_count
      2
      conditions
      window
      newest 90 days; successful visible nonzero runs
      dataset
      benchmark-leaderboard
      benchmark
      id
      gpqa_diamond
      name
      GPQA Diamond
      record_kind
      aggregate
      provider_name
      Crusoe
      attribution
      Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
      last_run_at
      2026-08-29T16:31:42.504Z
      license_url
      https://creativecommons.org/licenses/by/4.0/
      methodology
      Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
      metric_kind
      GPQA Diamond · Accuracy
      rank_method
      competition rank among reported evaluations in identical conditions, including unlinked observations
      sample_size
      396
      source_file
      benchmark-leaderboard.json
      source_page
      https://openrouter.ai/benchmarks/gpqa-diamond
      measurements
      accuracy
      0.468855
      accuracy_std_dev
      0
      cost_per_task_usd
      0.002208848484848485
      output_tokens_per_task
      2683.0429292929293
      latency_seconds_per_task
      36.376982323232326
      published_at
      2026-11-16T01:46:02.653Z
      snapshot_date
      2026-11-16
      source_revision
      7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
      transformations
      TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
      metric_population
      15
      benchmark_evaluation
      true
      openrouter_permaslug
      meta-llama/llama-3.3-70b-instruct
      source model id
      meta-llama/llama-3.3-70b-instruct
      Original result ↗

      OpenRouter Benchmarks

      46.955%

      source id
      openRouterBenchmarks
      leaderboard name
      GPQA Diamond · Accuracy · Cloudflare
      dimension
      knowledge
      metric
      GPQA Diamond · Accuracy
      rank
      16
      sample size
      1560
      aggregation scope
      status
      published
      harness
      url
      https://github.com/OpenRouterTeam/benchmark-harness
      name
      OpenRouter benchmark harness
      direction
      higherIsBetter
      run_count
      8
      conditions
      window
      newest 90 days; successful visible nonzero runs
      dataset
      benchmark-leaderboard
      benchmark
      id
      gpqa_diamond
      name
      GPQA Diamond
      record_kind
      aggregate
      provider_name
      Cloudflare
      attribution
      Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
      last_run_at
      2026-10-31T16:05:36.136Z
      license_url
      https://creativecommons.org/licenses/by/4.0/
      methodology
      Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
      metric_kind
      GPQA Diamond · Accuracy
      rank_method
      competition rank among reported evaluations in identical conditions, including unlinked observations
      sample_size
      1560
      source_file
      benchmark-leaderboard.json
      source_page
      https://openrouter.ai/benchmarks/gpqa-diamond
      measurements
      accuracy
      0.46955114615384613
      accuracy_std_dev
      0.06298705219874558
      cost_per_task_usd
      0.0048902935897435895
      output_tokens_per_task
      2063.5615384615385
      latency_seconds_per_task
      46.68221346153847
      published_at
      2026-11-16T01:46:02.653Z
      snapshot_date
      2026-11-16
      source_revision
      7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
      transformations
      TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
      metric_population
      18
      benchmark_evaluation
      true
      openrouter_permaslug
      meta-llama/llama-3.3-70b-instruct
      source model id
      meta-llama/llama-3.3-70b-instruct
      Original result ↗

      OpenRouter Benchmarks

      35.048%

      source id
      openRouterBenchmarks
      leaderboard name
      tau2-bench Verified Airline · Accuracy · CoreWeave
      dimension
      agentic
      metric
      tau2-bench Verified Airline · Accuracy
      rank
      19
      sample size
      350
      aggregation scope
      status
      published
      harness
      url
      https://github.com/OpenRouterTeam/benchmark-harness
      name
      OpenRouter benchmark harness
      direction
      higherIsBetter
      run_count
      7
      conditions
      window
      newest 90 days; successful visible nonzero runs
      dataset
      benchmark-leaderboard
      benchmark
      id
      tau_bench_verified_airline
      name
      tau2-bench Verified Airline
      record_kind
      aggregate
      provider_name
      CoreWeave
      attribution
      Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
      last_run_at
      2026-10-31T18:48:48.352Z
      license_url
      https://creativecommons.org/licenses/by/4.0/
      methodology
      Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
      metric_kind
      tau2-bench Verified Airline · Accuracy
      rank_method
      competition rank among reported evaluations in identical conditions, including unlinked observations
      sample_size
      350
      source_file
      benchmark-leaderboard.json
      source_page
      https://openrouter.ai/benchmarks/tau2-bench-airline
      measurements
      accuracy
      0.35047614285714285
      accuracy_std_dev
      0.03596402460566301
      cost_per_task_usd
      0.09883995142857144
      output_tokens_per_task
      1132.6742857142858
      latency_seconds_per_task
      29.505717142857144
      published_at
      2026-11-16T01:46:02.653Z
      snapshot_date
      2026-11-16
      source_revision
      7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
      transformations
      TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
      metric_population
      22
      benchmark_evaluation
      true
      openrouter_permaslug
      meta-llama/llama-3.3-70b-instruct
      source model id
      meta-llama/llama-3.3-70b-instruct
      Original result ↗

      OpenRouter Benchmarks

      31.333%

      source id
      openRouterBenchmarks
      leaderboard name
      tau2-bench Verified Airline · Accuracy · PrimeIntellect
      dimension
      agentic
      metric
      tau2-bench Verified Airline · Accuracy
      rank
      20
      sample size
      50
      aggregation scope
      status
      published
      harness
      url
      https://github.com/OpenRouterTeam/benchmark-harness
      name
      OpenRouter benchmark harness
      direction
      higherIsBetter
      run_count
      1
      conditions
      window
      newest 90 days; successful visible nonzero runs
      dataset
      benchmark-leaderboard
      benchmark
      id
      tau_bench_verified_airline
      name
      tau2-bench Verified Airline
      record_kind
      aggregate
      provider_name
      PrimeIntellect
      attribution
      Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
      last_run_at
      2026-10-04T03:00:16.406Z
      license_url
      https://creativecommons.org/licenses/by/4.0/
      methodology
      Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
      metric_kind
      tau2-bench Verified Airline · Accuracy
      rank_method
      competition rank among reported evaluations in identical conditions, including unlinked observations
      sample_size
      50
      source_file
      benchmark-leaderboard.json
      source_page
      https://openrouter.ai/benchmarks/tau2-bench-airline
      measurements
      accuracy
      0.313333
      cost_per_task_usd
      0.15186704
      output_tokens_per_task
      1294.54
      latency_seconds_per_task
      32.63552
      published_at
      2026-11-16T01:46:02.653Z
      snapshot_date
      2026-11-16
      source_revision
      7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
      transformations
      TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
      metric_population
      21
      benchmark_evaluation
      true
      openrouter_permaslug
      meta-llama/llama-3.3-70b-instruct
      source model id
      meta-llama/llama-3.3-70b-instruct
      Original result ↗

      OpenRouter Benchmarks

      40%

      source id
      openRouterBenchmarks
      leaderboard name
      tau2-bench Verified Airline · Accuracy · Nebius
      dimension
      agentic
      metric
      tau2-bench Verified Airline · Accuracy
      rank
      21
      sample size
      100
      aggregation scope
      status
      published
      harness
      url
      https://github.com/OpenRouterTeam/benchmark-harness
      name
      OpenRouter benchmark harness
      direction
      higherIsBetter
      run_count
      2
      conditions
      window
      newest 90 days; successful visible nonzero runs
      dataset
      benchmark-leaderboard
      benchmark
      id
      tau_bench_verified_airline
      name
      tau2-bench Verified Airline
      record_kind
      aggregate
      provider_name
      Nebius
      attribution
      Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
      last_run_at
      2026-08-29T17:06:47.617Z
      license_url
      https://creativecommons.org/licenses/by/4.0/
      methodology
      Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
      metric_kind
      tau2-bench Verified Airline · Accuracy
      rank_method
      competition rank among reported evaluations in identical conditions, including unlinked observations
      sample_size
      100
      source_file
      benchmark-leaderboard.json
      source_page
      https://openrouter.ai/benchmarks/tau2-bench-airline
      measurements
      accuracy
      0.4
      accuracy_std_dev
      0
      cost_per_task_usd
      0.016500189999999998
      output_tokens_per_task
      649.31
      latency_seconds_per_task
      114.96983999999999
      published_at
      2026-11-16T01:46:02.653Z
      snapshot_date
      2026-11-16
      source_revision
      7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
      transformations
      TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
      metric_population
      23
      benchmark_evaluation
      true
      openrouter_permaslug
      meta-llama/llama-3.3-70b-instruct
      source model id
      meta-llama/llama-3.3-70b-instruct
      Original result ↗

      OpenRouter Benchmarks

      46.97%

      source id
      openRouterBenchmarks
      leaderboard name
      GPQA Diamond · Accuracy · PrimeIntellect
      dimension
      knowledge
      metric
      GPQA Diamond · Accuracy
      rank
      22
      sample size
      198
      aggregation scope
      status
      published
      harness
      url
      https://github.com/OpenRouterTeam/benchmark-harness
      name
      OpenRouter benchmark harness
      direction
      higherIsBetter
      run_count
      1
      conditions
      window
      newest 90 days; successful visible nonzero runs
      dataset
      benchmark-leaderboard
      benchmark
      id
      gpqa_diamond
      name
      GPQA Diamond
      record_kind
      aggregate
      provider_name
      PrimeIntellect
      attribution
      Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
      last_run_at
      2026-10-04T03:02:23.034Z
      license_url
      https://creativecommons.org/licenses/by/4.0/
      methodology
      Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
      metric_kind
      GPQA Diamond · Accuracy
      rank_method
      competition rank among reported evaluations in identical conditions, including unlinked observations
      sample_size
      198
      source_file
      benchmark-leaderboard.json
      source_page
      https://openrouter.ai/benchmarks/gpqa-diamond
      measurements
      accuracy
      0.469697
      cost_per_task_usd
      0.006110727272727273
      output_tokens_per_task
      2318.919191919192
      latency_seconds_per_task
      42.87936868686869
      published_at
      2026-11-16T01:46:02.653Z
      snapshot_date
      2026-11-16
      source_revision
      7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
      transformations
      TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
      metric_population
      22
      benchmark_evaluation
      true
      openrouter_permaslug
      meta-llama/llama-3.3-70b-instruct
      source model id
      meta-llama/llama-3.3-70b-instruct
      Original result ↗

      OpenRouter Benchmarks

      28.533%

      source id
      openRouterBenchmarks
      leaderboard name
      tau2-bench Verified Airline · Accuracy · Together
      dimension
      agentic
      metric
      tau2-bench Verified Airline · Accuracy
      rank
      22
      sample size
      250
      aggregation scope
      status
      published
      harness
      url
      https://github.com/OpenRouterTeam/benchmark-harness
      name
      OpenRouter benchmark harness
      direction
      higherIsBetter
      run_count
      5
      conditions
      window
      newest 90 days; successful visible nonzero runs
      dataset
      benchmark-leaderboard
      benchmark
      id
      tau_bench_verified_airline
      name
      tau2-bench Verified Airline
      record_kind
      aggregate
      provider_name
      Together
      attribution
      Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
      last_run_at
      2026-10-17T19:19:08.724Z
      license_url
      https://creativecommons.org/licenses/by/4.0/
      methodology
      Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
      metric_kind
      tau2-bench Verified Airline · Accuracy
      rank_method
      competition rank among reported evaluations in identical conditions, including unlinked observations
      sample_size
      250
      source_file
      benchmark-leaderboard.json
      source_page
      https://openrouter.ai/benchmarks/tau2-bench-airline
      measurements
      accuracy
      0.28533339999999996
      accuracy_std_dev
      0.02872100586400135
      cost_per_task_usd
      0.174218524
      output_tokens_per_task
      1889.128
      latency_seconds_per_task
      145.567084
      published_at
      2026-11-16T01:46:02.653Z
      snapshot_date
      2026-11-16
      source_revision
      7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
      transformations
      TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
      metric_population
      22
      benchmark_evaluation
      true
      openrouter_permaslug
      meta-llama/llama-3.3-70b-instruct
      source model id
      meta-llama/llama-3.3-70b-instruct
      Original result ↗

      OpenRouter Benchmarks

      46.886%

      source id
      openRouterBenchmarks
      leaderboard name
      GPQA Diamond · Accuracy · Nebius
      dimension
      knowledge
      metric
      GPQA Diamond · Accuracy
      rank
      24
      sample size
      396
      aggregation scope
      status
      published
      harness
      url
      https://github.com/OpenRouterTeam/benchmark-harness
      name
      OpenRouter benchmark harness
      direction
      higherIsBetter
      run_count
      2
      conditions
      window
      newest 90 days; successful visible nonzero runs
      dataset
      benchmark-leaderboard
      benchmark
      id
      gpqa_diamond
      name
      GPQA Diamond
      record_kind
      aggregate
      provider_name
      Nebius
      attribution
      Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
      last_run_at
      2026-08-29T20:09:29.346Z
      license_url
      https://creativecommons.org/licenses/by/4.0/
      methodology
      Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
      metric_kind
      GPQA Diamond · Accuracy
      rank_method
      competition rank among reported evaluations in identical conditions, including unlinked observations
      sample_size
      396
      source_file
      benchmark-leaderboard.json
      source_page
      https://openrouter.ai/benchmarks/gpqa-diamond
      measurements
      accuracy
      0.468855
      accuracy_std_dev
      0.012625999999999998
      cost_per_task_usd
      0.001083489898989899
      output_tokens_per_task
      2445.7929292929293
      latency_seconds_per_task
      264.26116161616164
      published_at
      2026-11-16T01:46:02.653Z
      snapshot_date
      2026-11-16
      source_revision
      7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
      transformations
      TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
      metric_population
      25
      benchmark_evaluation
      true
      openrouter_permaslug
      meta-llama/llama-3.3-70b-instruct
      source model id
      meta-llama/llama-3.3-70b-instruct
      Original result ↗

      OpenRouter Benchmarks

      47.508%

      source id
      openRouterBenchmarks
      leaderboard name
      GPQA Diamond · Accuracy · Together
      dimension
      knowledge
      metric
      GPQA Diamond · Accuracy
      rank
      25
      sample size
      990
      aggregation scope
      status
      published
      harness
      url
      https://github.com/OpenRouterTeam/benchmark-harness
      name
      OpenRouter benchmark harness
      direction
      higherIsBetter
      run_count
      5
      conditions
      window
      newest 90 days; successful visible nonzero runs
      dataset
      benchmark-leaderboard
      benchmark
      id
      gpqa_diamond
      name
      GPQA Diamond
      record_kind
      aggregate
      provider_name
      Together
      attribution
      Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.
      last_run_at
      2026-10-17T16:05:55.476Z
      license_url
      https://creativecommons.org/licenses/by/4.0/
      methodology
      Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.
      metric_kind
      GPQA Diamond · Accuracy
      rank_method
      competition rank among reported evaluations in identical conditions, including unlinked observations
      sample_size
      990
      source_file
      benchmark-leaderboard.json
      source_page
      https://openrouter.ai/benchmarks/gpqa-diamond
      measurements
      accuracy
      0.475084
      accuracy_std_dev
      0.006683163113376778
      cost_per_task_usd
      0.0031415444444444444
      output_tokens_per_task
      2214.6858585858586
      latency_seconds_per_task
      73.69487777777778
      published_at
      2026-11-16T01:46:02.653Z
      snapshot_date
      2026-11-16
      source_revision
      7ccaf17664252a3053dcb7df8ad0a46e72954a3b4524ddc1c8a7ef8a41b15568
      transformations
      TensorTrace preserves source results, converts latency to seconds and derives ranks within matching conditions. Individual runs have no derived rank.
      metric_population
      25
      benchmark_evaluation
      true
      openrouter_permaslug
      meta-llama/llama-3.3-70b-instruct
      source model id
      meta-llama/llama-3.3-70b-instruct
      Original result ↗

      Published prices

      1–5 / 5

      Provider specifications & access

      1–3 / 3

      Llama-3.3-70B-Instruct ↗

      source model id
      meta-llama/Llama-3.3-70B-Instruct
      hosting
      siliconflow-hosted
      platforms
      • SiliconFlow
      architecture
      input_modalities
      • text
      output_modalities
      • text
      capabilities
      • structuredOutputs
      • toolCalling

      Llama-3-3-70B-Instruct ↗

      source model id
      publishers/hf-meta-llama/models/llama-3.3-70b-instruct@001
      hosting
      vertex-self-deployed
      platforms
      • Google Cloud Vertex AI
      architecture
      input_modalities
      • text
      output_modalities
      • text
      capabilities

        Llama-3-3-70B-Instruct ↗

        source model id
        publishers/meta/models/llama3-3@llama-3.3-70b-instruct
        hosting
        vertex-self-deployed
        platforms
        • Google Cloud Vertex AI
        architecture
        input_modalities
        • text
        output_modalities
        • text
        capabilities

          Safety assessments

          1–1 / 1

          Foundation Model Transparency Index 2025 ↗

          31 / 100

          source id
          stanfordFMTI
          canonical model id
          meta-llama/llama-3.3-70b-instruct
          taxonomy
          transparency
          hazard category
          transparency
          hazard name
          Organization transparency
          policy area
          Meta
          assessment type
          organization-level disclosure
          metric
          disclosure_score
          result label
          31 / 100
          sample size
          100
          updated at
          2026-11-09T18:08:05.227Z

          Source specification observations

          1–4 / 4

          openRouter ↗

          field
          context_length
          value
          131072
          normalized value
          131072
          scope
          source-model
          source id
          openRouter
          confidence
          1

          siliconFlow ↗

          field
          context_length
          value
          33K
          scope
          source-model
          source id
          siliconFlow
          confidence
          1

          openRouter ↗

          field
          knowledge_cutoff
          reported field
          knowledge_cutoff
          value
          2023-12-31
          normalized value
          2023-12-31
          precision
          day
          scope
          source-model
          source id
          openRouter
          confidence
          1

          openRouter ↗

          field
          max_output_tokens
          value
          115200
          normalized value
          115200
          scope
          source-model
          source id
          openRouter
          confidence
          1

          Original sources

          1–18 / 18

          SiliconFlow · Model page ↗

          Hugging Face · Model repository ↗

          DeepInfra · Model page ↗

          Google Cloud · Model page ↗

          Google Cloud · Model page ↗

          OpenRouter · Model page ↗

          SiliconFlow · Playground ↗

          SiliconFlow · API documentation ↗

          DeepInfra · Catalog ↗

          GitHub · Catalog ↗

          GitHub · Catalog ↗

          Google Cloud · Catalog ↗

          Hugging Face · Catalog ↗

          Hugging Face · Catalog ↗

          IBM · Catalog ↗

          OpenRouter · Catalog ↗

          OpenRouter · Catalog ↗

          SiliconFlow · Catalog ↗

          Source links

          1–12 / 12

          Explore more models

          Models

          Loading
          EVIDENCE / COMPARISON

          Benchmarks

          Explore every benchmark

          Loading

          These charts contain model-linked results. The charts above also include unlinked source profiles. Compare scores within each evaluation. Model quality, usage, transparency and system performance measure different things.

          Install TensorTrace