LLM-Perf

Indexed results from LLM-Perf. Each ranking retains its own metric and evaluation conditions.

decode latency p95

llmPerf · LLM-Perf · decode latency p95 · seconds · Lower is better

  1. openai-community/gpt2openai-community0.999
  2. openai-community/gpt2-largeopenai-community2.89
  3. microsoft/rho-math-1b-v0.1microsoft3.179
  4. microsoft/phi-1_5microsoft3.64
  5. stabilityai/stablelm-2-1_6bstabilityai3.802
  6. togethercomputer/RedPajama-INCITE-Base-3B-v1togethercomputer7.096
  7. stabilityai/stablelm-3b-4e1tstabilityai7.301
  8. stabilityai/stablelm-base-alpha-3bstabilityai8.721
  9. togethercomputer/RedPajama-INCITE-Base-7B-v0.1togethercomputer17.328
  10. LLaMA3_8Bmeta-llama18.898
  11. stabilityai/stablelm-base-alpha-7bstabilityai19.188
  12. Mistral-7B-v0.1mistralai19.656

prefill latency p50

llmPerf · LLM-Perf · prefill latency p50 · seconds · Lower is better

  1. facebook/opt-125mmeta-llama0.011

decode memory max_global_vram

llmPerf · LLM-Perf · decode memory max_global_vram · megabytes · Lower is better

  1. EleutherAI/pythia-70meleutherai878.182
  2. EleutherAI/pythia-160meleutherai1,027.08
  3. EleutherAI/pythia-410meleutherai1,318.584
  4. EleutherAI/pythia-1.3beleutherai2,157.445
  5. EleutherAI/pythia-1.4beleutherai2,157.445
  6. EleutherAI/pythia-2.7beleutherai3,380.085
  7. EleutherAI/pythia-6.7beleutherai6,418.858
  8. EleutherAI/pythia-12beleutherai10,116.137
  9. EleutherAI/polyglot-ko-12.8beleutherai10,460.07
  10. EleutherAI/gpt-neox-20beleutherai15,973.482

per_token latency mean

llmPerf · LLM-Perf · per_token latency mean · seconds · Lower is better

  1. Qwen/Qwen1.5-MoE-A2.7Bqwen0.138

load energy ram

llmPerf · LLM-Perf · load energy ram · kilowattHours · Lower is better

  1. facebook/opt-125mmeta-llama0

prefill energy ram

llmPerf · LLM-Perf · prefill energy ram · kilowattHours · Lower is better

  1. openai-community/gpt2-largeopenai-community0

prefill throughput value

llmPerf · LLM-Perf · prefill throughput value · tokensPerSecond · Higher is better

  1. Deci/DeciLM-7Bdeci145.975
  2. Deci/DeciLM-7Bdeci145.401
  3. Deci/DeciLM-7Bdeci122.653

decode latency p90

llmPerf · LLM-Perf · decode latency p90 · seconds · Lower is better

  1. 01-ai/Yi-6B01-ai15.095
  2. 01-ai/Yi-34B01-ai88.14

per_token throughput value

llmPerf · LLM-Perf · per_token throughput value · tokensPerSecond · Higher is better

  1. 01-ai/Yi-34B01-ai16.754

load latency mean

llmPerf · LLM-Perf · load latency mean · seconds · Lower is better

  1. TencentARC/Mistral_Pro_8B_v0.1tencentarc12.808

load memory max_ram

llmPerf · LLM-Perf · load memory max_ram · megabytes · Lower is better

  1. Salesforce/codegen-6B-nlsalesforce14,375.453

decode latency p90

llmPerf · LLM-Perf · decode latency p90 · seconds · Lower is better

  1. 01-ai/Yi-6B01-ai1.635
  2. 01-ai/Yi-34B01-ai3.535

Models

Loading
EVIDENCE / COMPARISON

Benchmarks

Explore every benchmark

Loading

These charts contain model-linked results. The charts above also include unlinked source profiles. Compare scores within each evaluation. Model quality, usage, transparency and system performance measure different things.

Install TensorTrace