LLM-Perf
Indexed results from LLM-Perf. Each ranking retains its own metric and evaluation conditions.
decode latency p95
llmPerf · LLM-Perf · decode latency p95 · seconds · Lower is better
- openai-community/gpt2openai-community0.999
- openai-community/gpt2-largeopenai-community2.89
- microsoft/rho-math-1b-v0.1microsoft3.179
- microsoft/phi-1_5microsoft3.64
- stabilityai/stablelm-2-1_6bstabilityai3.802
- togethercomputer/RedPajama-INCITE-Base-3B-v1togethercomputer7.096
- stabilityai/stablelm-3b-4e1tstabilityai7.301
- stabilityai/stablelm-base-alpha-3bstabilityai8.721
- togethercomputer/RedPajama-INCITE-Base-7B-v0.1togethercomputer17.328
- LLaMA3_8Bmeta-llama18.898
- stabilityai/stablelm-base-alpha-7bstabilityai19.188
- Mistral-7B-v0.1mistralai19.656
prefill latency p50
llmPerf · LLM-Perf · prefill latency p50 · seconds · Lower is better
decode memory max_global_vram
llmPerf · LLM-Perf · decode memory max_global_vram · megabytes · Lower is better
- EleutherAI/pythia-70meleutherai878.182
- EleutherAI/pythia-160meleutherai1,027.08
- EleutherAI/pythia-410meleutherai1,318.584
- EleutherAI/pythia-1.3beleutherai2,157.445
- EleutherAI/pythia-1.4beleutherai2,157.445
- EleutherAI/pythia-2.7beleutherai3,380.085
- EleutherAI/pythia-6.7beleutherai6,418.858
- EleutherAI/pythia-12beleutherai10,116.137
- EleutherAI/polyglot-ko-12.8beleutherai10,460.07
- EleutherAI/gpt-neox-20beleutherai15,973.482
per_token latency mean
llmPerf · LLM-Perf · per_token latency mean · seconds · Lower is better
load energy ram
llmPerf · LLM-Perf · load energy ram · kilowattHours · Lower is better
prefill energy ram
llmPerf · LLM-Perf · prefill energy ram · kilowattHours · Lower is better
prefill throughput value
llmPerf · LLM-Perf · prefill throughput value · tokensPerSecond · Higher is better
- Deci/DeciLM-7Bdeci145.975
- Deci/DeciLM-7Bdeci145.401
- Deci/DeciLM-7Bdeci122.653
decode latency p90
llmPerf · LLM-Perf · decode latency p90 · seconds · Lower is better
- 01-ai/Yi-6B01-ai15.095
- 01-ai/Yi-34B01-ai88.14
per_token throughput value
llmPerf · LLM-Perf · per_token throughput value · tokensPerSecond · Higher is better
- 01-ai/Yi-34B01-ai16.754
load latency mean
llmPerf · LLM-Perf · load latency mean · seconds · Lower is better
load memory max_ram
llmPerf · LLM-Perf · load memory max_ram · megabytes · Lower is better
- Salesforce/codegen-6B-nlsalesforce14,375.453
decode latency p90
llmPerf · LLM-Perf · decode latency p90 · seconds · Lower is better
- 01-ai/Yi-6B01-ai1.635
- 01-ai/Yi-34B01-ai3.535
Models
LoadingBenchmarks
Explore every benchmark
LoadingThese charts contain model-linked results. The charts above also include unlinked source profiles. Compare scores within each evaluation. Model quality, usage, transparency and system performance measure different things.