01-ai/Yi-34B

`01-ai/Yi-34B` powered by Text Generation Inference (TGI) - [Example Notebook](https://aka.ms/hugging-face-text-generation-streaming-online-endpoint) - [Original Model Card](https://huggingface.co/01…

azureFoundry/HuggingFace/01-ai-yi-34b@1

01-ai/Yi-34B powered by Text Generation Inference (TGI)

Send Request

You can use cURL or any REST Client to send a request to the AzureML endpoint with your AzureML token.

curl <AZUREML_ENDPOINT_URL> \
    -X POST \
    -d '{"inputs":"What is Deep Learning?"}' \
    -H "Authorization: Bearer <AZUREML_TOKEN>" \
    -H "Content-Type: application/json"

Supported Parameters

  • inputs (string): Input prompt.
  • parameters (object):
    • best_of (integer): Generate best_of sequences and return the one if the highest token logprobs.
    • decoder_input_details (boolean): Whether to return decoder input token logprobs and ids.
    • details (boolean): Whether to return generation details.
    • do_sample (boolean): Activate logits sampling.
    • frequency_penalty (float): The parameter for frequency penalty. 1.0 means no penalty Penalize new tokens based on their existing frequency in the text so far, decreasing the model’s likelihood to repeat the same line verbatim.
    • grammar (object): One of the following
      • #1 (object):
        • type (enum): Possible values: json.
        • value (string): A string that represents a JSON Schema. JSON Schema is a declarative language that allows to annotate JSON documents with types and descriptions.
      • #2 (object):
        • type (enum): Possible values: regex.
        • value (string): The regular expression.
      • #3 (object):
        • type (enum): Possible values: json_schema.
        • value (object):
          • name (string): Optional name identifier for the schema
          • schema (object): The actual JSON schema definition
    • max_new_tokens (integer): Maximum number of tokens to generate.
    • repetition_penalty (float): The parameter for repetition penalty. 1.0 means no penalty. See this paper for more details.
    • return_full_text (boolean): Whether to prepend the prompt to the generated text
    • seed (integer): Random sampling seed.
    • stop (string[]): Stop generating tokens if a member of stop is generated.
    • temperature (float): The value used to module the logits distribution.
    • top_k (integer): The number of highest probability vocabulary tokens to keep for top-k-filtering.
    • top_n_tokens (integer): The number of highest probability vocabulary tokens to keep for top-n-filtering.
    • top_p (float): Top-p value for nucleus sampling.
    • truncate (integer): Truncate inputs tokens to the given size.
    • typical_p (float): Typical Decoding mass See Typical Decoding for Natural Language Generation for more information.
    • watermark (boolean): Watermarking with A Watermark for Large Language Models.
    • stream (boolean): Whether to stream the output tokens or not. Defaults to false.

Example payload

{
  "inputs": "What is Deep Learning?",
  "parameters": {
    "do_sample": true,
    "top_p": 0.95,
    "temperature": 0.2,
    "top_k": 50,
    "max_new_tokens": 256,
    "repetition_penalty": 1.03,
    "stop": ["\nUser:", "<|endoftext|>", "</s>"]
  }
}

OpenAI Chat Completion API compatibility

Additionally, Text Generation Inference (TGI) offers an OpenAI Chat Completion API compatible layer under the endpoint /v1/chat/completions, check the full specification in the OpenAI Chat Completion Create Documentation.

Specifications & capabilities

Input
text
Output
text
Published model ID
Yi-34B

Evaluations

1–20 / 5,420

LLM-Perf

11,227,562.343

source id
llmPerf
leaderboard name
prefill efficiency value
dimension
runtime
metric
prefill efficiency value
rank
1
aggregation scope
status
published
direction
higherIsBetter
conditions
suite
optimum-benchmark/llm-perf-leaderboard/perf-df-pytorch-cuda-unquantized-1xA100.csv
execution
config.backend.name
pytorch
config.backend.seed
42
config.backend.task
text-generation
config.backend.device
cuda
config.backend.library
transformers
config.backend.version
2.3.0+cu121
config.environment.cpu
AMD EPYC 7742 64-Core Processor
config.environment.gpu
['NVIDIA A100-SXM4-80GB']
config.backend.eval_mode
True
config.backend.device_ids
0
config.backend.no_weights
True
config.environment.system
Linux
config.backend.torch_dtype
bfloat16
config.environment.machine
x86_64
config.backend.torch_compile
False
config.environment.cpu_count
128
config.environment.gpu_count
1
config.environment.processor
x86_64
config.environment.cpu_ram_mb
540671.627264
config.environment.gpu_vram_mb
85899345920
config.backend.autocast_enabled
False
config.environment.peft_version
0.11.0
config.environment.python_version
3.10.12
config.backend.attn_implementation
eager
config.backend.deepspeed_inference
False
config.environment.optimum_version
1.19.2
config.backend.to_bettertransformer
False
config.backend.torch_compile_target
forward
config.environment.accelerate_version
0.30.1
config.environment.transformers_version
4.40.2
config.scenario.input_shapes.batch_size
1
config.scenario.input_shapes.num_choices
2
config.backend.hub_kwargs.trust_remote_code
True
config.environment.optimum_benchmark_version
0.2.1
config.scenario.input_shapes.sequence_length
256
config.backend.model_kwargs.trust_remote_code
True
config.scenario.generate_kwargs.max_new_tokens
64
config.scenario.generate_kwargs.min_new_tokens
64
config.backend.processor_kwargs.trust_remote_code
True
metric_kind
prefill efficiency value
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
optimum-benchmark/llm-perf-leaderboard/perf-df-pytorch-cuda-unquantized-1xA100.csv
published_at
2024-12-13T10:07:03Z
source_revision
2fa8d8c300c4eb6e744d5e35999db926c0c26e90e767e4c38da326eebe21a92b
metric_population
1
benchmark_evaluation
true
comparison_configuration_sha256
5a7bfe2f08af6456f222f387892347ad2dc4618106b698f9ade8192bb1e13c85
complete_configuration_location
source record config and original CSV artifact
source model id
01-ai/Yi-34B
Original result ↗

LLM-Perf

11,227,562.343

source id
llmPerf
leaderboard name
prefill efficiency value
dimension
runtime
metric
prefill efficiency value
rank
1
aggregation scope
status
published
direction
higherIsBetter
conditions
suite
optimum-benchmark/llm-perf-leaderboard/perf-df-unquantized-1xA100.csv
execution
config.backend.name
pytorch
config.backend.seed
42
config.backend.task
text-generation
config.backend.device
cuda
config.backend.library
transformers
config.backend.version
2.3.0+cu121
config.environment.cpu
AMD EPYC 7742 64-Core Processor
config.environment.gpu
['NVIDIA A100-SXM4-80GB']
config.backend.eval_mode
True
config.backend.device_ids
0
config.backend.no_weights
True
config.environment.system
Linux
config.backend.torch_dtype
bfloat16
config.environment.machine
x86_64
config.backend.torch_compile
False
config.environment.cpu_count
128
config.environment.gpu_count
1
config.environment.processor
x86_64
config.environment.cpu_ram_mb
540671.627264
config.environment.gpu_vram_mb
85899345920
config.backend.autocast_enabled
False
config.environment.peft_version
0.11.0
config.environment.python_version
3.10.12
config.backend.attn_implementation
eager
config.backend.deepspeed_inference
False
config.environment.optimum_version
1.19.2
config.backend.to_bettertransformer
False
config.backend.torch_compile_target
forward
config.environment.accelerate_version
0.30.1
config.environment.transformers_version
4.40.2
config.scenario.input_shapes.batch_size
1
config.scenario.input_shapes.num_choices
2
config.backend.hub_kwargs.trust_remote_code
True
config.environment.optimum_benchmark_version
0.2.1
config.scenario.input_shapes.sequence_length
256
config.backend.model_kwargs.trust_remote_code
True
config.scenario.generate_kwargs.max_new_tokens
64
config.scenario.generate_kwargs.min_new_tokens
64
config.backend.processor_kwargs.trust_remote_code
True
metric_kind
prefill efficiency value
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
optimum-benchmark/llm-perf-leaderboard/perf-df-unquantized-1xA100.csv
published_at
2024-12-13T10:07:03Z
source_revision
2fa8d8c300c4eb6e744d5e35999db926c0c26e90e767e4c38da326eebe21a92b
metric_population
1
benchmark_evaluation
true
comparison_configuration_sha256
468e8460883515023a1c9c7cf5a09717acb57fa4558b60901a8ccbf9d5f72121
complete_configuration_location
source record config and original CSV artifact
source model id
01-ai/Yi-34B
Original result ↗

LLM-Perf

11,058,326.031

source id
llmPerf
leaderboard name
prefill efficiency value
dimension
runtime
metric
prefill efficiency value
rank
1
aggregation scope
status
published
direction
higherIsBetter
conditions
suite
optimum-benchmark/llm-perf-leaderboard/perf-df-pytorch-cuda-unquantized-1xA100.csv
execution
config.backend.name
pytorch
config.backend.seed
42
config.backend.task
text-generation
config.backend.device
cuda
config.backend.library
transformers
config.backend.version
2.3.0+cu121
config.environment.cpu
AMD EPYC 7742 64-Core Processor
config.environment.gpu
['NVIDIA A100-SXM4-80GB']
config.backend.eval_mode
True
config.backend.device_ids
0
config.backend.no_weights
True
config.environment.system
Linux
config.backend.torch_dtype
float16
config.environment.machine
x86_64
config.backend.torch_compile
False
config.environment.cpu_count
128
config.environment.gpu_count
1
config.environment.processor
x86_64
config.environment.cpu_ram_mb
540671.627264
config.environment.gpu_vram_mb
85899345920
config.backend.autocast_enabled
False
config.environment.peft_version
0.11.0
config.environment.python_version
3.10.12
config.backend.attn_implementation
sdpa
config.backend.deepspeed_inference
False
config.environment.optimum_version
1.19.2
config.backend.to_bettertransformer
False
config.backend.torch_compile_target
forward
config.environment.accelerate_version
0.30.1
config.environment.transformers_version
4.40.2
config.scenario.input_shapes.batch_size
1
config.scenario.input_shapes.num_choices
2
config.backend.hub_kwargs.trust_remote_code
True
config.environment.optimum_benchmark_version
0.2.1
config.scenario.input_shapes.sequence_length
256
config.backend.model_kwargs.trust_remote_code
True
config.scenario.generate_kwargs.max_new_tokens
64
config.scenario.generate_kwargs.min_new_tokens
64
config.backend.processor_kwargs.trust_remote_code
True
metric_kind
prefill efficiency value
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
optimum-benchmark/llm-perf-leaderboard/perf-df-pytorch-cuda-unquantized-1xA100.csv
published_at
2024-12-13T10:07:03Z
source_revision
2fa8d8c300c4eb6e744d5e35999db926c0c26e90e767e4c38da326eebe21a92b
metric_population
1
benchmark_evaluation
true
comparison_configuration_sha256
e98269fb859d479cf594053da82963997bf4c42df3b1a0f461c21547439bdc33
complete_configuration_location
source record config and original CSV artifact
source model id
01-ai/Yi-34B
Original result ↗

LLM-Perf

11,058,326.031

source id
llmPerf
leaderboard name
prefill efficiency value
dimension
runtime
metric
prefill efficiency value
rank
1
aggregation scope
status
published
direction
higherIsBetter
conditions
suite
optimum-benchmark/llm-perf-leaderboard/perf-df-unquantized-1xA100.csv
execution
config.backend.name
pytorch
config.backend.seed
42
config.backend.task
text-generation
config.backend.device
cuda
config.backend.library
transformers
config.backend.version
2.3.0+cu121
config.environment.cpu
AMD EPYC 7742 64-Core Processor
config.environment.gpu
['NVIDIA A100-SXM4-80GB']
config.backend.eval_mode
True
config.backend.device_ids
0
config.backend.no_weights
True
config.environment.system
Linux
config.backend.torch_dtype
float16
config.environment.machine
x86_64
config.backend.torch_compile
False
config.environment.cpu_count
128
config.environment.gpu_count
1
config.environment.processor
x86_64
config.environment.cpu_ram_mb
540671.627264
config.environment.gpu_vram_mb
85899345920
config.backend.autocast_enabled
False
config.environment.peft_version
0.11.0
config.environment.python_version
3.10.12
config.backend.attn_implementation
sdpa
config.backend.deepspeed_inference
False
config.environment.optimum_version
1.19.2
config.backend.to_bettertransformer
False
config.backend.torch_compile_target
forward
config.environment.accelerate_version
0.30.1
config.environment.transformers_version
4.40.2
config.scenario.input_shapes.batch_size
1
config.scenario.input_shapes.num_choices
2
config.backend.hub_kwargs.trust_remote_code
True
config.environment.optimum_benchmark_version
0.2.1
config.scenario.input_shapes.sequence_length
256
config.backend.model_kwargs.trust_remote_code
True
config.scenario.generate_kwargs.max_new_tokens
64
config.scenario.generate_kwargs.min_new_tokens
64
config.backend.processor_kwargs.trust_remote_code
True
metric_kind
prefill efficiency value
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
optimum-benchmark/llm-perf-leaderboard/perf-df-unquantized-1xA100.csv
published_at
2024-12-13T10:07:03Z
source_revision
2fa8d8c300c4eb6e744d5e35999db926c0c26e90e767e4c38da326eebe21a92b
metric_population
1
benchmark_evaluation
true
comparison_configuration_sha256
4d1ad767823f5984c1333a2c80a1b8794a0ef5ac80d44cde71dc4faf09969111
complete_configuration_location
source record config and original CSV artifact
source model id
01-ai/Yi-34B
Original result ↗

LLM-Perf

9,159,947.823

source id
llmPerf
leaderboard name
prefill efficiency value
dimension
runtime
metric
prefill efficiency value
rank
1
aggregation scope
status
published
direction
higherIsBetter
conditions
suite
optimum-benchmark/llm-perf-leaderboard/perf-df-bnb-1xA100.csv
execution
config.backend.name
pytorch
config.backend.seed
42
config.backend.task
text-generation
config.backend.device
cuda
config.backend.library
transformers
config.backend.version
2.3.0+cu121
config.environment.cpu
AMD EPYC 7742 64-Core Processor
config.environment.gpu
['NVIDIA A100-SXM4-80GB']
config.backend.eval_mode
True
config.backend.device_ids
0
config.backend.no_weights
True
config.environment.system
Linux
config.backend.torch_dtype
float16
config.environment.machine
x86_64
config.backend.torch_compile
False
config.environment.cpu_count
128
config.environment.gpu_count
1
config.environment.processor
x86_64
config.environment.cpu_ram_mb
540671.627264
config.environment.gpu_vram_mb
85899345920
config.backend.autocast_enabled
False
config.environment.peft_version
0.10.0
config.environment.python_version
3.10.12
config.backend.attn_implementation
flash_attention_2
config.backend.deepspeed_inference
False
config.backend.quantization_scheme
bnb
config.environment.optimum_version
1.19.2
config.backend.to_bettertransformer
False
config.backend.torch_compile_target
forward
config.environment.accelerate_version
0.30.1
config.environment.transformers_version
4.40.2
config.scenario.input_shapes.batch_size
1
config.backend.hub_kwargs.force_download
False
config.scenario.input_shapes.num_choices
2
config.backend.hub_kwargs.local_files_only
False
config.backend.hub_kwargs.trust_remote_code
True
config.environment.optimum_benchmark_version
0.2.0
config.scenario.input_shapes.sequence_length
256
config.scenario.generate_kwargs.max_new_tokens
64
config.scenario.generate_kwargs.min_new_tokens
64
config.backend.quantization_config.load_in_8bit
True
config.backend.quantization_config.llm_int8_threshold
0.0
metric_kind
prefill efficiency value
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
optimum-benchmark/llm-perf-leaderboard/perf-df-bnb-1xA100.csv
published_at
2024-12-13T10:07:03Z
model_revision
main
source_revision
2fa8d8c300c4eb6e744d5e35999db926c0c26e90e767e4c38da326eebe21a92b
metric_population
1
benchmark_evaluation
true
comparison_configuration_sha256
81adf66f4a19b726dd28a74703fc03e3e0575c6c378fa7ea9507a8798b73e641
complete_configuration_location
source record config and original CSV artifact
source model id
01-ai/Yi-34B
Original result ↗

LLM-Perf

9,159,947.823

source id
llmPerf
leaderboard name
prefill efficiency value
dimension
runtime
metric
prefill efficiency value
rank
1
aggregation scope
status
published
direction
higherIsBetter
conditions
suite
optimum-benchmark/llm-perf-leaderboard/perf-df-pytorch-cuda-bnb-1xA100.csv
execution
config.backend.name
pytorch
config.backend.seed
42
config.backend.task
text-generation
config.backend.device
cuda
config.backend.library
transformers
config.backend.version
2.3.0+cu121
config.environment.cpu
AMD EPYC 7742 64-Core Processor
config.environment.gpu
['NVIDIA A100-SXM4-80GB']
config.backend.eval_mode
True
config.backend.device_ids
0
config.backend.no_weights
True
config.environment.system
Linux
config.backend.torch_dtype
float16
config.environment.machine
x86_64
config.backend.torch_compile
False
config.environment.cpu_count
128
config.environment.gpu_count
1
config.environment.processor
x86_64
config.environment.cpu_ram_mb
540671.627264
config.environment.gpu_vram_mb
85899345920
config.backend.autocast_enabled
False
config.environment.peft_version
0.10.0
config.environment.python_version
3.10.12
config.backend.attn_implementation
flash_attention_2
config.backend.deepspeed_inference
False
config.backend.quantization_scheme
bnb
config.environment.optimum_version
1.19.2
config.backend.to_bettertransformer
False
config.backend.torch_compile_target
forward
config.environment.accelerate_version
0.30.1
config.environment.transformers_version
4.40.2
config.scenario.input_shapes.batch_size
1
config.backend.hub_kwargs.force_download
False
config.scenario.input_shapes.num_choices
2
config.backend.hub_kwargs.local_files_only
False
config.backend.hub_kwargs.trust_remote_code
True
config.environment.optimum_benchmark_version
0.2.0
config.scenario.input_shapes.sequence_length
256
config.scenario.generate_kwargs.max_new_tokens
64
config.scenario.generate_kwargs.min_new_tokens
64
config.backend.quantization_config.load_in_8bit
True
config.backend.quantization_config.llm_int8_threshold
0.0
metric_kind
prefill efficiency value
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
optimum-benchmark/llm-perf-leaderboard/perf-df-pytorch-cuda-bnb-1xA100.csv
published_at
2024-12-13T10:07:03Z
model_revision
main
source_revision
2fa8d8c300c4eb6e744d5e35999db926c0c26e90e767e4c38da326eebe21a92b
metric_population
1
benchmark_evaluation
true
comparison_configuration_sha256
cf77fea8612fd42a1c463e7d78f09523d155d530f59f4467d2af2bc79f575ff5
complete_configuration_location
source record config and original CSV artifact
source model id
01-ai/Yi-34B
Original result ↗

LLM-Perf

9,014,223.881

source id
llmPerf
leaderboard name
prefill efficiency value
dimension
runtime
metric
prefill efficiency value
rank
1
aggregation scope
status
published
direction
higherIsBetter
conditions
suite
optimum-benchmark/llm-perf-leaderboard/perf-df-bnb-1xA100.csv
execution
config.backend.name
pytorch
config.backend.seed
42
config.backend.task
text-generation
config.backend.device
cuda
config.backend.library
transformers
config.backend.version
2.3.0+cu121
config.environment.cpu
AMD EPYC 7742 64-Core Processor
config.environment.gpu
['NVIDIA A100-SXM4-80GB']
config.backend.eval_mode
True
config.backend.device_ids
0
config.backend.no_weights
True
config.environment.system
Linux
config.backend.torch_dtype
float16
config.environment.machine
x86_64
config.backend.torch_compile
False
config.environment.cpu_count
128
config.environment.gpu_count
1
config.environment.processor
x86_64
config.environment.cpu_ram_mb
540671.627264
config.environment.gpu_vram_mb
85899345920
config.backend.autocast_enabled
False
config.environment.peft_version
0.10.0
config.environment.python_version
3.10.12
config.backend.attn_implementation
sdpa
config.backend.deepspeed_inference
False
config.backend.quantization_scheme
bnb
config.environment.optimum_version
1.19.2
config.backend.to_bettertransformer
False
config.backend.torch_compile_target
forward
config.environment.accelerate_version
0.30.1
config.environment.transformers_version
4.40.2
config.scenario.input_shapes.batch_size
1
config.backend.hub_kwargs.force_download
False
config.scenario.input_shapes.num_choices
2
config.backend.hub_kwargs.local_files_only
False
config.backend.hub_kwargs.trust_remote_code
True
config.environment.optimum_benchmark_version
0.2.0
config.scenario.input_shapes.sequence_length
256
config.scenario.generate_kwargs.max_new_tokens
64
config.scenario.generate_kwargs.min_new_tokens
64
config.backend.quantization_config.load_in_8bit
True
config.backend.quantization_config.llm_int8_threshold
0.0
metric_kind
prefill efficiency value
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
optimum-benchmark/llm-perf-leaderboard/perf-df-bnb-1xA100.csv
published_at
2024-12-13T10:07:03Z
model_revision
main
source_revision
2fa8d8c300c4eb6e744d5e35999db926c0c26e90e767e4c38da326eebe21a92b
metric_population
1
benchmark_evaluation
true
comparison_configuration_sha256
f2ac0cf601f4900499ae6c19b07a7e27679f0285c79bb4bdb6323388016fcf66
complete_configuration_location
source record config and original CSV artifact
source model id
01-ai/Yi-34B
Original result ↗

LLM-Perf

9,014,223.881

source id
llmPerf
leaderboard name
prefill efficiency value
dimension
runtime
metric
prefill efficiency value
rank
1
aggregation scope
status
published
direction
higherIsBetter
conditions
suite
optimum-benchmark/llm-perf-leaderboard/perf-df-pytorch-cuda-bnb-1xA100.csv
execution
config.backend.name
pytorch
config.backend.seed
42
config.backend.task
text-generation
config.backend.device
cuda
config.backend.library
transformers
config.backend.version
2.3.0+cu121
config.environment.cpu
AMD EPYC 7742 64-Core Processor
config.environment.gpu
['NVIDIA A100-SXM4-80GB']
config.backend.eval_mode
True
config.backend.device_ids
0
config.backend.no_weights
True
config.environment.system
Linux
config.backend.torch_dtype
float16
config.environment.machine
x86_64
config.backend.torch_compile
False
config.environment.cpu_count
128
config.environment.gpu_count
1
config.environment.processor
x86_64
config.environment.cpu_ram_mb
540671.627264
config.environment.gpu_vram_mb
85899345920
config.backend.autocast_enabled
False
config.environment.peft_version
0.10.0
config.environment.python_version
3.10.12
config.backend.attn_implementation
sdpa
config.backend.deepspeed_inference
False
config.backend.quantization_scheme
bnb
config.environment.optimum_version
1.19.2
config.backend.to_bettertransformer
False
config.backend.torch_compile_target
forward
config.environment.accelerate_version
0.30.1
config.environment.transformers_version
4.40.2
config.scenario.input_shapes.batch_size
1
config.backend.hub_kwargs.force_download
False
config.scenario.input_shapes.num_choices
2
config.backend.hub_kwargs.local_files_only
False
config.backend.hub_kwargs.trust_remote_code
True
config.environment.optimum_benchmark_version
0.2.0
config.scenario.input_shapes.sequence_length
256
config.scenario.generate_kwargs.max_new_tokens
64
config.scenario.generate_kwargs.min_new_tokens
64
config.backend.quantization_config.load_in_8bit
True
config.backend.quantization_config.llm_int8_threshold
0.0
metric_kind
prefill efficiency value
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
optimum-benchmark/llm-perf-leaderboard/perf-df-pytorch-cuda-bnb-1xA100.csv
published_at
2024-12-13T10:07:03Z
model_revision
main
source_revision
2fa8d8c300c4eb6e744d5e35999db926c0c26e90e767e4c38da326eebe21a92b
metric_population
1
benchmark_evaluation
true
comparison_configuration_sha256
7c5addcc6af5705fdbbdac7fbfc6a2f7ec70088071eac3b878c47831c3ad1475
complete_configuration_location
source record config and original CSV artifact
source model id
01-ai/Yi-34B
Original result ↗

LLM-Perf

8,843,571.84

source id
llmPerf
leaderboard name
prefill efficiency value
dimension
runtime
metric
prefill efficiency value
rank
1
aggregation scope
status
published
direction
higherIsBetter
conditions
suite
optimum-benchmark/llm-perf-leaderboard/perf-df-bnb-1xA100.csv
execution
config.backend.name
pytorch
config.backend.seed
42
config.backend.task
text-generation
config.backend.device
cuda
config.backend.library
transformers
config.backend.version
2.3.0+cu121
config.environment.cpu
AMD EPYC 7742 64-Core Processor
config.environment.gpu
['NVIDIA A100-SXM4-80GB']
config.backend.eval_mode
True
config.backend.device_ids
0
config.backend.no_weights
True
config.environment.system
Linux
config.backend.torch_dtype
float16
config.environment.machine
x86_64
config.backend.torch_compile
False
config.environment.cpu_count
128
config.environment.gpu_count
1
config.environment.processor
x86_64
config.environment.cpu_ram_mb
540671.627264
config.environment.gpu_vram_mb
85899345920
config.backend.autocast_enabled
False
config.environment.peft_version
0.11.0
config.environment.python_version
3.10.12
config.backend.attn_implementation
eager
config.backend.deepspeed_inference
False
config.backend.quantization_scheme
bnb
config.environment.optimum_version
1.19.2
config.backend.to_bettertransformer
False
config.backend.torch_compile_target
forward
config.environment.accelerate_version
0.30.1
config.environment.transformers_version
4.40.2
config.scenario.input_shapes.batch_size
1
config.scenario.input_shapes.num_choices
2
config.backend.hub_kwargs.trust_remote_code
True
config.environment.optimum_benchmark_version
0.2.1
config.scenario.input_shapes.sequence_length
256
config.backend.model_kwargs.trust_remote_code
True
config.scenario.generate_kwargs.max_new_tokens
64
config.scenario.generate_kwargs.min_new_tokens
64
config.backend.quantization_config.load_in_8bit
True
config.backend.processor_kwargs.trust_remote_code
True
config.backend.quantization_config.llm_int8_threshold
0.0
metric_kind
prefill efficiency value
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
optimum-benchmark/llm-perf-leaderboard/perf-df-bnb-1xA100.csv
published_at
2024-12-13T10:07:03Z
source_revision
2fa8d8c300c4eb6e744d5e35999db926c0c26e90e767e4c38da326eebe21a92b
metric_population
1
benchmark_evaluation
true
comparison_configuration_sha256
ce5a8f864df92bc59b7eff41f383aa36e4d30eb3ed3fb951c5332decbd6cad53
complete_configuration_location
source record config and original CSV artifact
source model id
01-ai/Yi-34B
Original result ↗

LLM-Perf

8,843,571.84

source id
llmPerf
leaderboard name
prefill efficiency value
dimension
runtime
metric
prefill efficiency value
rank
1
aggregation scope
status
published
direction
higherIsBetter
conditions
suite
optimum-benchmark/llm-perf-leaderboard/perf-df-pytorch-cuda-bnb-1xA100.csv
execution
config.backend.name
pytorch
config.backend.seed
42
config.backend.task
text-generation
config.backend.device
cuda
config.backend.library
transformers
config.backend.version
2.3.0+cu121
config.environment.cpu
AMD EPYC 7742 64-Core Processor
config.environment.gpu
['NVIDIA A100-SXM4-80GB']
config.backend.eval_mode
True
config.backend.device_ids
0
config.backend.no_weights
True
config.environment.system
Linux
config.backend.torch_dtype
float16
config.environment.machine
x86_64
config.backend.torch_compile
False
config.environment.cpu_count
128
config.environment.gpu_count
1
config.environment.processor
x86_64
config.environment.cpu_ram_mb
540671.627264
config.environment.gpu_vram_mb
85899345920
config.backend.autocast_enabled
False
config.environment.peft_version
0.11.0
config.environment.python_version
3.10.12
config.backend.attn_implementation
eager
config.backend.deepspeed_inference
False
config.backend.quantization_scheme
bnb
config.environment.optimum_version
1.19.2
config.backend.to_bettertransformer
False
config.backend.torch_compile_target
forward
config.environment.accelerate_version
0.30.1
config.environment.transformers_version
4.40.2
config.scenario.input_shapes.batch_size
1
config.scenario.input_shapes.num_choices
2
config.backend.hub_kwargs.trust_remote_code
True
config.environment.optimum_benchmark_version
0.2.1
config.scenario.input_shapes.sequence_length
256
config.backend.model_kwargs.trust_remote_code
True
config.scenario.generate_kwargs.max_new_tokens
64
config.scenario.generate_kwargs.min_new_tokens
64
config.backend.quantization_config.load_in_8bit
True
config.backend.processor_kwargs.trust_remote_code
True
config.backend.quantization_config.llm_int8_threshold
0.0
metric_kind
prefill efficiency value
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
optimum-benchmark/llm-perf-leaderboard/perf-df-pytorch-cuda-bnb-1xA100.csv
published_at
2024-12-13T10:07:03Z
source_revision
2fa8d8c300c4eb6e744d5e35999db926c0c26e90e767e4c38da326eebe21a92b
metric_population
1
benchmark_evaluation
true
comparison_configuration_sha256
02553f087b1a1ea648046e966ceb20a8fdc6a18636a9c361c517ceba6e273a9b
complete_configuration_location
source record config and original CSV artifact
source model id
01-ai/Yi-34B
Original result ↗

LLM-Perf

1,994,044.42

source id
llmPerf
leaderboard name
prefill efficiency value
dimension
runtime
metric
prefill efficiency value
rank
1
aggregation scope
status
published
direction
higherIsBetter
conditions
suite
optimum-benchmark/llm-perf-leaderboard/perf-df-pytorch-cuda-gptq-1xA100.csv
execution
config.backend.name
pytorch
config.backend.seed
42
config.backend.task
text-generation
config.backend.device
cuda
config.backend.library
transformers
config.backend.version
2.3.0+cu121
config.environment.cpu
AMD EPYC 7742 64-Core Processor
config.environment.gpu
['NVIDIA A100-SXM4-80GB']
config.backend.eval_mode
True
config.backend.device_ids
0
config.backend.no_weights
True
config.environment.system
Linux
config.backend.torch_dtype
float16
config.environment.machine
x86_64
config.backend.torch_compile
False
config.environment.cpu_count
128
config.environment.gpu_count
1
config.environment.processor
x86_64
config.environment.cpu_ram_mb
540671.627264
config.environment.gpu_vram_mb
85899345920
config.backend.autocast_enabled
False
config.environment.peft_version
0.10.0
config.environment.python_version
3.10.12
config.backend.attn_implementation
sdpa
config.backend.deepspeed_inference
False
config.backend.quantization_scheme
gptq
config.environment.optimum_version
1.19.2
config.backend.to_bettertransformer
False
config.backend.torch_compile_target
forward
config.environment.accelerate_version
0.30.1
config.backend.quantization_config.bits
4
config.environment.transformers_version
4.40.2
config.scenario.input_shapes.batch_size
1
config.backend.hub_kwargs.force_download
False
config.scenario.input_shapes.num_choices
2
config.backend.hub_kwargs.local_files_only
False
config.backend.quantization_config.version
1
config.backend.hub_kwargs.trust_remote_code
True
config.environment.optimum_benchmark_version
0.2.0
config.scenario.input_shapes.sequence_length
256
config.scenario.generate_kwargs.max_new_tokens
64
config.scenario.generate_kwargs.min_new_tokens
64
config.backend.quantization_config.model_seqlen
256
config.backend.quantization_config.use_exllama
True
metric_kind
prefill efficiency value
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
optimum-benchmark/llm-perf-leaderboard/perf-df-pytorch-cuda-gptq-1xA100.csv
published_at
2024-12-13T10:07:03Z
model_revision
main
source_revision
2fa8d8c300c4eb6e744d5e35999db926c0c26e90e767e4c38da326eebe21a92b
metric_population
1
benchmark_evaluation
true
comparison_configuration_sha256
d1334f19a2e8cc6d6739c228bf6cbaf2bc31020df463ee0be55108e5ffbb1c90
complete_configuration_location
source record config and original CSV artifact
source model id
01-ai/Yi-34B
Original result ↗

LLM-Perf

1,994,044.42

source id
llmPerf
leaderboard name
prefill efficiency value
dimension
runtime
metric
prefill efficiency value
rank
1
aggregation scope
status
published
direction
higherIsBetter
conditions
suite
optimum-benchmark/llm-perf-leaderboard/perf-df-gptq-1xA100.csv
execution
config.backend.name
pytorch
config.backend.seed
42
config.backend.task
text-generation
config.backend.device
cuda
config.backend.library
transformers
config.backend.version
2.3.0+cu121
config.environment.cpu
AMD EPYC 7742 64-Core Processor
config.environment.gpu
['NVIDIA A100-SXM4-80GB']
config.backend.eval_mode
True
config.backend.device_ids
0
config.backend.no_weights
True
config.environment.system
Linux
config.backend.torch_dtype
float16
config.environment.machine
x86_64
config.backend.torch_compile
False
config.environment.cpu_count
128
config.environment.gpu_count
1
config.environment.processor
x86_64
config.environment.cpu_ram_mb
540671.627264
config.environment.gpu_vram_mb
85899345920
config.backend.autocast_enabled
False
config.environment.peft_version
0.10.0
config.environment.python_version
3.10.12
config.backend.attn_implementation
sdpa
config.backend.deepspeed_inference
False
config.backend.quantization_scheme
gptq
config.environment.optimum_version
1.19.2
config.backend.to_bettertransformer
False
config.backend.torch_compile_target
forward
config.environment.accelerate_version
0.30.1
config.backend.quantization_config.bits
4
config.environment.transformers_version
4.40.2
config.scenario.input_shapes.batch_size
1
config.backend.hub_kwargs.force_download
False
config.scenario.input_shapes.num_choices
2
config.backend.hub_kwargs.local_files_only
False
config.backend.quantization_config.version
1
config.backend.hub_kwargs.trust_remote_code
True
config.environment.optimum_benchmark_version
0.2.0
config.scenario.input_shapes.sequence_length
256
config.scenario.generate_kwargs.max_new_tokens
64
config.scenario.generate_kwargs.min_new_tokens
64
config.backend.quantization_config.model_seqlen
256
config.backend.quantization_config.use_exllama
True
metric_kind
prefill efficiency value
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
optimum-benchmark/llm-perf-leaderboard/perf-df-gptq-1xA100.csv
published_at
2024-12-13T10:07:03Z
model_revision
main
source_revision
2fa8d8c300c4eb6e744d5e35999db926c0c26e90e767e4c38da326eebe21a92b
metric_population
1
benchmark_evaluation
true
comparison_configuration_sha256
e5a4420a9cb7ff04871954cbbce718935f1308c698283f4fbf22c20d0be1296e
complete_configuration_location
source record config and original CSV artifact
source model id
01-ai/Yi-34B
Original result ↗

LLM-Perf

1,975,211.511

source id
llmPerf
leaderboard name
prefill efficiency value
dimension
runtime
metric
prefill efficiency value
rank
1
aggregation scope
status
published
direction
higherIsBetter
conditions
suite
optimum-benchmark/llm-perf-leaderboard/perf-df-pytorch-cuda-gptq-1xA100.csv
execution
config.backend.name
pytorch
config.backend.seed
42
config.backend.task
text-generation
config.backend.device
cuda
config.backend.library
transformers
config.backend.version
2.3.0+cu121
config.environment.cpu
AMD EPYC 7742 64-Core Processor
config.environment.gpu
['NVIDIA A100-SXM4-80GB']
config.backend.eval_mode
True
config.backend.device_ids
0
config.backend.no_weights
True
config.environment.system
Linux
config.backend.torch_dtype
float16
config.environment.machine
x86_64
config.backend.torch_compile
False
config.environment.cpu_count
128
config.environment.gpu_count
1
config.environment.processor
x86_64
config.environment.cpu_ram_mb
540671.627264
config.environment.gpu_vram_mb
85899345920
config.backend.autocast_enabled
False
config.environment.peft_version
0.10.0
config.environment.python_version
3.10.12
config.backend.attn_implementation
eager
config.backend.deepspeed_inference
False
config.backend.quantization_scheme
gptq
config.environment.optimum_version
1.19.2
config.backend.to_bettertransformer
False
config.backend.torch_compile_target
forward
config.environment.accelerate_version
0.30.1
config.backend.quantization_config.bits
4
config.environment.transformers_version
4.40.2
config.scenario.input_shapes.batch_size
1
config.backend.hub_kwargs.force_download
False
config.scenario.input_shapes.num_choices
2
config.backend.hub_kwargs.local_files_only
False
config.backend.quantization_config.version
2
config.backend.hub_kwargs.trust_remote_code
True
config.environment.optimum_benchmark_version
0.2.0
config.scenario.input_shapes.sequence_length
256
config.scenario.generate_kwargs.max_new_tokens
64
config.scenario.generate_kwargs.min_new_tokens
64
config.backend.quantization_config.model_seqlen
256
config.backend.quantization_config.use_exllama
True
metric_kind
prefill efficiency value
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
optimum-benchmark/llm-perf-leaderboard/perf-df-pytorch-cuda-gptq-1xA100.csv
published_at
2024-12-13T10:07:03Z
model_revision
main
source_revision
2fa8d8c300c4eb6e744d5e35999db926c0c26e90e767e4c38da326eebe21a92b
metric_population
1
benchmark_evaluation
true
comparison_configuration_sha256
12eff847f8ebdc03e2c014f8754f0e9a8e7fb6b431448eada33b87d618876de9
complete_configuration_location
source record config and original CSV artifact
source model id
01-ai/Yi-34B
Original result ↗

LLM-Perf

1,975,211.511

source id
llmPerf
leaderboard name
prefill efficiency value
dimension
runtime
metric
prefill efficiency value
rank
1
aggregation scope
status
published
direction
higherIsBetter
conditions
suite
optimum-benchmark/llm-perf-leaderboard/perf-df-gptq-1xA100.csv
execution
config.backend.name
pytorch
config.backend.seed
42
config.backend.task
text-generation
config.backend.device
cuda
config.backend.library
transformers
config.backend.version
2.3.0+cu121
config.environment.cpu
AMD EPYC 7742 64-Core Processor
config.environment.gpu
['NVIDIA A100-SXM4-80GB']
config.backend.eval_mode
True
config.backend.device_ids
0
config.backend.no_weights
True
config.environment.system
Linux
config.backend.torch_dtype
float16
config.environment.machine
x86_64
config.backend.torch_compile
False
config.environment.cpu_count
128
config.environment.gpu_count
1
config.environment.processor
x86_64
config.environment.cpu_ram_mb
540671.627264
config.environment.gpu_vram_mb
85899345920
config.backend.autocast_enabled
False
config.environment.peft_version
0.10.0
config.environment.python_version
3.10.12
config.backend.attn_implementation
eager
config.backend.deepspeed_inference
False
config.backend.quantization_scheme
gptq
config.environment.optimum_version
1.19.2
config.backend.to_bettertransformer
False
config.backend.torch_compile_target
forward
config.environment.accelerate_version
0.30.1
config.backend.quantization_config.bits
4
config.environment.transformers_version
4.40.2
config.scenario.input_shapes.batch_size
1
config.backend.hub_kwargs.force_download
False
config.scenario.input_shapes.num_choices
2
config.backend.hub_kwargs.local_files_only
False
config.backend.quantization_config.version
2
config.backend.hub_kwargs.trust_remote_code
True
config.environment.optimum_benchmark_version
0.2.0
config.scenario.input_shapes.sequence_length
256
config.scenario.generate_kwargs.max_new_tokens
64
config.scenario.generate_kwargs.min_new_tokens
64
config.backend.quantization_config.model_seqlen
256
config.backend.quantization_config.use_exllama
True
metric_kind
prefill efficiency value
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
optimum-benchmark/llm-perf-leaderboard/perf-df-gptq-1xA100.csv
published_at
2024-12-13T10:07:03Z
model_revision
main
source_revision
2fa8d8c300c4eb6e744d5e35999db926c0c26e90e767e4c38da326eebe21a92b
metric_population
1
benchmark_evaluation
true
comparison_configuration_sha256
e132651cf7ee3cbf4f4ea84dd66035b22de5757212d49e870d974e32b551a37f
complete_configuration_location
source record config and original CSV artifact
source model id
01-ai/Yi-34B
Original result ↗

LLM-Perf

1,966,806.699

source id
llmPerf
leaderboard name
prefill efficiency value
dimension
runtime
metric
prefill efficiency value
rank
1
aggregation scope
status
published
direction
higherIsBetter
conditions
suite
optimum-benchmark/llm-perf-leaderboard/perf-df-pytorch-cuda-gptq-1xA100.csv
execution
config.backend.name
pytorch
config.backend.seed
42
config.backend.task
text-generation
config.backend.device
cuda
config.backend.library
transformers
config.backend.version
2.3.0+cu121
config.environment.cpu
AMD EPYC 7742 64-Core Processor
config.environment.gpu
['NVIDIA A100-SXM4-80GB']
config.backend.eval_mode
True
config.backend.device_ids
0
config.backend.no_weights
True
config.environment.system
Linux
config.backend.torch_dtype
float16
config.environment.machine
x86_64
config.backend.torch_compile
False
config.environment.cpu_count
128
config.environment.gpu_count
1
config.environment.processor
x86_64
config.environment.cpu_ram_mb
540671.627264
config.environment.gpu_vram_mb
85899345920
config.backend.autocast_enabled
False
config.environment.peft_version
0.11.0
config.environment.python_version
3.10.12
config.backend.attn_implementation
eager
config.backend.deepspeed_inference
False
config.backend.quantization_scheme
gptq
config.environment.optimum_version
1.19.2
config.backend.to_bettertransformer
False
config.backend.torch_compile_target
forward
config.environment.accelerate_version
0.30.1
config.backend.quantization_config.bits
4
config.environment.transformers_version
4.40.2
config.scenario.input_shapes.batch_size
1
config.scenario.input_shapes.num_choices
2
config.backend.quantization_config.version
1
config.backend.hub_kwargs.trust_remote_code
True
config.environment.optimum_benchmark_version
0.2.1
config.scenario.input_shapes.sequence_length
256
config.backend.model_kwargs.trust_remote_code
True
config.scenario.generate_kwargs.max_new_tokens
64
config.scenario.generate_kwargs.min_new_tokens
64
config.backend.quantization_config.model_seqlen
256
config.backend.quantization_config.use_exllama
True
config.backend.processor_kwargs.trust_remote_code
True
metric_kind
prefill efficiency value
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
optimum-benchmark/llm-perf-leaderboard/perf-df-pytorch-cuda-gptq-1xA100.csv
published_at
2024-12-13T10:07:03Z
source_revision
2fa8d8c300c4eb6e744d5e35999db926c0c26e90e767e4c38da326eebe21a92b
metric_population
1
benchmark_evaluation
true
comparison_configuration_sha256
7e3cdb02acc41e26c3e5f5daee224ebfb329b99bc0024ca4f42e8f640ad412a9
complete_configuration_location
source record config and original CSV artifact
source model id
01-ai/Yi-34B
Original result ↗

LLM-Perf

1,966,806.699

source id
llmPerf
leaderboard name
prefill efficiency value
dimension
runtime
metric
prefill efficiency value
rank
1
aggregation scope
status
published
direction
higherIsBetter
conditions
suite
optimum-benchmark/llm-perf-leaderboard/perf-df-gptq-1xA100.csv
execution
config.backend.name
pytorch
config.backend.seed
42
config.backend.task
text-generation
config.backend.device
cuda
config.backend.library
transformers
config.backend.version
2.3.0+cu121
config.environment.cpu
AMD EPYC 7742 64-Core Processor
config.environment.gpu
['NVIDIA A100-SXM4-80GB']
config.backend.eval_mode
True
config.backend.device_ids
0
config.backend.no_weights
True
config.environment.system
Linux
config.backend.torch_dtype
float16
config.environment.machine
x86_64
config.backend.torch_compile
False
config.environment.cpu_count
128
config.environment.gpu_count
1
config.environment.processor
x86_64
config.environment.cpu_ram_mb
540671.627264
config.environment.gpu_vram_mb
85899345920
config.backend.autocast_enabled
False
config.environment.peft_version
0.11.0
config.environment.python_version
3.10.12
config.backend.attn_implementation
eager
config.backend.deepspeed_inference
False
config.backend.quantization_scheme
gptq
config.environment.optimum_version
1.19.2
config.backend.to_bettertransformer
False
config.backend.torch_compile_target
forward
config.environment.accelerate_version
0.30.1
config.backend.quantization_config.bits
4
config.environment.transformers_version
4.40.2
config.scenario.input_shapes.batch_size
1
config.scenario.input_shapes.num_choices
2
config.backend.quantization_config.version
1
config.backend.hub_kwargs.trust_remote_code
True
config.environment.optimum_benchmark_version
0.2.1
config.scenario.input_shapes.sequence_length
256
config.backend.model_kwargs.trust_remote_code
True
config.scenario.generate_kwargs.max_new_tokens
64
config.scenario.generate_kwargs.min_new_tokens
64
config.backend.quantization_config.model_seqlen
256
config.backend.quantization_config.use_exllama
True
config.backend.processor_kwargs.trust_remote_code
True
metric_kind
prefill efficiency value
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
optimum-benchmark/llm-perf-leaderboard/perf-df-gptq-1xA100.csv
published_at
2024-12-13T10:07:03Z
source_revision
2fa8d8c300c4eb6e744d5e35999db926c0c26e90e767e4c38da326eebe21a92b
metric_population
1
benchmark_evaluation
true
comparison_configuration_sha256
059d2f584e12ee092f3030f35f8375c19c9216536cc65f260e2ee196125b2df9
complete_configuration_location
source record config and original CSV artifact
source model id
01-ai/Yi-34B
Original result ↗

LLM-Perf

1,965,610.825

source id
llmPerf
leaderboard name
prefill efficiency value
dimension
runtime
metric
prefill efficiency value
rank
1
aggregation scope
status
published
direction
higherIsBetter
conditions
suite
optimum-benchmark/llm-perf-leaderboard/perf-df-gptq-1xA100.csv
execution
config.backend.name
pytorch
config.backend.seed
42
config.backend.task
text-generation
config.backend.device
cuda
config.backend.library
transformers
config.backend.version
2.3.0+cu121
config.environment.cpu
AMD EPYC 7742 64-Core Processor
config.environment.gpu
['NVIDIA A100-SXM4-80GB']
config.backend.eval_mode
True
config.backend.device_ids
0
config.backend.no_weights
True
config.environment.system
Linux
config.backend.torch_dtype
float16
config.environment.machine
x86_64
config.backend.torch_compile
False
config.environment.cpu_count
128
config.environment.gpu_count
1
config.environment.processor
x86_64
config.environment.cpu_ram_mb
540671.627264
config.environment.gpu_vram_mb
85899345920
config.backend.autocast_enabled
False
config.environment.peft_version
0.10.0
config.environment.python_version
3.10.12
config.backend.attn_implementation
sdpa
config.backend.deepspeed_inference
False
config.backend.quantization_scheme
gptq
config.environment.optimum_version
1.19.2
config.backend.to_bettertransformer
False
config.backend.torch_compile_target
forward
config.environment.accelerate_version
0.30.1
config.backend.quantization_config.bits
4
config.environment.transformers_version
4.40.2
config.scenario.input_shapes.batch_size
1
config.backend.hub_kwargs.force_download
False
config.scenario.input_shapes.num_choices
2
config.backend.hub_kwargs.local_files_only
False
config.backend.quantization_config.version
2
config.backend.hub_kwargs.trust_remote_code
True
config.environment.optimum_benchmark_version
0.2.0
config.scenario.input_shapes.sequence_length
256
config.scenario.generate_kwargs.max_new_tokens
64
config.scenario.generate_kwargs.min_new_tokens
64
config.backend.quantization_config.model_seqlen
256
config.backend.quantization_config.use_exllama
True
metric_kind
prefill efficiency value
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
optimum-benchmark/llm-perf-leaderboard/perf-df-gptq-1xA100.csv
published_at
2024-12-13T10:07:03Z
model_revision
main
source_revision
2fa8d8c300c4eb6e744d5e35999db926c0c26e90e767e4c38da326eebe21a92b
metric_population
1
benchmark_evaluation
true
comparison_configuration_sha256
4f32d4b029d55b60c1a85cbe4a9099d889cce2022771559c3021008d932aec30
complete_configuration_location
source record config and original CSV artifact
source model id
01-ai/Yi-34B
Original result ↗

LLM-Perf

1,965,610.825

source id
llmPerf
leaderboard name
prefill efficiency value
dimension
runtime
metric
prefill efficiency value
rank
1
aggregation scope
status
published
direction
higherIsBetter
conditions
suite
optimum-benchmark/llm-perf-leaderboard/perf-df-pytorch-cuda-gptq-1xA100.csv
execution
config.backend.name
pytorch
config.backend.seed
42
config.backend.task
text-generation
config.backend.device
cuda
config.backend.library
transformers
config.backend.version
2.3.0+cu121
config.environment.cpu
AMD EPYC 7742 64-Core Processor
config.environment.gpu
['NVIDIA A100-SXM4-80GB']
config.backend.eval_mode
True
config.backend.device_ids
0
config.backend.no_weights
True
config.environment.system
Linux
config.backend.torch_dtype
float16
config.environment.machine
x86_64
config.backend.torch_compile
False
config.environment.cpu_count
128
config.environment.gpu_count
1
config.environment.processor
x86_64
config.environment.cpu_ram_mb
540671.627264
config.environment.gpu_vram_mb
85899345920
config.backend.autocast_enabled
False
config.environment.peft_version
0.10.0
config.environment.python_version
3.10.12
config.backend.attn_implementation
sdpa
config.backend.deepspeed_inference
False
config.backend.quantization_scheme
gptq
config.environment.optimum_version
1.19.2
config.backend.to_bettertransformer
False
config.backend.torch_compile_target
forward
config.environment.accelerate_version
0.30.1
config.backend.quantization_config.bits
4
config.environment.transformers_version
4.40.2
config.scenario.input_shapes.batch_size
1
config.backend.hub_kwargs.force_download
False
config.scenario.input_shapes.num_choices
2
config.backend.hub_kwargs.local_files_only
False
config.backend.quantization_config.version
2
config.backend.hub_kwargs.trust_remote_code
True
config.environment.optimum_benchmark_version
0.2.0
config.scenario.input_shapes.sequence_length
256
config.scenario.generate_kwargs.max_new_tokens
64
config.scenario.generate_kwargs.min_new_tokens
64
config.backend.quantization_config.model_seqlen
256
config.backend.quantization_config.use_exllama
True
metric_kind
prefill efficiency value
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
optimum-benchmark/llm-perf-leaderboard/perf-df-pytorch-cuda-gptq-1xA100.csv
published_at
2024-12-13T10:07:03Z
model_revision
main
source_revision
2fa8d8c300c4eb6e744d5e35999db926c0c26e90e767e4c38da326eebe21a92b
metric_population
1
benchmark_evaluation
true
comparison_configuration_sha256
5a9608597dac309e5c764b89e2c4a5f1ca26b20588a25311756b47d7148cd835
complete_configuration_location
source record config and original CSV artifact
source model id
01-ai/Yi-34B
Original result ↗

LLM-Perf

1,261,918.213

source id
llmPerf
leaderboard name
prefill efficiency value
dimension
runtime
metric
prefill efficiency value
rank
1
aggregation scope
status
published
direction
higherIsBetter
conditions
suite
optimum-benchmark/llm-perf-leaderboard/perf-df-pytorch-cuda-bnb-1xA100.csv
execution
config.backend.name
pytorch
config.backend.seed
42
config.backend.task
text-generation
config.backend.device
cuda
config.backend.library
transformers
config.backend.version
2.3.0+cu121
config.environment.cpu
AMD EPYC 7742 64-Core Processor
config.environment.gpu
['NVIDIA A100-SXM4-80GB']
config.backend.eval_mode
True
config.backend.device_ids
0
config.backend.no_weights
True
config.environment.system
Linux
config.backend.torch_dtype
float16
config.environment.machine
x86_64
config.backend.torch_compile
False
config.environment.cpu_count
128
config.environment.gpu_count
1
config.environment.processor
x86_64
config.environment.cpu_ram_mb
540671.627264
config.environment.gpu_vram_mb
85899345920
config.backend.autocast_enabled
False
config.environment.peft_version
0.10.0
config.environment.python_version
3.10.12
config.backend.attn_implementation
flash_attention_2
config.backend.deepspeed_inference
False
config.backend.quantization_scheme
bnb
config.environment.optimum_version
1.19.2
config.backend.to_bettertransformer
False
config.backend.torch_compile_target
forward
config.environment.accelerate_version
0.30.1
config.environment.transformers_version
4.40.2
config.scenario.input_shapes.batch_size
1
config.backend.hub_kwargs.force_download
False
config.scenario.input_shapes.num_choices
2
config.backend.hub_kwargs.local_files_only
False
config.backend.hub_kwargs.trust_remote_code
True
config.environment.optimum_benchmark_version
0.2.0
config.scenario.input_shapes.sequence_length
256
config.scenario.generate_kwargs.max_new_tokens
64
config.scenario.generate_kwargs.min_new_tokens
64
config.backend.quantization_config.load_in_4bit
True
config.backend.quantization_config.llm_int8_threshold
0.0
metric_kind
prefill efficiency value
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
optimum-benchmark/llm-perf-leaderboard/perf-df-pytorch-cuda-bnb-1xA100.csv
published_at
2024-12-13T10:07:03Z
model_revision
main
source_revision
2fa8d8c300c4eb6e744d5e35999db926c0c26e90e767e4c38da326eebe21a92b
metric_population
1
benchmark_evaluation
true
comparison_configuration_sha256
28aaeb5136de0857454d29743a750c6d633cab655a6dfe37474c6ec505fbe93b
complete_configuration_location
source record config and original CSV artifact
source model id
01-ai/Yi-34B
Original result ↗

LLM-Perf

1,261,918.213

source id
llmPerf
leaderboard name
prefill efficiency value
dimension
runtime
metric
prefill efficiency value
rank
1
aggregation scope
status
published
direction
higherIsBetter
conditions
suite
optimum-benchmark/llm-perf-leaderboard/perf-df-bnb-1xA100.csv
execution
config.backend.name
pytorch
config.backend.seed
42
config.backend.task
text-generation
config.backend.device
cuda
config.backend.library
transformers
config.backend.version
2.3.0+cu121
config.environment.cpu
AMD EPYC 7742 64-Core Processor
config.environment.gpu
['NVIDIA A100-SXM4-80GB']
config.backend.eval_mode
True
config.backend.device_ids
0
config.backend.no_weights
True
config.environment.system
Linux
config.backend.torch_dtype
float16
config.environment.machine
x86_64
config.backend.torch_compile
False
config.environment.cpu_count
128
config.environment.gpu_count
1
config.environment.processor
x86_64
config.environment.cpu_ram_mb
540671.627264
config.environment.gpu_vram_mb
85899345920
config.backend.autocast_enabled
False
config.environment.peft_version
0.10.0
config.environment.python_version
3.10.12
config.backend.attn_implementation
flash_attention_2
config.backend.deepspeed_inference
False
config.backend.quantization_scheme
bnb
config.environment.optimum_version
1.19.2
config.backend.to_bettertransformer
False
config.backend.torch_compile_target
forward
config.environment.accelerate_version
0.30.1
config.environment.transformers_version
4.40.2
config.scenario.input_shapes.batch_size
1
config.backend.hub_kwargs.force_download
False
config.scenario.input_shapes.num_choices
2
config.backend.hub_kwargs.local_files_only
False
config.backend.hub_kwargs.trust_remote_code
True
config.environment.optimum_benchmark_version
0.2.0
config.scenario.input_shapes.sequence_length
256
config.scenario.generate_kwargs.max_new_tokens
64
config.scenario.generate_kwargs.min_new_tokens
64
config.backend.quantization_config.load_in_4bit
True
config.backend.quantization_config.llm_int8_threshold
0.0
metric_kind
prefill efficiency value
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
optimum-benchmark/llm-perf-leaderboard/perf-df-bnb-1xA100.csv
published_at
2024-12-13T10:07:03Z
model_revision
main
source_revision
2fa8d8c300c4eb6e744d5e35999db926c0c26e90e767e4c38da326eebe21a92b
metric_population
1
benchmark_evaluation
true
comparison_configuration_sha256
bc35ff745249f78d01cfa8d08ea2fbb7b9d3c59553abe1f941c48a34847571ad
complete_configuration_location
source record config and original CSV artifact
source model id
01-ai/Yi-34B
Original result ↗

Provider specifications & access

1–1 / 1

01-ai/Yi-34B ↗

source model id
HuggingFace/01-ai-yi-34b@1
hosting
foundry-managed-compute
platforms
  • Microsoft Foundry
license
apache-2.0
architecture
input_modalities
  • text
output_modalities
  • text
capabilities

    Original sources

    1–8 / 8

    Hugging Face · 01-ai/Yi-34B ↗

    Microsoft Foundry · Model page ↗

    Microsoft Foundry · Model documentation ↗

    Hugging Face · Catalog ↗

    Hugging Face · Catalog ↗

    Hugging Face · Catalog ↗

    Hugging Face · Catalog ↗

    Microsoft Foundry · Catalog ↗

    Source links

    1–5 / 5

    Explore more models

    Models

    Loading
    EVIDENCE / COMPARISON

    Benchmarks

    Explore every benchmark

    Loading

    These charts contain model-linked results. The charts above also include unlinked source profiles. Compare scores within each evaluation. Model quality, usage, transparency and system performance measure different things.

    Install TensorTrace