01-ai/Yi-34B

`01-ai/Yi-34B` powered by Text Generation Inference (TGI) - [Example Notebook](https://aka.ms/hugging-face-text-generation-streaming-online-endpoint) - [Original Model Card](https://huggingface.co/01…

azureFoundry/HuggingFace/01-ai-yi-34b@1

01-ai/Yi-34B powered by Text Generation Inference (TGI)

Send Request

You can use cURL or any REST Client to send a request to the AzureML endpoint with your AzureML token.

curl <AZUREML_ENDPOINT_URL> \
    -X POST \
    -d '{"inputs":"What is Deep Learning?"}' \
    -H "Authorization: Bearer <AZUREML_TOKEN>" \
    -H "Content-Type: application/json"

Supported Parameters

  • inputs (string): Input prompt.
  • parameters (object):
    • best_of (integer): Generate best_of sequences and return the one if the highest token logprobs.
    • decoder_input_details (boolean): Whether to return decoder input token logprobs and ids.
    • details (boolean): Whether to return generation details.
    • do_sample (boolean): Activate logits sampling.
    • frequency_penalty (float): The parameter for frequency penalty. 1.0 means no penalty Penalize new tokens based on their existing frequency in the text so far, decreasing the model’s likelihood to repeat the same line verbatim.
    • grammar (object): One of the following
      • #1 (object):
        • type (enum): Possible values: json.
        • value (string): A string that represents a JSON Schema. JSON Schema is a declarative language that allows to annotate JSON documents with types and descriptions.
      • #2 (object):
        • type (enum): Possible values: regex.
        • value (string): The regular expression.
      • #3 (object):
        • type (enum): Possible values: json_schema.
        • value (object):
          • name (string): Optional name identifier for the schema
          • schema (object): The actual JSON schema definition
    • max_new_tokens (integer): Maximum number of tokens to generate.
    • repetition_penalty (float): The parameter for repetition penalty. 1.0 means no penalty. See this paper for more details.
    • return_full_text (boolean): Whether to prepend the prompt to the generated text
    • seed (integer): Random sampling seed.
    • stop (string[]): Stop generating tokens if a member of stop is generated.
    • temperature (float): The value used to module the logits distribution.
    • top_k (integer): The number of highest probability vocabulary tokens to keep for top-k-filtering.
    • top_n_tokens (integer): The number of highest probability vocabulary tokens to keep for top-n-filtering.
    • top_p (float): Top-p value for nucleus sampling.
    • truncate (integer): Truncate inputs tokens to the given size.
    • typical_p (float): Typical Decoding mass See Typical Decoding for Natural Language Generation for more information.
    • watermark (boolean): Watermarking with A Watermark for Large Language Models.
    • stream (boolean): Whether to stream the output tokens or not. Defaults to false.

Example payload

{
  "inputs": "What is Deep Learning?",
  "parameters": {
    "do_sample": true,
    "top_p": 0.95,
    "temperature": 0.2,
    "top_k": 50,
    "max_new_tokens": 256,
    "repetition_penalty": 1.03,
    "stop": ["\nUser:", "<|endoftext|>", "</s>"]
  }
}

OpenAI Chat Completion API compatibility

Additionally, Text Generation Inference (TGI) offers an OpenAI Chat Completion API compatible layer under the endpoint /v1/chat/completions, check the full specification in the OpenAI Chat Completion Create Documentation.

Specifications & capabilities

Input
text
Output
text
Published model ID
Yi-34B

Evaluations

21–40 / 5,420

LLM-Perf

1,255,728.58

source id
llmPerf
leaderboard name
prefill efficiency value
dimension
runtime
metric
prefill efficiency value
rank
1
aggregation scope
status
published
direction
higherIsBetter
conditions
suite
optimum-benchmark/llm-perf-leaderboard/perf-df-bnb-1xA100.csv
execution
config.backend.name
pytorch
config.backend.seed
42
config.backend.task
text-generation
config.backend.device
cuda
config.backend.library
transformers
config.backend.version
2.3.0+cu121
config.environment.cpu
AMD EPYC 7742 64-Core Processor
config.environment.gpu
['NVIDIA A100-SXM4-80GB']
config.backend.eval_mode
True
config.backend.device_ids
0
config.backend.no_weights
True
config.environment.system
Linux
config.backend.torch_dtype
float16
config.environment.machine
x86_64
config.backend.torch_compile
False
config.environment.cpu_count
128
config.environment.gpu_count
1
config.environment.processor
x86_64
config.environment.cpu_ram_mb
540671.627264
config.environment.gpu_vram_mb
85899345920
config.backend.autocast_enabled
False
config.environment.peft_version
0.11.0
config.environment.python_version
3.10.12
config.backend.attn_implementation
eager
config.backend.deepspeed_inference
False
config.backend.quantization_scheme
bnb
config.environment.optimum_version
1.19.2
config.backend.to_bettertransformer
False
config.backend.torch_compile_target
forward
config.environment.accelerate_version
0.30.1
config.environment.transformers_version
4.40.2
config.scenario.input_shapes.batch_size
1
config.scenario.input_shapes.num_choices
2
config.backend.hub_kwargs.trust_remote_code
True
config.environment.optimum_benchmark_version
0.2.1
config.scenario.input_shapes.sequence_length
256
config.backend.model_kwargs.trust_remote_code
True
config.scenario.generate_kwargs.max_new_tokens
64
config.scenario.generate_kwargs.min_new_tokens
64
config.backend.quantization_config.load_in_4bit
True
config.backend.processor_kwargs.trust_remote_code
True
config.backend.quantization_config.llm_int8_threshold
0.0
metric_kind
prefill efficiency value
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
optimum-benchmark/llm-perf-leaderboard/perf-df-bnb-1xA100.csv
published_at
2024-12-13T10:07:03Z
source_revision
2fa8d8c300c4eb6e744d5e35999db926c0c26e90e767e4c38da326eebe21a92b
metric_population
1
benchmark_evaluation
true
comparison_configuration_sha256
0bdf3a832b838b8251f578962ba7bf694d7a23a4a6b8c09c493bf0dd953a687d
complete_configuration_location
source record config and original CSV artifact
source model id
01-ai/Yi-34B
Original result ↗

LLM-Perf

1,255,728.58

source id
llmPerf
leaderboard name
prefill efficiency value
dimension
runtime
metric
prefill efficiency value
rank
1
aggregation scope
status
published
direction
higherIsBetter
conditions
suite
optimum-benchmark/llm-perf-leaderboard/perf-df-pytorch-cuda-bnb-1xA100.csv
execution
config.backend.name
pytorch
config.backend.seed
42
config.backend.task
text-generation
config.backend.device
cuda
config.backend.library
transformers
config.backend.version
2.3.0+cu121
config.environment.cpu
AMD EPYC 7742 64-Core Processor
config.environment.gpu
['NVIDIA A100-SXM4-80GB']
config.backend.eval_mode
True
config.backend.device_ids
0
config.backend.no_weights
True
config.environment.system
Linux
config.backend.torch_dtype
float16
config.environment.machine
x86_64
config.backend.torch_compile
False
config.environment.cpu_count
128
config.environment.gpu_count
1
config.environment.processor
x86_64
config.environment.cpu_ram_mb
540671.627264
config.environment.gpu_vram_mb
85899345920
config.backend.autocast_enabled
False
config.environment.peft_version
0.11.0
config.environment.python_version
3.10.12
config.backend.attn_implementation
eager
config.backend.deepspeed_inference
False
config.backend.quantization_scheme
bnb
config.environment.optimum_version
1.19.2
config.backend.to_bettertransformer
False
config.backend.torch_compile_target
forward
config.environment.accelerate_version
0.30.1
config.environment.transformers_version
4.40.2
config.scenario.input_shapes.batch_size
1
config.scenario.input_shapes.num_choices
2
config.backend.hub_kwargs.trust_remote_code
True
config.environment.optimum_benchmark_version
0.2.1
config.scenario.input_shapes.sequence_length
256
config.backend.model_kwargs.trust_remote_code
True
config.scenario.generate_kwargs.max_new_tokens
64
config.scenario.generate_kwargs.min_new_tokens
64
config.backend.quantization_config.load_in_4bit
True
config.backend.processor_kwargs.trust_remote_code
True
config.backend.quantization_config.llm_int8_threshold
0.0
metric_kind
prefill efficiency value
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
optimum-benchmark/llm-perf-leaderboard/perf-df-pytorch-cuda-bnb-1xA100.csv
published_at
2024-12-13T10:07:03Z
source_revision
2fa8d8c300c4eb6e744d5e35999db926c0c26e90e767e4c38da326eebe21a92b
metric_population
1
benchmark_evaluation
true
comparison_configuration_sha256
77357ed1fc3d0cc02ac8f2d82794d5fdcade738cbabb8a405d6d543d950e27b7
complete_configuration_location
source record config and original CSV artifact
source model id
01-ai/Yi-34B
Original result ↗

LLM-Perf

1,248,022.137

source id
llmPerf
leaderboard name
prefill efficiency value
dimension
runtime
metric
prefill efficiency value
rank
1
aggregation scope
status
published
direction
higherIsBetter
conditions
suite
optimum-benchmark/llm-perf-leaderboard/perf-df-pytorch-cuda-bnb-1xA100.csv
execution
config.backend.name
pytorch
config.backend.seed
42
config.backend.task
text-generation
config.backend.device
cuda
config.backend.library
transformers
config.backend.version
2.3.0+cu121
config.environment.cpu
AMD EPYC 7742 64-Core Processor
config.environment.gpu
['NVIDIA A100-SXM4-80GB']
config.backend.eval_mode
True
config.backend.device_ids
0
config.backend.no_weights
True
config.environment.system
Linux
config.backend.torch_dtype
float16
config.environment.machine
x86_64
config.backend.torch_compile
False
config.environment.cpu_count
128
config.environment.gpu_count
1
config.environment.processor
x86_64
config.environment.cpu_ram_mb
540671.627264
config.environment.gpu_vram_mb
85899345920
config.backend.autocast_enabled
False
config.environment.peft_version
0.10.0
config.environment.python_version
3.10.12
config.backend.attn_implementation
sdpa
config.backend.deepspeed_inference
False
config.backend.quantization_scheme
bnb
config.environment.optimum_version
1.19.2
config.backend.to_bettertransformer
False
config.backend.torch_compile_target
forward
config.environment.accelerate_version
0.30.1
config.environment.transformers_version
4.40.2
config.scenario.input_shapes.batch_size
1
config.backend.hub_kwargs.force_download
False
config.scenario.input_shapes.num_choices
2
config.backend.hub_kwargs.local_files_only
False
config.backend.hub_kwargs.trust_remote_code
True
config.environment.optimum_benchmark_version
0.2.0
config.scenario.input_shapes.sequence_length
256
config.scenario.generate_kwargs.max_new_tokens
64
config.scenario.generate_kwargs.min_new_tokens
64
config.backend.quantization_config.load_in_4bit
True
config.backend.quantization_config.llm_int8_threshold
0.0
metric_kind
prefill efficiency value
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
optimum-benchmark/llm-perf-leaderboard/perf-df-pytorch-cuda-bnb-1xA100.csv
published_at
2024-12-13T10:07:03Z
model_revision
main
source_revision
2fa8d8c300c4eb6e744d5e35999db926c0c26e90e767e4c38da326eebe21a92b
metric_population
1
benchmark_evaluation
true
comparison_configuration_sha256
4050e13d24ac300fddccaac29a59d5142e0d3b1cecac94dd2f668a4e80913eae
complete_configuration_location
source record config and original CSV artifact
source model id
01-ai/Yi-34B
Original result ↗

LLM-Perf

1,248,022.137

source id
llmPerf
leaderboard name
prefill efficiency value
dimension
runtime
metric
prefill efficiency value
rank
1
aggregation scope
status
published
direction
higherIsBetter
conditions
suite
optimum-benchmark/llm-perf-leaderboard/perf-df-bnb-1xA100.csv
execution
config.backend.name
pytorch
config.backend.seed
42
config.backend.task
text-generation
config.backend.device
cuda
config.backend.library
transformers
config.backend.version
2.3.0+cu121
config.environment.cpu
AMD EPYC 7742 64-Core Processor
config.environment.gpu
['NVIDIA A100-SXM4-80GB']
config.backend.eval_mode
True
config.backend.device_ids
0
config.backend.no_weights
True
config.environment.system
Linux
config.backend.torch_dtype
float16
config.environment.machine
x86_64
config.backend.torch_compile
False
config.environment.cpu_count
128
config.environment.gpu_count
1
config.environment.processor
x86_64
config.environment.cpu_ram_mb
540671.627264
config.environment.gpu_vram_mb
85899345920
config.backend.autocast_enabled
False
config.environment.peft_version
0.10.0
config.environment.python_version
3.10.12
config.backend.attn_implementation
sdpa
config.backend.deepspeed_inference
False
config.backend.quantization_scheme
bnb
config.environment.optimum_version
1.19.2
config.backend.to_bettertransformer
False
config.backend.torch_compile_target
forward
config.environment.accelerate_version
0.30.1
config.environment.transformers_version
4.40.2
config.scenario.input_shapes.batch_size
1
config.backend.hub_kwargs.force_download
False
config.scenario.input_shapes.num_choices
2
config.backend.hub_kwargs.local_files_only
False
config.backend.hub_kwargs.trust_remote_code
True
config.environment.optimum_benchmark_version
0.2.0
config.scenario.input_shapes.sequence_length
256
config.scenario.generate_kwargs.max_new_tokens
64
config.scenario.generate_kwargs.min_new_tokens
64
config.backend.quantization_config.load_in_4bit
True
config.backend.quantization_config.llm_int8_threshold
0.0
metric_kind
prefill efficiency value
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
optimum-benchmark/llm-perf-leaderboard/perf-df-bnb-1xA100.csv
published_at
2024-12-13T10:07:03Z
model_revision
main
source_revision
2fa8d8c300c4eb6e744d5e35999db926c0c26e90e767e4c38da326eebe21a92b
metric_population
1
benchmark_evaluation
true
comparison_configuration_sha256
5f6a20e216f1fa020e1e0ea97565429a905b43f0ab260586bf95b55ea085d606
complete_configuration_location
source record config and original CSV artifact
source model id
01-ai/Yi-34B
Original result ↗

LLM-Perf

108,446.829

source id
llmPerf
leaderboard name
decode efficiency value
dimension
runtime
metric
decode efficiency value
rank
1
aggregation scope
status
published
direction
higherIsBetter
conditions
suite
optimum-benchmark/llm-perf-leaderboard/perf-df-pytorch-cuda-unquantized-1xA100.csv
execution
config.backend.name
pytorch
config.backend.seed
42
config.backend.task
text-generation
config.backend.device
cuda
config.backend.library
transformers
config.backend.version
2.3.0+cu121
config.environment.cpu
AMD EPYC 7742 64-Core Processor
config.environment.gpu
['NVIDIA A100-SXM4-80GB']
config.backend.eval_mode
True
config.backend.device_ids
0
config.backend.no_weights
True
config.environment.system
Linux
config.backend.torch_dtype
float16
config.environment.machine
x86_64
config.backend.torch_compile
False
config.environment.cpu_count
128
config.environment.gpu_count
1
config.environment.processor
x86_64
config.environment.cpu_ram_mb
540671.627264
config.environment.gpu_vram_mb
85899345920
config.backend.autocast_enabled
False
config.environment.peft_version
0.11.0
config.environment.python_version
3.10.12
config.backend.attn_implementation
sdpa
config.backend.deepspeed_inference
False
config.environment.optimum_version
1.19.2
config.backend.to_bettertransformer
False
config.backend.torch_compile_target
forward
config.environment.accelerate_version
0.30.1
config.environment.transformers_version
4.40.2
config.scenario.input_shapes.batch_size
1
config.scenario.input_shapes.num_choices
2
config.backend.hub_kwargs.trust_remote_code
True
config.environment.optimum_benchmark_version
0.2.1
config.scenario.input_shapes.sequence_length
256
config.backend.model_kwargs.trust_remote_code
True
config.scenario.generate_kwargs.max_new_tokens
64
config.scenario.generate_kwargs.min_new_tokens
64
config.backend.processor_kwargs.trust_remote_code
True
metric_kind
decode efficiency value
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
optimum-benchmark/llm-perf-leaderboard/perf-df-pytorch-cuda-unquantized-1xA100.csv
published_at
2024-12-13T10:07:03Z
source_revision
2fa8d8c300c4eb6e744d5e35999db926c0c26e90e767e4c38da326eebe21a92b
metric_population
1
benchmark_evaluation
true
comparison_configuration_sha256
e98269fb859d479cf594053da82963997bf4c42df3b1a0f461c21547439bdc33
complete_configuration_location
source record config and original CSV artifact
source model id
01-ai/Yi-34B
Original result ↗

LLM-Perf

108,446.829

source id
llmPerf
leaderboard name
decode efficiency value
dimension
runtime
metric
decode efficiency value
rank
1
aggregation scope
status
published
direction
higherIsBetter
conditions
suite
optimum-benchmark/llm-perf-leaderboard/perf-df-unquantized-1xA100.csv
execution
config.backend.name
pytorch
config.backend.seed
42
config.backend.task
text-generation
config.backend.device
cuda
config.backend.library
transformers
config.backend.version
2.3.0+cu121
config.environment.cpu
AMD EPYC 7742 64-Core Processor
config.environment.gpu
['NVIDIA A100-SXM4-80GB']
config.backend.eval_mode
True
config.backend.device_ids
0
config.backend.no_weights
True
config.environment.system
Linux
config.backend.torch_dtype
float16
config.environment.machine
x86_64
config.backend.torch_compile
False
config.environment.cpu_count
128
config.environment.gpu_count
1
config.environment.processor
x86_64
config.environment.cpu_ram_mb
540671.627264
config.environment.gpu_vram_mb
85899345920
config.backend.autocast_enabled
False
config.environment.peft_version
0.11.0
config.environment.python_version
3.10.12
config.backend.attn_implementation
sdpa
config.backend.deepspeed_inference
False
config.environment.optimum_version
1.19.2
config.backend.to_bettertransformer
False
config.backend.torch_compile_target
forward
config.environment.accelerate_version
0.30.1
config.environment.transformers_version
4.40.2
config.scenario.input_shapes.batch_size
1
config.scenario.input_shapes.num_choices
2
config.backend.hub_kwargs.trust_remote_code
True
config.environment.optimum_benchmark_version
0.2.1
config.scenario.input_shapes.sequence_length
256
config.backend.model_kwargs.trust_remote_code
True
config.scenario.generate_kwargs.max_new_tokens
64
config.scenario.generate_kwargs.min_new_tokens
64
config.backend.processor_kwargs.trust_remote_code
True
metric_kind
decode efficiency value
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
optimum-benchmark/llm-perf-leaderboard/perf-df-unquantized-1xA100.csv
published_at
2024-12-13T10:07:03Z
source_revision
2fa8d8c300c4eb6e744d5e35999db926c0c26e90e767e4c38da326eebe21a92b
metric_population
1
benchmark_evaluation
true
comparison_configuration_sha256
4d1ad767823f5984c1333a2c80a1b8794a0ef5ac80d44cde71dc4faf09969111
complete_configuration_location
source record config and original CSV artifact
source model id
01-ai/Yi-34B
Original result ↗

LLM-Perf

98,579.367

source id
llmPerf
leaderboard name
decode efficiency value
dimension
runtime
metric
decode efficiency value
rank
1
aggregation scope
status
published
direction
higherIsBetter
conditions
suite
optimum-benchmark/llm-perf-leaderboard/perf-df-unquantized-1xA100.csv
execution
config.backend.name
pytorch
config.backend.seed
42
config.backend.task
text-generation
config.backend.device
cuda
config.backend.library
transformers
config.backend.version
2.3.0+cu121
config.environment.cpu
AMD EPYC 7742 64-Core Processor
config.environment.gpu
['NVIDIA A100-SXM4-80GB']
config.backend.eval_mode
True
config.backend.device_ids
0
config.backend.no_weights
True
config.environment.system
Linux
config.backend.torch_dtype
bfloat16
config.environment.machine
x86_64
config.backend.torch_compile
False
config.environment.cpu_count
128
config.environment.gpu_count
1
config.environment.processor
x86_64
config.environment.cpu_ram_mb
540671.627264
config.environment.gpu_vram_mb
85899345920
config.backend.autocast_enabled
False
config.environment.peft_version
0.11.0
config.environment.python_version
3.10.12
config.backend.attn_implementation
eager
config.backend.deepspeed_inference
False
config.environment.optimum_version
1.19.2
config.backend.to_bettertransformer
False
config.backend.torch_compile_target
forward
config.environment.accelerate_version
0.30.1
config.environment.transformers_version
4.40.2
config.scenario.input_shapes.batch_size
1
config.scenario.input_shapes.num_choices
2
config.backend.hub_kwargs.trust_remote_code
True
config.environment.optimum_benchmark_version
0.2.1
config.scenario.input_shapes.sequence_length
256
config.backend.model_kwargs.trust_remote_code
True
config.scenario.generate_kwargs.max_new_tokens
64
config.scenario.generate_kwargs.min_new_tokens
64
config.backend.processor_kwargs.trust_remote_code
True
metric_kind
decode efficiency value
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
optimum-benchmark/llm-perf-leaderboard/perf-df-unquantized-1xA100.csv
published_at
2024-12-13T10:07:03Z
source_revision
2fa8d8c300c4eb6e744d5e35999db926c0c26e90e767e4c38da326eebe21a92b
metric_population
1
benchmark_evaluation
true
comparison_configuration_sha256
468e8460883515023a1c9c7cf5a09717acb57fa4558b60901a8ccbf9d5f72121
complete_configuration_location
source record config and original CSV artifact
source model id
01-ai/Yi-34B
Original result ↗

LLM-Perf

98,579.367

source id
llmPerf
leaderboard name
decode efficiency value
dimension
runtime
metric
decode efficiency value
rank
1
aggregation scope
status
published
direction
higherIsBetter
conditions
suite
optimum-benchmark/llm-perf-leaderboard/perf-df-pytorch-cuda-unquantized-1xA100.csv
execution
config.backend.name
pytorch
config.backend.seed
42
config.backend.task
text-generation
config.backend.device
cuda
config.backend.library
transformers
config.backend.version
2.3.0+cu121
config.environment.cpu
AMD EPYC 7742 64-Core Processor
config.environment.gpu
['NVIDIA A100-SXM4-80GB']
config.backend.eval_mode
True
config.backend.device_ids
0
config.backend.no_weights
True
config.environment.system
Linux
config.backend.torch_dtype
bfloat16
config.environment.machine
x86_64
config.backend.torch_compile
False
config.environment.cpu_count
128
config.environment.gpu_count
1
config.environment.processor
x86_64
config.environment.cpu_ram_mb
540671.627264
config.environment.gpu_vram_mb
85899345920
config.backend.autocast_enabled
False
config.environment.peft_version
0.11.0
config.environment.python_version
3.10.12
config.backend.attn_implementation
eager
config.backend.deepspeed_inference
False
config.environment.optimum_version
1.19.2
config.backend.to_bettertransformer
False
config.backend.torch_compile_target
forward
config.environment.accelerate_version
0.30.1
config.environment.transformers_version
4.40.2
config.scenario.input_shapes.batch_size
1
config.scenario.input_shapes.num_choices
2
config.backend.hub_kwargs.trust_remote_code
True
config.environment.optimum_benchmark_version
0.2.1
config.scenario.input_shapes.sequence_length
256
config.backend.model_kwargs.trust_remote_code
True
config.scenario.generate_kwargs.max_new_tokens
64
config.scenario.generate_kwargs.min_new_tokens
64
config.backend.processor_kwargs.trust_remote_code
True
metric_kind
decode efficiency value
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
optimum-benchmark/llm-perf-leaderboard/perf-df-pytorch-cuda-unquantized-1xA100.csv
published_at
2024-12-13T10:07:03Z
source_revision
2fa8d8c300c4eb6e744d5e35999db926c0c26e90e767e4c38da326eebe21a92b
metric_population
1
benchmark_evaluation
true
comparison_configuration_sha256
5a7bfe2f08af6456f222f387892347ad2dc4618106b698f9ade8192bb1e13c85
complete_configuration_location
source record config and original CSV artifact
source model id
01-ai/Yi-34B
Original result ↗

LLM-Perf

70,718.063

source id
llmPerf
leaderboard name
decode memory max_global_vram
dimension
runtime
metric
decode memory max_global_vram
rank
1
aggregation scope
status
published
direction
lowerIsBetter
conditions
suite
optimum-benchmark/llm-perf-leaderboard/perf-df-unquantized-1xA100.csv
execution
config.backend.name
pytorch
config.backend.seed
42
config.backend.task
text-generation
config.backend.device
cuda
config.backend.library
transformers
config.backend.version
2.3.0+cu121
config.environment.cpu
AMD EPYC 7742 64-Core Processor
config.environment.gpu
['NVIDIA A100-SXM4-80GB']
config.backend.eval_mode
True
config.backend.device_ids
0
config.backend.no_weights
True
config.environment.system
Linux
config.backend.torch_dtype
float16
config.environment.machine
x86_64
config.backend.torch_compile
False
config.environment.cpu_count
128
config.environment.gpu_count
1
config.environment.processor
x86_64
config.environment.cpu_ram_mb
540671.627264
config.environment.gpu_vram_mb
85899345920
config.backend.autocast_enabled
False
config.environment.peft_version
0.11.0
config.environment.python_version
3.10.12
config.backend.attn_implementation
sdpa
config.backend.deepspeed_inference
False
config.environment.optimum_version
1.19.2
config.backend.to_bettertransformer
False
config.backend.torch_compile_target
forward
config.environment.accelerate_version
0.30.1
config.environment.transformers_version
4.40.2
config.scenario.input_shapes.batch_size
1
config.scenario.input_shapes.num_choices
2
config.backend.hub_kwargs.trust_remote_code
True
config.environment.optimum_benchmark_version
0.2.1
config.scenario.input_shapes.sequence_length
256
config.backend.model_kwargs.trust_remote_code
True
config.scenario.generate_kwargs.max_new_tokens
64
config.scenario.generate_kwargs.min_new_tokens
64
config.backend.processor_kwargs.trust_remote_code
True
metric_kind
decode memory max_global_vram
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
optimum-benchmark/llm-perf-leaderboard/perf-df-unquantized-1xA100.csv
published_at
2024-12-13T10:07:03Z
source_revision
2fa8d8c300c4eb6e744d5e35999db926c0c26e90e767e4c38da326eebe21a92b
metric_population
1
benchmark_evaluation
true
comparison_configuration_sha256
4d1ad767823f5984c1333a2c80a1b8794a0ef5ac80d44cde71dc4faf09969111
complete_configuration_location
source record config and original CSV artifact
source model id
01-ai/Yi-34B
Original result ↗

LLM-Perf

70,718.063

source id
llmPerf
leaderboard name
prefill memory max_global_vram
dimension
runtime
metric
prefill memory max_global_vram
rank
1
aggregation scope
status
published
direction
lowerIsBetter
conditions
suite
optimum-benchmark/llm-perf-leaderboard/perf-df-unquantized-1xA100.csv
execution
config.backend.name
pytorch
config.backend.seed
42
config.backend.task
text-generation
config.backend.device
cuda
config.backend.library
transformers
config.backend.version
2.3.0+cu121
config.environment.cpu
AMD EPYC 7742 64-Core Processor
config.environment.gpu
['NVIDIA A100-SXM4-80GB']
config.backend.eval_mode
True
config.backend.device_ids
0
config.backend.no_weights
True
config.environment.system
Linux
config.backend.torch_dtype
float16
config.environment.machine
x86_64
config.backend.torch_compile
False
config.environment.cpu_count
128
config.environment.gpu_count
1
config.environment.processor
x86_64
config.environment.cpu_ram_mb
540671.627264
config.environment.gpu_vram_mb
85899345920
config.backend.autocast_enabled
False
config.environment.peft_version
0.11.0
config.environment.python_version
3.10.12
config.backend.attn_implementation
sdpa
config.backend.deepspeed_inference
False
config.environment.optimum_version
1.19.2
config.backend.to_bettertransformer
False
config.backend.torch_compile_target
forward
config.environment.accelerate_version
0.30.1
config.environment.transformers_version
4.40.2
config.scenario.input_shapes.batch_size
1
config.scenario.input_shapes.num_choices
2
config.backend.hub_kwargs.trust_remote_code
True
config.environment.optimum_benchmark_version
0.2.1
config.scenario.input_shapes.sequence_length
256
config.backend.model_kwargs.trust_remote_code
True
config.scenario.generate_kwargs.max_new_tokens
64
config.scenario.generate_kwargs.min_new_tokens
64
config.backend.processor_kwargs.trust_remote_code
True
metric_kind
prefill memory max_global_vram
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
optimum-benchmark/llm-perf-leaderboard/perf-df-unquantized-1xA100.csv
published_at
2024-12-13T10:07:03Z
source_revision
2fa8d8c300c4eb6e744d5e35999db926c0c26e90e767e4c38da326eebe21a92b
metric_population
1
benchmark_evaluation
true
comparison_configuration_sha256
4d1ad767823f5984c1333a2c80a1b8794a0ef5ac80d44cde71dc4faf09969111
complete_configuration_location
source record config and original CSV artifact
source model id
01-ai/Yi-34B
Original result ↗

LLM-Perf

70,718.063

source id
llmPerf
leaderboard name
prefill memory max_global_vram
dimension
runtime
metric
prefill memory max_global_vram
rank
1
aggregation scope
status
published
direction
lowerIsBetter
conditions
suite
optimum-benchmark/llm-perf-leaderboard/perf-df-pytorch-cuda-unquantized-1xA100.csv
execution
config.backend.name
pytorch
config.backend.seed
42
config.backend.task
text-generation
config.backend.device
cuda
config.backend.library
transformers
config.backend.version
2.3.0+cu121
config.environment.cpu
AMD EPYC 7742 64-Core Processor
config.environment.gpu
['NVIDIA A100-SXM4-80GB']
config.backend.eval_mode
True
config.backend.device_ids
0
config.backend.no_weights
True
config.environment.system
Linux
config.backend.torch_dtype
float16
config.environment.machine
x86_64
config.backend.torch_compile
False
config.environment.cpu_count
128
config.environment.gpu_count
1
config.environment.processor
x86_64
config.environment.cpu_ram_mb
540671.627264
config.environment.gpu_vram_mb
85899345920
config.backend.autocast_enabled
False
config.environment.peft_version
0.11.0
config.environment.python_version
3.10.12
config.backend.attn_implementation
sdpa
config.backend.deepspeed_inference
False
config.environment.optimum_version
1.19.2
config.backend.to_bettertransformer
False
config.backend.torch_compile_target
forward
config.environment.accelerate_version
0.30.1
config.environment.transformers_version
4.40.2
config.scenario.input_shapes.batch_size
1
config.scenario.input_shapes.num_choices
2
config.backend.hub_kwargs.trust_remote_code
True
config.environment.optimum_benchmark_version
0.2.1
config.scenario.input_shapes.sequence_length
256
config.backend.model_kwargs.trust_remote_code
True
config.scenario.generate_kwargs.max_new_tokens
64
config.scenario.generate_kwargs.min_new_tokens
64
config.backend.processor_kwargs.trust_remote_code
True
metric_kind
prefill memory max_global_vram
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
optimum-benchmark/llm-perf-leaderboard/perf-df-pytorch-cuda-unquantized-1xA100.csv
published_at
2024-12-13T10:07:03Z
source_revision
2fa8d8c300c4eb6e744d5e35999db926c0c26e90e767e4c38da326eebe21a92b
metric_population
1
benchmark_evaluation
true
comparison_configuration_sha256
e98269fb859d479cf594053da82963997bf4c42df3b1a0f461c21547439bdc33
complete_configuration_location
source record config and original CSV artifact
source model id
01-ai/Yi-34B
Original result ↗

LLM-Perf

70,718.063

source id
llmPerf
leaderboard name
decode memory max_global_vram
dimension
runtime
metric
decode memory max_global_vram
rank
1
aggregation scope
status
published
direction
lowerIsBetter
conditions
suite
optimum-benchmark/llm-perf-leaderboard/perf-df-pytorch-cuda-unquantized-1xA100.csv
execution
config.backend.name
pytorch
config.backend.seed
42
config.backend.task
text-generation
config.backend.device
cuda
config.backend.library
transformers
config.backend.version
2.3.0+cu121
config.environment.cpu
AMD EPYC 7742 64-Core Processor
config.environment.gpu
['NVIDIA A100-SXM4-80GB']
config.backend.eval_mode
True
config.backend.device_ids
0
config.backend.no_weights
True
config.environment.system
Linux
config.backend.torch_dtype
float16
config.environment.machine
x86_64
config.backend.torch_compile
False
config.environment.cpu_count
128
config.environment.gpu_count
1
config.environment.processor
x86_64
config.environment.cpu_ram_mb
540671.627264
config.environment.gpu_vram_mb
85899345920
config.backend.autocast_enabled
False
config.environment.peft_version
0.11.0
config.environment.python_version
3.10.12
config.backend.attn_implementation
sdpa
config.backend.deepspeed_inference
False
config.environment.optimum_version
1.19.2
config.backend.to_bettertransformer
False
config.backend.torch_compile_target
forward
config.environment.accelerate_version
0.30.1
config.environment.transformers_version
4.40.2
config.scenario.input_shapes.batch_size
1
config.scenario.input_shapes.num_choices
2
config.backend.hub_kwargs.trust_remote_code
True
config.environment.optimum_benchmark_version
0.2.1
config.scenario.input_shapes.sequence_length
256
config.backend.model_kwargs.trust_remote_code
True
config.scenario.generate_kwargs.max_new_tokens
64
config.scenario.generate_kwargs.min_new_tokens
64
config.backend.processor_kwargs.trust_remote_code
True
metric_kind
decode memory max_global_vram
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
optimum-benchmark/llm-perf-leaderboard/perf-df-pytorch-cuda-unquantized-1xA100.csv
published_at
2024-12-13T10:07:03Z
source_revision
2fa8d8c300c4eb6e744d5e35999db926c0c26e90e767e4c38da326eebe21a92b
metric_population
1
benchmark_evaluation
true
comparison_configuration_sha256
e98269fb859d479cf594053da82963997bf4c42df3b1a0f461c21547439bdc33
complete_configuration_location
source record config and original CSV artifact
source model id
01-ai/Yi-34B
Original result ↗

LLM-Perf

70,715.965

source id
llmPerf
leaderboard name
decode memory max_global_vram
dimension
runtime
metric
decode memory max_global_vram
rank
1
aggregation scope
status
published
direction
lowerIsBetter
conditions
suite
optimum-benchmark/llm-perf-leaderboard/perf-df-unquantized-1xA100.csv
execution
config.backend.name
pytorch
config.backend.seed
42
config.backend.task
text-generation
config.backend.device
cuda
config.backend.library
transformers
config.backend.version
2.3.0+cu121
config.environment.cpu
AMD EPYC 7742 64-Core Processor
config.environment.gpu
['NVIDIA A100-SXM4-80GB']
config.backend.eval_mode
True
config.backend.device_ids
0
config.backend.no_weights
True
config.environment.system
Linux
config.backend.torch_dtype
bfloat16
config.environment.machine
x86_64
config.backend.torch_compile
False
config.environment.cpu_count
128
config.environment.gpu_count
1
config.environment.processor
x86_64
config.environment.cpu_ram_mb
540671.627264
config.environment.gpu_vram_mb
85899345920
config.backend.autocast_enabled
False
config.environment.peft_version
0.11.0
config.environment.python_version
3.10.12
config.backend.attn_implementation
eager
config.backend.deepspeed_inference
False
config.environment.optimum_version
1.19.2
config.backend.to_bettertransformer
False
config.backend.torch_compile_target
forward
config.environment.accelerate_version
0.30.1
config.environment.transformers_version
4.40.2
config.scenario.input_shapes.batch_size
1
config.scenario.input_shapes.num_choices
2
config.backend.hub_kwargs.trust_remote_code
True
config.environment.optimum_benchmark_version
0.2.1
config.scenario.input_shapes.sequence_length
256
config.backend.model_kwargs.trust_remote_code
True
config.scenario.generate_kwargs.max_new_tokens
64
config.scenario.generate_kwargs.min_new_tokens
64
config.backend.processor_kwargs.trust_remote_code
True
metric_kind
decode memory max_global_vram
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
optimum-benchmark/llm-perf-leaderboard/perf-df-unquantized-1xA100.csv
published_at
2024-12-13T10:07:03Z
source_revision
2fa8d8c300c4eb6e744d5e35999db926c0c26e90e767e4c38da326eebe21a92b
metric_population
1
benchmark_evaluation
true
comparison_configuration_sha256
468e8460883515023a1c9c7cf5a09717acb57fa4558b60901a8ccbf9d5f72121
complete_configuration_location
source record config and original CSV artifact
source model id
01-ai/Yi-34B
Original result ↗

LLM-Perf

70,715.965

source id
llmPerf
leaderboard name
decode memory max_global_vram
dimension
runtime
metric
decode memory max_global_vram
rank
1
aggregation scope
status
published
direction
lowerIsBetter
conditions
suite
optimum-benchmark/llm-perf-leaderboard/perf-df-pytorch-cuda-unquantized-1xA100.csv
execution
config.backend.name
pytorch
config.backend.seed
42
config.backend.task
text-generation
config.backend.device
cuda
config.backend.library
transformers
config.backend.version
2.3.0+cu121
config.environment.cpu
AMD EPYC 7742 64-Core Processor
config.environment.gpu
['NVIDIA A100-SXM4-80GB']
config.backend.eval_mode
True
config.backend.device_ids
0
config.backend.no_weights
True
config.environment.system
Linux
config.backend.torch_dtype
bfloat16
config.environment.machine
x86_64
config.backend.torch_compile
False
config.environment.cpu_count
128
config.environment.gpu_count
1
config.environment.processor
x86_64
config.environment.cpu_ram_mb
540671.627264
config.environment.gpu_vram_mb
85899345920
config.backend.autocast_enabled
False
config.environment.peft_version
0.11.0
config.environment.python_version
3.10.12
config.backend.attn_implementation
eager
config.backend.deepspeed_inference
False
config.environment.optimum_version
1.19.2
config.backend.to_bettertransformer
False
config.backend.torch_compile_target
forward
config.environment.accelerate_version
0.30.1
config.environment.transformers_version
4.40.2
config.scenario.input_shapes.batch_size
1
config.scenario.input_shapes.num_choices
2
config.backend.hub_kwargs.trust_remote_code
True
config.environment.optimum_benchmark_version
0.2.1
config.scenario.input_shapes.sequence_length
256
config.backend.model_kwargs.trust_remote_code
True
config.scenario.generate_kwargs.max_new_tokens
64
config.scenario.generate_kwargs.min_new_tokens
64
config.backend.processor_kwargs.trust_remote_code
True
metric_kind
decode memory max_global_vram
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
optimum-benchmark/llm-perf-leaderboard/perf-df-pytorch-cuda-unquantized-1xA100.csv
published_at
2024-12-13T10:07:03Z
source_revision
2fa8d8c300c4eb6e744d5e35999db926c0c26e90e767e4c38da326eebe21a92b
metric_population
1
benchmark_evaluation
true
comparison_configuration_sha256
5a7bfe2f08af6456f222f387892347ad2dc4618106b698f9ade8192bb1e13c85
complete_configuration_location
source record config and original CSV artifact
source model id
01-ai/Yi-34B
Original result ↗

LLM-Perf

70,715.965

source id
llmPerf
leaderboard name
prefill memory max_global_vram
dimension
runtime
metric
prefill memory max_global_vram
rank
1
aggregation scope
status
published
direction
lowerIsBetter
conditions
suite
optimum-benchmark/llm-perf-leaderboard/perf-df-unquantized-1xA100.csv
execution
config.backend.name
pytorch
config.backend.seed
42
config.backend.task
text-generation
config.backend.device
cuda
config.backend.library
transformers
config.backend.version
2.3.0+cu121
config.environment.cpu
AMD EPYC 7742 64-Core Processor
config.environment.gpu
['NVIDIA A100-SXM4-80GB']
config.backend.eval_mode
True
config.backend.device_ids
0
config.backend.no_weights
True
config.environment.system
Linux
config.backend.torch_dtype
bfloat16
config.environment.machine
x86_64
config.backend.torch_compile
False
config.environment.cpu_count
128
config.environment.gpu_count
1
config.environment.processor
x86_64
config.environment.cpu_ram_mb
540671.627264
config.environment.gpu_vram_mb
85899345920
config.backend.autocast_enabled
False
config.environment.peft_version
0.11.0
config.environment.python_version
3.10.12
config.backend.attn_implementation
eager
config.backend.deepspeed_inference
False
config.environment.optimum_version
1.19.2
config.backend.to_bettertransformer
False
config.backend.torch_compile_target
forward
config.environment.accelerate_version
0.30.1
config.environment.transformers_version
4.40.2
config.scenario.input_shapes.batch_size
1
config.scenario.input_shapes.num_choices
2
config.backend.hub_kwargs.trust_remote_code
True
config.environment.optimum_benchmark_version
0.2.1
config.scenario.input_shapes.sequence_length
256
config.backend.model_kwargs.trust_remote_code
True
config.scenario.generate_kwargs.max_new_tokens
64
config.scenario.generate_kwargs.min_new_tokens
64
config.backend.processor_kwargs.trust_remote_code
True
metric_kind
prefill memory max_global_vram
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
optimum-benchmark/llm-perf-leaderboard/perf-df-unquantized-1xA100.csv
published_at
2024-12-13T10:07:03Z
source_revision
2fa8d8c300c4eb6e744d5e35999db926c0c26e90e767e4c38da326eebe21a92b
metric_population
1
benchmark_evaluation
true
comparison_configuration_sha256
468e8460883515023a1c9c7cf5a09717acb57fa4558b60901a8ccbf9d5f72121
complete_configuration_location
source record config and original CSV artifact
source model id
01-ai/Yi-34B
Original result ↗

LLM-Perf

70,715.965

source id
llmPerf
leaderboard name
prefill memory max_global_vram
dimension
runtime
metric
prefill memory max_global_vram
rank
1
aggregation scope
status
published
direction
lowerIsBetter
conditions
suite
optimum-benchmark/llm-perf-leaderboard/perf-df-pytorch-cuda-unquantized-1xA100.csv
execution
config.backend.name
pytorch
config.backend.seed
42
config.backend.task
text-generation
config.backend.device
cuda
config.backend.library
transformers
config.backend.version
2.3.0+cu121
config.environment.cpu
AMD EPYC 7742 64-Core Processor
config.environment.gpu
['NVIDIA A100-SXM4-80GB']
config.backend.eval_mode
True
config.backend.device_ids
0
config.backend.no_weights
True
config.environment.system
Linux
config.backend.torch_dtype
bfloat16
config.environment.machine
x86_64
config.backend.torch_compile
False
config.environment.cpu_count
128
config.environment.gpu_count
1
config.environment.processor
x86_64
config.environment.cpu_ram_mb
540671.627264
config.environment.gpu_vram_mb
85899345920
config.backend.autocast_enabled
False
config.environment.peft_version
0.11.0
config.environment.python_version
3.10.12
config.backend.attn_implementation
eager
config.backend.deepspeed_inference
False
config.environment.optimum_version
1.19.2
config.backend.to_bettertransformer
False
config.backend.torch_compile_target
forward
config.environment.accelerate_version
0.30.1
config.environment.transformers_version
4.40.2
config.scenario.input_shapes.batch_size
1
config.scenario.input_shapes.num_choices
2
config.backend.hub_kwargs.trust_remote_code
True
config.environment.optimum_benchmark_version
0.2.1
config.scenario.input_shapes.sequence_length
256
config.backend.model_kwargs.trust_remote_code
True
config.scenario.generate_kwargs.max_new_tokens
64
config.scenario.generate_kwargs.min_new_tokens
64
config.backend.processor_kwargs.trust_remote_code
True
metric_kind
prefill memory max_global_vram
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
optimum-benchmark/llm-perf-leaderboard/perf-df-pytorch-cuda-unquantized-1xA100.csv
published_at
2024-12-13T10:07:03Z
source_revision
2fa8d8c300c4eb6e744d5e35999db926c0c26e90e767e4c38da326eebe21a92b
metric_population
1
benchmark_evaluation
true
comparison_configuration_sha256
5a7bfe2f08af6456f222f387892347ad2dc4618106b698f9ade8192bb1e13c85
complete_configuration_location
source record config and original CSV artifact
source model id
01-ai/Yi-34B
Original result ↗

LLM-Perf

69,791.121

source id
llmPerf
leaderboard name
decode memory max_process_vram
dimension
runtime
metric
decode memory max_process_vram
rank
1
aggregation scope
status
published
direction
lowerIsBetter
conditions
suite
optimum-benchmark/llm-perf-leaderboard/perf-df-pytorch-cuda-unquantized-1xA100.csv
execution
config.backend.name
pytorch
config.backend.seed
42
config.backend.task
text-generation
config.backend.device
cuda
config.backend.library
transformers
config.backend.version
2.3.0+cu121
config.environment.cpu
AMD EPYC 7742 64-Core Processor
config.environment.gpu
['NVIDIA A100-SXM4-80GB']
config.backend.eval_mode
True
config.backend.device_ids
0
config.backend.no_weights
True
config.environment.system
Linux
config.backend.torch_dtype
float16
config.environment.machine
x86_64
config.backend.torch_compile
False
config.environment.cpu_count
128
config.environment.gpu_count
1
config.environment.processor
x86_64
config.environment.cpu_ram_mb
540671.627264
config.environment.gpu_vram_mb
85899345920
config.backend.autocast_enabled
False
config.environment.peft_version
0.11.0
config.environment.python_version
3.10.12
config.backend.attn_implementation
sdpa
config.backend.deepspeed_inference
False
config.environment.optimum_version
1.19.2
config.backend.to_bettertransformer
False
config.backend.torch_compile_target
forward
config.environment.accelerate_version
0.30.1
config.environment.transformers_version
4.40.2
config.scenario.input_shapes.batch_size
1
config.scenario.input_shapes.num_choices
2
config.backend.hub_kwargs.trust_remote_code
True
config.environment.optimum_benchmark_version
0.2.1
config.scenario.input_shapes.sequence_length
256
config.backend.model_kwargs.trust_remote_code
True
config.scenario.generate_kwargs.max_new_tokens
64
config.scenario.generate_kwargs.min_new_tokens
64
config.backend.processor_kwargs.trust_remote_code
True
metric_kind
decode memory max_process_vram
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
optimum-benchmark/llm-perf-leaderboard/perf-df-pytorch-cuda-unquantized-1xA100.csv
published_at
2024-12-13T10:07:03Z
source_revision
2fa8d8c300c4eb6e744d5e35999db926c0c26e90e767e4c38da326eebe21a92b
metric_population
1
benchmark_evaluation
true
comparison_configuration_sha256
e98269fb859d479cf594053da82963997bf4c42df3b1a0f461c21547439bdc33
complete_configuration_location
source record config and original CSV artifact
source model id
01-ai/Yi-34B
Original result ↗

LLM-Perf

69,791.121

source id
llmPerf
leaderboard name
decode memory max_process_vram
dimension
runtime
metric
decode memory max_process_vram
rank
1
aggregation scope
status
published
direction
lowerIsBetter
conditions
suite
optimum-benchmark/llm-perf-leaderboard/perf-df-unquantized-1xA100.csv
execution
config.backend.name
pytorch
config.backend.seed
42
config.backend.task
text-generation
config.backend.device
cuda
config.backend.library
transformers
config.backend.version
2.3.0+cu121
config.environment.cpu
AMD EPYC 7742 64-Core Processor
config.environment.gpu
['NVIDIA A100-SXM4-80GB']
config.backend.eval_mode
True
config.backend.device_ids
0
config.backend.no_weights
True
config.environment.system
Linux
config.backend.torch_dtype
float16
config.environment.machine
x86_64
config.backend.torch_compile
False
config.environment.cpu_count
128
config.environment.gpu_count
1
config.environment.processor
x86_64
config.environment.cpu_ram_mb
540671.627264
config.environment.gpu_vram_mb
85899345920
config.backend.autocast_enabled
False
config.environment.peft_version
0.11.0
config.environment.python_version
3.10.12
config.backend.attn_implementation
sdpa
config.backend.deepspeed_inference
False
config.environment.optimum_version
1.19.2
config.backend.to_bettertransformer
False
config.backend.torch_compile_target
forward
config.environment.accelerate_version
0.30.1
config.environment.transformers_version
4.40.2
config.scenario.input_shapes.batch_size
1
config.scenario.input_shapes.num_choices
2
config.backend.hub_kwargs.trust_remote_code
True
config.environment.optimum_benchmark_version
0.2.1
config.scenario.input_shapes.sequence_length
256
config.backend.model_kwargs.trust_remote_code
True
config.scenario.generate_kwargs.max_new_tokens
64
config.scenario.generate_kwargs.min_new_tokens
64
config.backend.processor_kwargs.trust_remote_code
True
metric_kind
decode memory max_process_vram
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
optimum-benchmark/llm-perf-leaderboard/perf-df-unquantized-1xA100.csv
published_at
2024-12-13T10:07:03Z
source_revision
2fa8d8c300c4eb6e744d5e35999db926c0c26e90e767e4c38da326eebe21a92b
metric_population
1
benchmark_evaluation
true
comparison_configuration_sha256
4d1ad767823f5984c1333a2c80a1b8794a0ef5ac80d44cde71dc4faf09969111
complete_configuration_location
source record config and original CSV artifact
source model id
01-ai/Yi-34B
Original result ↗

LLM-Perf

69,791.121

source id
llmPerf
leaderboard name
prefill memory max_process_vram
dimension
runtime
metric
prefill memory max_process_vram
rank
1
aggregation scope
status
published
direction
lowerIsBetter
conditions
suite
optimum-benchmark/llm-perf-leaderboard/perf-df-unquantized-1xA100.csv
execution
config.backend.name
pytorch
config.backend.seed
42
config.backend.task
text-generation
config.backend.device
cuda
config.backend.library
transformers
config.backend.version
2.3.0+cu121
config.environment.cpu
AMD EPYC 7742 64-Core Processor
config.environment.gpu
['NVIDIA A100-SXM4-80GB']
config.backend.eval_mode
True
config.backend.device_ids
0
config.backend.no_weights
True
config.environment.system
Linux
config.backend.torch_dtype
float16
config.environment.machine
x86_64
config.backend.torch_compile
False
config.environment.cpu_count
128
config.environment.gpu_count
1
config.environment.processor
x86_64
config.environment.cpu_ram_mb
540671.627264
config.environment.gpu_vram_mb
85899345920
config.backend.autocast_enabled
False
config.environment.peft_version
0.11.0
config.environment.python_version
3.10.12
config.backend.attn_implementation
sdpa
config.backend.deepspeed_inference
False
config.environment.optimum_version
1.19.2
config.backend.to_bettertransformer
False
config.backend.torch_compile_target
forward
config.environment.accelerate_version
0.30.1
config.environment.transformers_version
4.40.2
config.scenario.input_shapes.batch_size
1
config.scenario.input_shapes.num_choices
2
config.backend.hub_kwargs.trust_remote_code
True
config.environment.optimum_benchmark_version
0.2.1
config.scenario.input_shapes.sequence_length
256
config.backend.model_kwargs.trust_remote_code
True
config.scenario.generate_kwargs.max_new_tokens
64
config.scenario.generate_kwargs.min_new_tokens
64
config.backend.processor_kwargs.trust_remote_code
True
metric_kind
prefill memory max_process_vram
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
optimum-benchmark/llm-perf-leaderboard/perf-df-unquantized-1xA100.csv
published_at
2024-12-13T10:07:03Z
source_revision
2fa8d8c300c4eb6e744d5e35999db926c0c26e90e767e4c38da326eebe21a92b
metric_population
1
benchmark_evaluation
true
comparison_configuration_sha256
4d1ad767823f5984c1333a2c80a1b8794a0ef5ac80d44cde71dc4faf09969111
complete_configuration_location
source record config and original CSV artifact
source model id
01-ai/Yi-34B
Original result ↗

LLM-Perf

69,791.121

source id
llmPerf
leaderboard name
prefill memory max_process_vram
dimension
runtime
metric
prefill memory max_process_vram
rank
1
aggregation scope
status
published
direction
lowerIsBetter
conditions
suite
optimum-benchmark/llm-perf-leaderboard/perf-df-pytorch-cuda-unquantized-1xA100.csv
execution
config.backend.name
pytorch
config.backend.seed
42
config.backend.task
text-generation
config.backend.device
cuda
config.backend.library
transformers
config.backend.version
2.3.0+cu121
config.environment.cpu
AMD EPYC 7742 64-Core Processor
config.environment.gpu
['NVIDIA A100-SXM4-80GB']
config.backend.eval_mode
True
config.backend.device_ids
0
config.backend.no_weights
True
config.environment.system
Linux
config.backend.torch_dtype
float16
config.environment.machine
x86_64
config.backend.torch_compile
False
config.environment.cpu_count
128
config.environment.gpu_count
1
config.environment.processor
x86_64
config.environment.cpu_ram_mb
540671.627264
config.environment.gpu_vram_mb
85899345920
config.backend.autocast_enabled
False
config.environment.peft_version
0.11.0
config.environment.python_version
3.10.12
config.backend.attn_implementation
sdpa
config.backend.deepspeed_inference
False
config.environment.optimum_version
1.19.2
config.backend.to_bettertransformer
False
config.backend.torch_compile_target
forward
config.environment.accelerate_version
0.30.1
config.environment.transformers_version
4.40.2
config.scenario.input_shapes.batch_size
1
config.scenario.input_shapes.num_choices
2
config.backend.hub_kwargs.trust_remote_code
True
config.environment.optimum_benchmark_version
0.2.1
config.scenario.input_shapes.sequence_length
256
config.backend.model_kwargs.trust_remote_code
True
config.scenario.generate_kwargs.max_new_tokens
64
config.scenario.generate_kwargs.min_new_tokens
64
config.backend.processor_kwargs.trust_remote_code
True
metric_kind
prefill memory max_process_vram
rank_method
competition rank among reported evaluations in identical conditions, including unlinked observations
source_file
optimum-benchmark/llm-perf-leaderboard/perf-df-pytorch-cuda-unquantized-1xA100.csv
published_at
2024-12-13T10:07:03Z
source_revision
2fa8d8c300c4eb6e744d5e35999db926c0c26e90e767e4c38da326eebe21a92b
metric_population
1
benchmark_evaluation
true
comparison_configuration_sha256
e98269fb859d479cf594053da82963997bf4c42df3b1a0f461c21547439bdc33
complete_configuration_location
source record config and original CSV artifact
source model id
01-ai/Yi-34B
Original result ↗

Provider specifications & access

1–1 / 1

01-ai/Yi-34B ↗

source model id
HuggingFace/01-ai-yi-34b@1
hosting
foundry-managed-compute
platforms
  • Microsoft Foundry
license
apache-2.0
architecture
input_modalities
  • text
output_modalities
  • text
capabilities

    Original sources

    1–8 / 8

    Hugging Face · 01-ai/Yi-34B ↗

    Microsoft Foundry · Model page ↗

    Microsoft Foundry · Model documentation ↗

    Hugging Face · Catalog ↗

    Hugging Face · Catalog ↗

    Hugging Face · Catalog ↗

    Hugging Face · Catalog ↗

    Microsoft Foundry · Catalog ↗

    Source links

    1–5 / 5

    Explore more models

    Models

    Loading
    EVIDENCE / COMPARISON

    Benchmarks

    Explore every benchmark

    Loading

    These charts contain model-linked results. The charts above also include unlinked source profiles. Compare scores within each evaluation. Model quality, usage, transparency and system performance measure different things.

    Install TensorTrace