google-vit-base-patch16-224 · azureml v18

The Vision Transformer (ViT) model, as introduced in the <a href="https://arxiv.org/abs/2010.11929" target="_blank">paper</a> "An Image is Worth 16x16 Words: Transformers for Image Recognition at Sca…

azureFoundry/azureml/google-vit-base-patch16-224@18

The Vision Transformer (ViT) model, as introduced in the paper "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale" by Dosovitskiy et al., underwent pre-training on ImageNet-21k with a resolution of 224x224. Subsequently, it was fine-tuned on ImageNet 2012, consisting of 1 million images and 1,000 classes, also at a resolution of 224x224. The model was first released in this repository, but the weights were converted to PyTorch from the timm repository by Ross Wightman, who had previously converted the weights from JAX to PyTorch.

An image is treated as a sequence of patches and it is processed by a standard Transformer encoder as used in NLP. These patches are linearly embedded, and a [CLS] token is added at the beginning of the sequence for classification tasks. The model also requires absolute position embeddings before feeding the sequence Transformer encoder. So the pre-training creates an inner representation of images that can be used to extract features that are useful for downstream tasks. For instance, if a dataset of labeled images is available, a linear layer can be placed on top of the pre-trained encoder, to train a standard classifier.

Training Details

Training Data

The ViT model is pre-trained on ImageNet-21k dataset with a resolution of 224x224 and fine-tuned on ImageNet 2012, consisting of 1 million images and 1,000 classes.

Training Procedure

In the preprocessing step, images are resized to the same resolution 224x224. Then normalized across the RGB channels with mean (0.5, 0.5, 0.5) and standard deviation (0.5, 0.5, 0.5).

The model was trained on TPUv3 hardware (8 cores). All models are trained using Adam with β1 = 0.9, β2 = 0.999, with a batch size of 4096, a high weight decay of 0.1, learning rate warmup of 10k steps. Authors found that it is beneficial to additionally apply gradient clipping at global norm 1. Training resolution is 224. For more details on hyperparameters refer to table 3 of the original-paper.

For more details on self-supervised pre-training (ImageNet-21k) followed by supervised fine-tuning (ImageNet-1k) refer to the section 3 and 4 of the original-paper.

Evaluation Results

For ViT image classification benchmark results, Refer to table 2 and table 5 of the original-paper.

License

apache-2.0

Inference Samples

Inference typePython sample (Notebook)CLI with YAML
Real timeimage-classification-online-endpoint.ipynbimage-classification-online-endpoint.sh
Batchimage-classification-batch-endpoint.ipynbimage-classification-batch-endpoint.sh

Finetuning Samples

TaskUse caseDatasetPython sample (Notebook)CLI with YAML
Image Multi-class classificationImage Multi-class classificationfridgeObjectsfridgeobjects-multiclass-classification.ipynbfridgeobjects-multiclass-classification.sh
Image Multi-label classificationImage Multi-label classificationmultilabel fridgeObjectsfridgeobjects-multilabel-classification.ipynbfridgeobjects-multilabel-classification.sh

Evaluation Samples

TaskUse caseDatasetPython sample (Notebook)
Image Multi-class classificationImage Multi-class classificationfridgeObjectsimage-multiclass-classification.ipynb
Image Multi-label classificationImage Multi-label classificationmultilabel fridgeObjectsimage-multilabel-classification.ipynb

Sample input and output

Sample input

{
  "input_data": ["image1", "image2"]
}

Note: "image1" and "image2" string should be in base64 format or publicly accessible urls.

Sample output

[
  [
    {
      "label" : "can",
      "score" : 0.91
    },
    {
      "label" : "carton",
      "score" : 0.09
    },
  ],
  [
    {
      "label" : "carton",
      "score" : 0.9
    },
    {
      "label" : "can",
      "score" : 0.1
    },
  ]
]

Visualization of inference result for a sample image

mc visualization

Specifications & capabilities

Provider specifications & access

1–1 / 1

Safety assessments

1–1 / 1

Foundation Model Transparency Index 2025 ↗

41 / 100

source id
stanfordFMTI
canonical model id
azureFoundry/azureml/google-vit-base-patch16-224@18
taxonomy
transparency
hazard category
transparency
hazard name
Organization transparency
policy area
Google
assessment type
organization-level disclosure
metric
disclosure_score
result label
41 / 100
sample size
100
updated at
2026-11-09T18:08:05.227Z

Original sources

1–4 / 4

Microsoft Foundry · Model page ↗

Microsoft Foundry · Model documentation ↗

GitHub · Catalog ↗

Microsoft Foundry · Catalog ↗

Source links

1–2 / 2

Explore more models

Models

Loading
EVIDENCE / COMPARISON

Benchmarks

Explore every benchmark

Loading

These charts contain model-linked results. The charts above also include unlinked source profiles. Compare scores within each evaluation. Model quality, usage, transparency and system performance measure different things.

Install TensorTrace