OpenAI-CLIP-Image-Text-Embeddings-ViT-Large-Patch14-336 · azureml v3

The `CLIP` model was developed by researchers at OpenAI to learn about what contributes to robustness in computer vision tasks. The model was also developed to test the ability of models to generaliz…

azureFoundry/azureml/OpenAI-CLIP-Image-Text-Embeddings-ViT-Large-Patch14-336@3

The CLIP model was developed by researchers at OpenAI to learn about what contributes to robustness in computer vision tasks. The model was also developed to test the ability of models to generalize to arbitrary image classification tasks in a zero-shot manner. It was not developed for general model deployment - to deploy models like `CLIP, researchers will first need to carefully study their capabilities in relation to the specific context they’re being deployed within.

This model uses a ViT-L/14 Transformer architecture trained at 336x336 pixel resolution as an image encoder and uses a masked self-attention Transformer as a text encoder. These encoders are trained to maximize the similarity of (image, text) pairs via a contrastive loss.

The primary intended users of these models are AI researchers for tasks requiring image and/or text embeddings such as text and image retrieval.

Training Details

Training Data

The model was trained on publicly available image-caption data. This was done through a combination of crawling a handful of websites and using commonly-used pre-existing image datasets such as YFCC100M. A large portion of the training data comes from the authors' crawling of the internet. This means that the data is more representative of people and societies most connected to the internet which tend to skew towards more developed nations, and younger, male users.

Evaluation Results

This model was evaluated for text retrieval and image retrieval tasks on the Flickr30k and MSCOCO datasets. The results from Table 13 in the original CLIP paper are summarized below

Text Retrieval

DatasetR@1R@5
Flickr30k88.098.7
MSCOCO58.481.5

Image Retrieval

DatasetR@1R@5
Flickr30k68.790.6
MSCOCO37.862.4

Limitations and Biases

Limitations

CLIP currently struggles with respect to certain tasks such as fine grained classification and counting objects. CLIP also poses issues with regards to fairness and bias which the authors discuss in the paper and is described briefly in the next section. Additionally, the authors' approach to testing CLIP also has an important limitation- in many cases they have used linear probes to evaluate the performance of CLIP and there is evidence suggesting that linear probes can underestimate model performance.

Bias

The authors of the original CLIP paper found that the performance of the model and its biases can depend significantly on class design and the choices one makes for categories to include and exclude. They tested the risk of certain kinds of denigration with CLIP by classifying images of people from Fairface into crime-related and non-human animal categories. They found significant disparities with respect to race and gender, which could shift based on how the classes were constructed. The authors also tested the performance of CLIP on gender, race, and age classification using the Fairface dataset. They found that the accuracy for gender classification was greater than 96% across all races, with 'Middle Eastern' having the highest accuracy (98.4%) and 'White' having the lowest (96.5%). Additionally, CLIP averaged ~93% for racial classification and ~63% for age classification.

License

MIT License

Inference Samples

Inference typePython sample (Notebook)CLI with YAML
Real timeimage-text-embeddings-online-endpoint.ipynbimage-text-embeddings-online-endpoint.sh
Batchimage-text-embeddings-batch-endpoint.ipynbimage-text-embeddings-batch-endpoint.sh

Sample input and output for image embeddings

Sample input

{
   "input_data":{
      "columns":[
         "image", "text"
      ],
      "index":[0, 1],
      "data":[
         ["image1", ""],
         ["image2", ""]
      ]
   }
}

Note: "image1" and "image2" should be publicly accessible urls or strings in base64 format

Sample output

[
    {
        "image_features": [-0.92, -0.13, 0.02, ... , 0.13],
    },
    {
        "image_features": [0.54, -0.83, 0.13, ... , 0.26],
    }
]

Note: returned embeddings have dimension 768 and are not normalized

Sample input and output for text embeddings

Sample input

{
   "input_data":{
      "columns":[
         "image", "text"
      ],
      "index":[0, 1],
      "data":[
         ["", "sample text 1"],
         ["", "sample text 2"]
      ]
   }
}

Sample output

[
    {
        "text_features": [0.42, -0.13, -0.92, ... , 0.63],
    },
    {
        "text_features": [-0.14, 0.93, -0.15, ... , 0.66],
    }
]

Note: returned embeddings have dimension 768 and are not normalized

Sample input and output for image and text embeddings

Sample input

{
   "input_data":{
      "columns":[
         "image", "text"
      ],
      "index":[0, 1],
      "data":[
         ["image1", "sample text 1"],
         ["image2", "sample text 2"]
      ]
   }
}

Note: "image1" and "image2" should be publicly accessible urls or strings in base64 format

Sample output

[
    {
        "image_features": [0.92, -0.13, 0.02, ... , -0.13],
        "text_features": [0.42, 0.13, -0.92, ... , -0.63]
    },
    {
        "image_features": [-0.54, -0.83, 0.13, ... , -0.26],
        "text_features": [-0.14, -0.93, 0.15, ... , 0.66]
    }
]

Note: returned embeddings have dimension 768 and are not normalized

Specifications & capabilities

Input
text
Output
embeddings
Capabilities
embeddings

Provider specifications & access

1–1 / 1

Safety assessments

1–1 / 1

Foundation Model Transparency Index 2025 ↗

35 / 100

source id
stanfordFMTI
canonical model id
azureFoundry/azureml/OpenAI-CLIP-Image-Text-Embeddings-ViT-Large-Patch14-336@3
taxonomy
transparency
hazard category
transparency
hazard name
Organization transparency
policy area
OpenAI
assessment type
organization-level disclosure
metric
disclosure_score
result label
35 / 100
sample size
100
updated at
2026-11-09T18:08:05.227Z

Original sources

1–4 / 4

Microsoft Foundry · Model page ↗

Microsoft Foundry · Model documentation ↗

GitHub · Catalog ↗

Microsoft Foundry · Catalog ↗

Source links

1–2 / 2

Explore more models

Models

Loading
EVIDENCE / COMPARISON

Benchmarks

Explore every benchmark

Loading

These charts contain model-linked results. The charts above also include unlinked source profiles. Compare scores within each evaluation. Model quality, usage, transparency and system performance measure different things.

Install TensorTrace