microsoft/git-base-vatex

# GIT (GenerativeImage2Text), base-sized, fine-tuned on VATEX GIT (short for GenerativeImage2Text) model, base-sized version, fine-tuned on VATEX. It was introduced in the paper [GIT: A Generative Im…

azureFoundry/HuggingFace/microsoft-git-base-vatex@3

GIT (GenerativeImage2Text), base-sized, fine-tuned on VATEX

GIT (short for GenerativeImage2Text) model, base-sized version, fine-tuned on VATEX. It was introduced in the paper GIT: A Generative Image-to-text Transformer for Vision and Language by Wang et al. and first released in this repository.

Disclaimer: The team releasing GIT did not write a model card for this model so this model card has been written by the Hugging Face team.

Model description

GIT is a Transformer decoder conditioned on both CLIP image tokens and text tokens. The model is trained using "teacher forcing" on a lot of (image, text) pairs.

The goal for the model is simply to predict the next text token, giving the image tokens and previous text tokens.

The model has full access to (i.e. a bidirectional attention mask is used for) the image patch tokens, but only has access to the previous text tokens (i.e. a causal attention mask is used for the text tokens) when predicting the next text token.

GIT architecture

This allows the model to be used for tasks like:

  • image and video captioning
  • visual question answering (VQA) on images and videos
  • even image classification (by simply conditioning the model on the image and asking it to generate a class for it in text).

Intended uses & limitations

You can use the raw model for video captioning. See the model hub to look for fine-tuned versions on a task that interests you.

How to use

For code examples, we refer to the documentation.

Training data

From the paper:

We collect 0.8B image-text pairs for pre-training, which include COCO (Lin et al., 2014), Conceptual Captions (CC3M) (Sharma et al., 2018), SBU (Ordonez et al., 2011), Visual Genome (VG) (Krishna et al., 2016), Conceptual Captions (CC12M) (Changpinyo et al., 2021), ALT200M (Hu et al., 2021a), and an extra 0.6B data following a similar collection procedure in Hu et al. (2021a).

=> however this is for the model referred to as "GIT" in the paper, which is not open-sourced.

This checkpoint is "GIT-base", which is a smaller variant of GIT trained on 10 million image-text pairs.

Next, the model was fine-tuned on VATEX.

See table 11 in the paper for more details.

Preprocessing

We refer to the original repo regarding details for preprocessing during training.

During validation, one resizes the shorter edge of each image, after which center cropping is performed to a fixed-size resolution. Next, frames are normalized across the RGB channels with the ImageNet mean and standard deviation.

Evaluation results

For evaluation results, we refer readers to the paper.


microsoft/git-base-vatex is a pre-trained language model available on the Hugging Face Hub. It's specifically designed for the text-generation task in the transformers library. If you want to learn more about the model's architecture, hyperparameters, limitations, and biases, you can find this information on the model's dedicated Model Card on the Hugging Face Hub.

Here's an example API request payload that you can use to obtain predictions from the model:

{
  "inputs": "My name is Julien and I like to"
}

Specifications & capabilities

Input
text
Output
text
Published model ID
git-base-vatex

Provider specifications & access

1–1 / 1

microsoft/git-base-vatex ↗

source model id
HuggingFace/microsoft-git-base-vatex@3
hosting
foundry-managed-compute
platforms
  • Microsoft Foundry
license
mit
architecture
input_modalities
  • text
output_modalities
  • text
capabilities

    Original sources

    1–4 / 4

    Hugging Face · microsoft/git-base-vatex ↗

    Microsoft Foundry · Model page ↗

    Microsoft Foundry · Model documentation ↗

    Microsoft Foundry · Catalog ↗

    Source links

    1–1 / 1

    Explore more models

    Models

    Loading
    EVIDENCE / COMPARISON

    Benchmarks

    Explore every benchmark

    Loading

    These charts contain model-linked results. The charts above also include unlinked source profiles. Compare scores within each evaluation. Model quality, usage, transparency and system performance measure different things.

    Install TensorTrace