microsoft/dit-large-finetuned-rvlcdip ↗
- source model id
- HuggingFace/microsoft-dit-large-finetuned-rvlcdip@1
- hosting
- foundry-managed-compute
- platforms
- Microsoft Foundry
- license
- unknown
- architecture
- input_modalities
- output_modalities
- capabilities
# Document Image Transformer (large-sized model) Document Image Transformer (DiT) model pre-trained on IIT-CDIP (Lewis et al., 2006), a dataset that includes 42 million document images and fine-tuned…
azureFoundry/HuggingFace/microsoft-dit-large-finetuned-rvlcdip@1
Document Image Transformer (DiT) model pre-trained on IIT-CDIP (Lewis et al., 2006), a dataset that includes 42 million document images and fine-tuned on RVL-CDIP, a dataset consisting of 400,000 grayscale images in 16 classes, with 25,000 images per class. It was introduced in the paper DiT: Self-supervised Pre-training for Document Image Transformer by Li et al. and first released in this repository. Note that DiT is identical to the architecture of BEiT.
Disclaimer: The team releasing DiT did not write a model card for this model so this model card has been written by the Hugging Face team.
The Document Image Transformer (DiT) is a transformer encoder model (BERT-like) pre-trained on a large collection of images in a self-supervised fashion. The pre-training objective for the model is to predict visual tokens from the encoder of a discrete VAE (dVAE), based on masked patches.
Images are presented to the model as a sequence of fixed-size patches (resolution 16x16), which are linearly embedded. One also adds absolute position embeddings before feeding the sequence to the layers of the Transformer encoder.
By pre-training the model, it learns an inner representation of images that can then be used to extract features useful for downstream tasks: if you have a dataset of labeled document images for instance, you can train a standard classifier by placing a linear layer on top of the pre-trained encoder.
You can use the raw model for encoding document images into a vector space, but it's mostly meant to be fine-tuned on tasks like document image classification, table detection or document layout analysis. See the model hub to look for fine-tuned versions on a task that interests you.
Here is how to use this model in PyTorch:
from transformers import AutoImageProcessor, AutoModelForImageClassification
import torch
from PIL import Image
image = Image.open('path_to_your_document_image').convert('RGB')
processor = AutoImageProcessor.from_pretrained("microsoft/dit-large-finetuned-rvlcdip")
model = AutoModelForImageClassification.from_pretrained("microsoft/dit-large-finetuned-rvlcdip")
inputs = processor(images=image, return_tensors="pt")
outputs = model(**inputs)
logits = outputs.logits
# model predicts one of the 16 RVL-CDIP classes
predicted_class_idx = logits.argmax(-1).item()
print("Predicted class:", model.config.id2label[predicted_class_idx])
@article{Lewis2006BuildingAT,
title={Building a test collection for complex document information processing},
author={David D. Lewis and Gady Agam and Shlomo Engelson Argamon and Ophir Frieder and David A. Grossman and Jefferson Heard},
journal={Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval},
year={2006}
}
microsoft/dit-large-finetuned-rvlcdip powered by Hugging Face Inference Toolkit
You can use cURL or any REST Client to send a request to the AzureML endpoint with your AzureML token.
curl <AZUREML_ENDPOINT_URL> \
-X POST \
-H "Authorization: Bearer <AZUREML_TOKEN>" \
-H "Content-Type: image/jpeg" \
--data-binary @"image.jpg"
Check the full API Specification at the Hugging Face Inference documentation.
1–1 / 1
1–4 / 4
Hugging Face · microsoft/dit-large-finetuned-rvlcdip ↗
Microsoft Foundry · Model page ↗
1–1 / 1
These charts contain model-linked results. The charts above also include unlinked source profiles. Compare scores within each evaluation. Model quality, usage, transparency and system performance measure different things.