mmeft · azureml v5

__Multimodal Early Fusion Transformer (MMEFT)__ is a transformer-based model tailored for processing both structured and unstructured data. It can be used for multi-class and multi-label multimodal c…

azureFoundry/azureml/mmeft@5

Multimodal Early Fusion Transformer (MMEFT) is a transformer-based model tailored for processing both structured and unstructured data.

It can be used for multi-class and multi-label multimodal classification tasks, and is capable of handling datasets with features from diverse modes, including categorical, numerical, image, and text. MMEFT architecture is composed of embedding, fusion, aggregation, and output layers. The embedding layer produces independent non-contextual embeddings for features of varying modes. Then, the fusion Layer integrates the non-contextual embeddings to yield contextual multimodal embeddings. The aggregation layer consolidates these contextual multimodal embeddings into a single multimodal embedding vector. Lastly, the output Layer, processes the final multimodal embedding to generate the model's prediction based on task for which it is used. MMEFT uses bert-base-uncased from HuggingFace for text data embeddings, resnet-18 from HuggingFace for image data embeddings, Feature Tokenizer + Transformer (FT-Transofrmer) from Revisiting Deep Learning Models for Tabular Data for tabular data embeddings. This model is designed to offer a comprehensive approach to multimodal data, ensuring accurate and efficient classification across varied datasets.

NOTE: We highly recommend to finetune the model on your dataset before deploying.

Inference samples

Inference typePython sample (Notebook)CLI with YAML
Real timemultimodal-classification-online-endpoint.ipynbmultimodal-classification-online-endpoint.sh
Batchmultimodal-classification-batch-endpoint.ipynbmultimodal-classification-batch-endpoint.sh

Finetuning samples

TaskDatasetPython sample (Notebook)CLI with YAML
Multimodal multi-class classificationAirbnb listings datasetmultimodal-multiclass-classification.ipynbmultimodal-multiclass-classification.sh
Multimodal multi-label classificationChest X-Rays datasetmultimodal-multilabel-classification.ipynbmultimodal-multilabel-classification.sh

Sample inputs and outputs (for real-time inference)

Sample input

{
 "input_data": {
        "columns": ["column1","column2","column3","column4","column5","column6"],
        "data": [[22,11.2,"It was a great experience!",image1,"Categorical value",True],
                 [111,8.2,"I may not consider this option again.",image2,"Categorical value",False]
                ]
     }
}

Note:

  • "image1", "image2" are strings in base64 format.

Sample output

[ 
     {
        "label1": 0.1,
        "label2": 0.7,
        "label3": 0.2
     }, 
     {
        "label1": 0.3,
        "label2": 0.3,
        "label3": 0.4
     },
] 

Specifications & capabilities

Provider specifications & access

1–1 / 1

mmeft · azureml v5 ↗

source model id
azureml/mmeft@5
hosting
foundry-managed-compute
platforms
  • Microsoft Foundry
license
mit
architecture
input_modalities
    output_modalities
      capabilities

        Original sources

        1–3 / 3

        Microsoft Foundry · Model page ↗

        Microsoft Foundry · Model documentation ↗

        Microsoft Foundry · Catalog ↗

        Source links

        1–1 / 1

        Explore more models

        Models

        Loading
        EVIDENCE / COMPARISON

        Benchmarks

        Explore every benchmark

        Loading

        These charts contain model-linked results. The charts above also include unlinked source profiles. Compare scores within each evaluation. Model quality, usage, transparency and system performance measure different things.

        Install TensorTrace