microsoft-deberta-xlarge · azureml v18

[DeBERTa](https://arxiv.org/abs/2006.03654) (Decoding-enhanced BERT with Disentangled Attention) improves the BERT and RoBERTa models using disentangled attention and enhanced mask decoder. With thos…

azureFoundry/azureml/microsoft-deberta-xlarge@18

DeBERTa (Decoding-enhanced BERT with Disentangled Attention) improves the BERT and RoBERTa models using disentangled attention and enhanced mask decoder. With those two improvements, DeBERTa out perform RoBERTa on a majority of NLU tasks with 80GB training data.

Please check the official repository for more details and updates.

This the DeBERTa XLarge model with 48 layers, 1024 hidden size. Total parameters 750M.

Evaluation Results

We present the dev results on SQuAD 1.1/2.0 and several GLUE benchmark tasks.

ModelSQuAD 1.1SQuAD 2.0MNLI-m/mmSST-2QNLICoLARTEMRPCQQPSTS-B
F1/EMF1/EMAccAccAccMCCAccAcc/F1Acc/F1P/S
BERT-Large90.9/84.181.8/79.086.6/-93.292.360.670.488.0/-91.3/-90.0/-
RoBERTa-Large94.6/88.989.4/86.590.2/-96.493.968.086.690.9/-92.2/-92.4/-
XLNet-Large95.1/89.790.6/87.990.8/-97.094.969.085.990.8/-92.3/-92.5/-
DeBERTa-Large195.5/90.190.7/88.091.3/91.196.595.369.591.092.6/94.692.3/-92.8/92.5
DeBERTa-XLarge1-/--/-91.5/91.297.0--93.192.1/94.3-92.9/92.7
DeBERTa-V2-XLarge195.8/90.891.4/88.991.7/91.697.595.871.193.992.0/94.292.3/89.892.9/92.9
DeBERTa-V2-XXLarge1,296.1/91.492.2/89.791.7/91.997.296.072.093.593.1/94.992.7/90.393.2/93.1

Model Evaluation samples

TaskUse caseDatasetPython sample (Notebook)CLI with YAML
Fill MaskFill Maskrcds/wikipedia-for-mask-fillingevaluate-model-fill-mask.ipynbevaluate-model-fill-mask.yml

Inference samples

Inference typePython sample (Notebook)
Real timesdk-example.ipynb
Real timefill-mask-online-endpoint.ipynb

Sample inputs and outputs

Sample input

{
    "input_data": [
        "Paris is the [MASK] of France.",
        "Today is a [MASK] day!"
    ]
}

Sample output

[
  "plex",
  "pron"
]

Specifications & capabilities

Provider specifications & access

1–1 / 1

microsoft-deberta-xlarge · azureml v18 ↗

source model id
azureml/microsoft-deberta-xlarge@18
hosting
foundry-managed-compute
platforms
  • Microsoft Foundry
license
mit
architecture
input_modalities
    output_modalities
      capabilities

        Original sources

        1–3 / 3

        Microsoft Foundry · Model page ↗

        Microsoft Foundry · Model documentation ↗

        Microsoft Foundry · Catalog ↗

        Source links

        1–1 / 1

        Explore more models

        Models

        Loading
        EVIDENCE / COMPARISON

        Benchmarks

        Explore every benchmark

        Loading

        These charts contain model-linked results. The charts above also include unlinked source profiles. Compare scores within each evaluation. Model quality, usage, transparency and system performance measure different things.

        Install TensorTrace