microsoft/OmniParser-v2.0

📢 [[GitHub Repo](https://github.com/microsoft/OmniParser/tree/master)] [[OmniParser V2 Blog Post](https://www.microsoft.com/en-us/research/articles/omniparser-v2-turning-any-llm-into-a-computer-use-a…

azureFoundry/HuggingFace/microsoft-omniparser-v2-0@3

📢 [GitHub Repo] [OmniParser V2 Blog Post] Huggingface demo

Model Summary

OmniParser is a general screen parsing tool, which interprets/converts UI screenshot to structured format, to improve existing LLM based UI agent. Training Datasets include: 1) an interactable icon detection dataset, which was curated from popular web pages and automatically annotated to highlight clickable and actionable regions, and 2) an icon description dataset, designed to associate each UI element with its corresponding function.

This model hub includes a finetuned version of YOLOv8 and a finetuned Florence-2 base model on the above dataset respectively. For more details of the models used and finetuning, please refer to the paper.

What's new in V2?

  • Larger and cleaner set of icon caption + grounding dataset
  • 60% improvement in latency compared to V1. Avg latency: 0.6s/frame on A100, 0.8s on single 4090.
  • Strong performance: 39.6 average accuracy on ScreenSpot Pro
  • Your agent only need one tool: OmniTool. Control a Windows 11 VM with OmniParser + your vision model of choice. OmniTool supports out of the box the following large language models - OpenAI (4o/o1/o3-mini), DeepSeek (R1), Qwen (2.5VL) or Anthropic Computer Use. Check out our github repo for details.

Responsible AI Considerations

Intended Use

  • OmniParser is designed to be able to convert unstructured screenshot image into structured list of elements including interactable regions location and captions of icons on its potential functionality.
  • OmniParser is intended to be used in settings where users are already trained on responsible analytic approaches and critical reasoning is expected. OmniParser is capable of providing extracted information from the screenshot, however human judgement is needed for the output of OmniParser.
  • OmniParser is intended to be used on various screenshots, which includes both PC and Phone, and also on various applications.

limitations

  • OmniParser is designed to faithfully convert screenshot image into structured elements of interactable regions and semantics of the screen, while it does not detect harmful content in its input (like users have freedom to decide the input of any LLMs), users are expected to provide input to the OmniParser that is not harmful.
  • While OmniParser only converts screenshot image into texts, it can be used to construct an GUI agent based on LLMs that is actionable. When developing and operating the agent using OmniParser, the developers need to be responsible and follow common safety standard.

License

Please note that icon_detect model is under AGPL license, and icon_caption is under MIT license. Please refer to the LICENSE file in the folder of each model.


microsoft/OmniParser-v2.0 powered by Hugging Face Inference Toolkit

Send Request

You can use cURL or any REST Client to send a request to the AzureML endpoint with your AzureML token.

curl <AZUREML_ENDPOINT_URL> \
    -X POST \
    -d '{"inputs":{"image": "https://raw.githubusercontent.com/microsoft/OmniParser/refs/heads/master/imgs/demo_image.jpg"}}' \
    -H "Authorization: Bearer <AZUREML_TOKEN>" \
    -H "Content-Type: application/json"

Supported Parameters

  • image_size: Dictionary with the keys "w" for the width, and "h" for the height, if the image wants to be resized. Defaults to the actual image width and height.
  • bbox_threshold: Threshold float value between 0.0 and 1.0, that defines the minimum value for an identified bounding box to be considered or discarded. Defaults to 0.05.
  • iou_threshold: Threshold float value between 0.0 and 1.0, that defines the minimum value for the intersection over union to be considered when evaluating overlapping bounding boxes

Example payload

{
  "image": "What is Deep Learning?",
  "image_size": {"w": 1024, "h": 1024},
  "bbox_threshold": 0.05,
  "iou_threshold": 0.7,
}

Specifications & capabilities

Published model ID
OmniParser-v2.0

Provider specifications & access

1–3 / 3

microsoft/OmniParser-v2.0 ↗

source model id
HuggingFace/microsoft-omniparser-v2-0@1
hosting
foundry-managed-compute
platforms
  • Microsoft Foundry
license
mit
architecture
input_modalities
    output_modalities
      capabilities

        microsoft/OmniParser-v2.0 ↗

        source model id
        HuggingFace/microsoft-omniparser-v2-0@2
        hosting
        foundry-managed-compute
        platforms
        • Microsoft Foundry
        license
        mit
        architecture
        input_modalities
          output_modalities
            capabilities

              microsoft/OmniParser-v2.0 ↗

              source model id
              HuggingFace/microsoft-omniparser-v2-0@3
              hosting
              foundry-managed-compute
              platforms
              • Microsoft Foundry
              license
              mit
              architecture
              input_modalities
                output_modalities
                  capabilities

                    Original sources

                    1–6 / 6

                    Hugging Face · microsoft/OmniParser-v2.0 ↗

                    Microsoft Foundry · Model page ↗

                    Microsoft Foundry · Model page ↗

                    Microsoft Foundry · Model page ↗

                    Microsoft Foundry · Model documentation ↗

                    Microsoft Foundry · Catalog ↗

                    Source links

                    1–1 / 1

                    Explore more models

                    Models

                    Loading
                    EVIDENCE / COMPARISON

                    Benchmarks

                    Explore every benchmark

                    Loading

                    These charts contain model-linked results. The charts above also include unlinked source profiles. Compare scores within each evaluation. Model quality, usage, transparency and system performance measure different things.

                    Install TensorTrace