CatalogWantedMirrorsLedgerOperating StatementDocsStudio statusCard system
alt.freeweights.text / OmniParser
From: microsoft <huggingface.co/microsoft>Newsgroups: alt.freeweights.textSubject:

OmniParser

Date: 2026-11-15Commit: 9a02350Signed: 2DC28CC7B26A283B ✓

Image text to text model. Safetensors weights, 8.62 GB in all. Published by microsoft under MIT.

text8.62GBOmniParsersafetensorspermissive
FactsParamsnot statedSize8.62 GBFormatssafetensorsTaskimage text to textLicenseMITFrom sourcepermissive
$ fw get microsoft/OmniParser
# manifest sha256 6eb8a8179a94… · 8.62 GB · 20 files
$ fw verify ./OmniParser microsoft/OmniParser
From sourceHosted (signed)microsoft/OmniParserLast check: 2026-11-15 03:36 UTC
Fine-tune in StudioRun at homeGet help

Get help: custom fine-tune or private deploy. We review your setup before we quote.

Runs withAIBrixBentoMLCTranslate2ExLlamaV2ExLlamaV3exofastllmFlashInferKTransformersLitServeLMCacheLMDeployLocalAImistral.rsMLC LLMMooncakeSGLangText Generation WebUItinygradTransformersTriton Inference ServervLLMXinferenceFine-tune withAccelerateAlignment HandbookAxolotlColossal-AIDeepSpeedEasyR1FlashAttentionH2O LLM StudioJAXKerasLiger KernelLitGPTLLaMA-FactoryMegatron-LMms-swiftnanotronOLMo-coreOpenRLHFOumiPEFTPyTorch Lightningsd-scriptsSimpleTunerTorchTitantorchtuneTransformer EngineTRLUnslothverlConvert withAQLMAutoRoundCore ML ToolsExecuTorchGPTQModelLLM CompressormergekitNeural CompressorOliveONNXOptimum QuantoOptimumsafetensorsTensorRT Model OptimizertorchaoEvaluate withAutoevalsDeepEvalEvaluateHELMLightEvallm-evaluation-harnessOpenCompassOpikRagassimple-evalsSWE-benchTruLensWeaveScan withAgentDojoAI Fairness 360DeepTeamDetoxifyFairlearngarakGiskardGuardrailsMCP ScannerModelScanNeMo GuardrailspicklescanpromptfooWorks withAgent Development KitAG2Google AI Edge GalleryAiderany-llmAnythingLLMBAMLbrowser-useCAMELHugging Face Chat UIChatboxCherry StudioClineCodex CLICogneeCrewAIDeerFlowdocTRDSPyEasyOCREnchantedGemini CLIGitHub MCP ServergooseGPT ResearcherGradioGraphitiGuidanceHollamaInstructorKilo CodeLangflowLangGraphLiteLLMLlamaIndexLLMLocalAGILoLLMs WebUIMCP Python SDKMem0MetaGPTNextChatOllama JavaScriptOllama PythonolmOCROpenAI Agents SDKopencodeOpenHandsotermOutlinesPage AssistPlaywright MCPPortkey AI GatewayPrivateGPTQwen CodeRWKV RunnerSemantic KernelsmolagentsStrands AgentsSuryaVaneXGrammar
OverviewModel cardFilesUse itLicenseProvenanceModel tree

Overview

Size
8.62 GB
Weights
1.09 GB (safetensors)
Parameters
not stated
Formats
safetensors
Task
image text to text
License
MIT (permissive)
Files
20, each with a sha256
Publisher
microsoft
Added
2026-11-15
Section
alt.freeweights.text
Lineage

Original release from the publisher.

Where these bytes come from
Publisher at the pinned commit→You

Sources are tried in order. A file counts only when its size and sha256 match the manifest. Sources are checked when you ask for them, not on a timer.

Sources
BadgeHostLast checkResult
From sourcehuggingface.co2026-11-15 03:36 UTCpassed

Model card

From the publisher at commit 9a02350. Shown as plain text with links; images open at the publisher.

📢 [Project Page] [Blog Post] [Demo]

Model Summary#

OmniParser is a general screen parsing tool, which interprets/converts UI screenshot to structured format, to improve existing LLM based UI agent. Training Datasets include: 1) an interactable icon detection dataset, which was curated from popular web pages and automatically annotated to highlight clickable and actionable regions, and 2) an icon description dataset, designed to associate each UI element with its corresponding function.

This model hub includes a finetuned version of YOLOv8 and a finetuned BLIP-2 model on the above dataset respectively. For more details of the models used and finetuning, please refer to the paper.

Responsible AI Considerations#

Intended Use#

  • OmniParser is designed to be able to convert unstructured screenshot image into structured list of elements including interactable regions location and captions of icons on its potential functionality.
  • OmniParser is intended to be used in settings where users are already trained on responsible analytic approaches and critical reasoning is expected. OmniParser is capable of providing extracted information from the screenshot, however human judgement is needed for the output of OmniParser.
  • OmniParser is intended to be used on various screenshots, which includes both PC and Phone, and also on various applications.

limitations#

  • OmniParser is designed to faithfully convert screenshot image into structured elements of interactable regions and semantics of the screen, while it does not detect harmful content in its input (like users have freedom to decide the input of any LLMs), users are expected to provide input to the OmniParser that is not harmful.
  • While OmniParser only converts screenshot image into texts, it can be used to construct an GUI agent based on LLMs that is actionable. When developing and operating the agent using OmniParser, the developers need to be responsible and follow common safety standard.
  • For OmniPaser-BLIP2, it may incorrectly infer the gender or other sensitive attribute (e.g., race, religion etc.) of individuals in icon images. Inference of sensitive attributes may rely upon stereotypes and generalizations rather than information about specific individuals and are more likely to be incorrect for marginalized people. Incorrect inferences may result in significant physical or psychological injury or restrict, infringe upon or undermine the ability to realize an individual’s human rights. We do not recommend use of OmniParser in any workplace-like use case scenario.

License#

Please note that icon_detect model is under AGPL license, and icon_caption_blip2 & icon_caption_florence is under MIT license. Please refer to the LICENSE file in the folder of each model.

Files

20 files · 8.62 GBWeights 1.09 GB (safetensors). Sizes and sha256 from the signed manifest.
Hide all 20 files with their sha256
FileSizesha256
.gitattributes1.52 KB11ad7efa24975ee4b0c3c3a38ed18737f0658a5f75a0a96787b576a78a023361
README.md3.10 KB6d18ebc72b9c33a465b81769e7874f409713238b61092f2dc8f3c01fb6731967
config.json985 B01a7eeeaa2668fd4af6398130c093971e2d21e0287a021873a627748f660bd03
icon_caption_blip2/LICENSE1.14 KBd9a1b1e30d633d5732ea18e3cba9538d293ebc53e1a9e4e96ab739e0c5c4f1cb
icon_caption_blip2/config.json985 B01a7eeeaa2668fd4af6398130c093971e2d21e0287a021873a627748f660bd03
icon_caption_blip2/generation_config.json136 B368cc378b4c0aa87b138440da9c4998aefbc2ed400e8a8e33edf7c2f28a79d08
icon_caption_blip2/pytorch_model-00001-of-00002.bin5.00 GB7d7c9438dc62272d956079e7bf54a9bfc888dd8337841f3f67ea813a4a21a957
icon_caption_blip2/pytorch_model-00002-of-00002.bin2.49 GBb7b1fd628d150593ba14b937fac61c2b41529a21bf991c227f828113712813d0
icon_caption_blip2/pytorch_model.bin.index.json122 KB0289144f33d99c85611344c1d198dd96d669f6f1a173b7410d5bdec26915a145
icon_caption_florence/LICENSE1.14 KBd9a1b1e30d633d5732ea18e3cba9538d293ebc53e1a9e4e96ab739e0c5c4f1cb
icon_caption_florence/config.json5.66 KBe007cec0964325bd954d27a5630b6014887c883031e4d40efad248f71c570368
icon_caption_florence/generation_config.json292 B14ab95a544c32907ce4ddb5dfb905634a55be9b50d1300ac89004674427563eb
icon_caption_florence/model.safetensors1.08 GBc0d7a6bedf8c6dbabe7d40b9f78ada36e78c1a93617506bc06a93279a78dfb14
icon_detect/LICENSE34.5 KB4df3c306dddaaf4baffdff5ca820cc679ac8cd6dc263c6a74517783e42fa7a3b
icon_detect/model.safetensors6.08 MB84caa6c7e0607d7d6edddb46f2affd0d0d86ddf34e71fbcce7b4ba461bd97574
icon_detect/model.yaml1.09 KBadb1fbac524d6dac8da049ed4bef180e180d4a056f8fcad459c93fe0cb2a54d9
icon_detect_v1_5/LICENSE34.5 KB4df3c306dddaaf4baffdff5ca820cc679ac8cd6dc263c6a74517783e42fa7a3b
icon_detect_v1_5/model.yaml1.14 KB0edced8dfb9c619ca187cf0f84139d63b4cff9011890a9bc1e9bd6bb08f43b8a
icon_detect_v1_5/model_v1_5.pt40.6 MBdab3d4351ad00b035db829909a4db98354d5a90f6990e4ac00222a9a95d4bf57
icon_detect_v1_5/train_args.yaml1.69 KB6acc8dcdc8a38ccfafc47ccfdb0087b8545eab6f9d9a373c0504bbb5a45a0277
VersionsEach time the pinned upstream commit or file list changed in this repo. Newest first.
Upstream commitDateContents
9a02350f93 pinned now2026-11-15 03:36 UTC20 files · 8.62 GB

Use it

$ fw get microsoft/OmniParser
# manifest sha256 6eb8a8179a94… · 8.62 GB · 20 files
$ fw verify ./OmniParser microsoft/OmniParser
fw.pyChecks the signature and every sha256, tries each source in order, resumes.
$ python3 fw.py get https://freeweights.allternit.com/manifests/microsoft/OmniParser.json --dir ./OmniParser $ python3 fw.py verify ./OmniParser ./microsoft__OmniParser.json
hfThe Hugging Face CLI with one setting. Same repo id.
$ HF_ENDPOINT=https://freeweights.allternit.com hf download microsoft/OmniParser --revision 9a02350f935534e9c9c29d0e6f28e4d1aff36a5c --local-dir ./OmniParser
transformersLoad the weights in Python.
import os os.environ["HF_ENDPOINT"] = "https://freeweights.allternit.com" from transformers import AutoModelForImageTextToText, AutoProcessor processor = AutoProcessor.from_pretrained("microsoft/OmniParser", revision="9a02350f935534e9c9c29d0e6f28e4d1aff36a5c") model = AutoModelForImageTextToText.from_pretrained("microsoft/OmniParser", revision="9a02350f935534e9c9c29d0e6f28e4d1aff36a5c", device_map="auto")
vLLMServe an OpenAI-compatible API.
$ HF_ENDPOINT=https://freeweights.allternit.com vllm serve microsoft/OmniParser --revision 9a02350f935534e9c9c29d0e6f28e4d1aff36a5c
Run at homeFine-tune in Studio

License

Name
MIT License
Id
MIT
Class
Permissive
Redistribution
yes
Listing
Hosted (signed)
Decided
License as declared by the publisher. No person read it. 2026-11-15
Text in
README.md
Text sha256
6d18ebc72b9c33a465b81769e7874f409713238b61092f2dc8f3c01fb6731967
Terms gate
No extra terms beyond the license.

Read the license text

Licenses are checked against the license table. A custom license passes only after a person reads it.

Provenance

From: microsoft <huggingface.co/microsoft>Newsgroups: alt.freeweights.textSubject: OmniParserDate: 2026-11-15Commit: 9a02350Signed: 2DC28CC7B26A283B ✓
Upstream
microsoft/OmniParser
Pinned commit
9a02350f935534e9c9c29d0e6f28e4d1aff36a5c
Manifest
OmniParser.json · .minisig
Manifest sha256
6eb8a8179a947eea84fceacd420a17373a4d6fa98a4ad5b2a621a9583a8ff9ad
Signature
minisign, Ed25519, key 2DC28CC7B26A283B
Created
2026-11-15 03:36 UTC
Generator
freeweights-manifest/0.1.0
Takedowns
None.

How to verify

$ curl -O https://freeweights.allternit.com/manifests/microsoft/OmniParser.json $ curl -O https://freeweights.allternit.com/manifests/microsoft/OmniParser.json.minisig $ minisign -Vm OmniParser.json -P RWQ7KGqyx4zCLXtUFOqvfv8L3g+8fpKJ1FsuxPyOYMF+4KufbUxKq1Oq $ shasum -a 256 <file> # compare with the Files tab

A manifest whose signature fails must be rejected. A file whose size or sha256 does not match must be discarded. Verify a download

Model tree

No related models in this catalog.Nothing here lists this model as its parent, and it names no parent.
Get it
Download
$ HF_ENDPOINT=https://freeweights.allternit.com hf download microsoft/OmniParser
Open in Freeweights Studio

Pull, verify and run it in the Allternit app on Desktop, web or phone.

Signed manifest
minisign, key 2DC28CC7B26A283B. OmniParser.json. Provenance. Takedown history: none.
Source
huggingface.co/microsoft/OmniParser

Related models

All of alt.freeweights.text →
From: laion <huggingface.co/laion>Subject: a2-rl-e2egit_large-50-32B
From: laion <huggingface.co/laion>Subject: a3-rl-DCAgent_exp_rpt_e2egit-large-15-8B
From: laion <huggingface.co/laion>Subject: a3-rl-DCAgent_exp_rpt_e2egit-v2-10-8B
From: laion <huggingface.co/laion>Subject: a3-rl-DCAgent_exp_rpt_pymethods2test-large-80-8B
From: laion <huggingface.co/laion>Subject: a3-rl-DCAgent_exp_rpt_unitsyn-python-large-10-8B
From: laion <huggingface.co/laion>Subject: a3-rl-DCAgent_mix_h2_language_proportional-65-8B