alt.freeweights.text / Florence-2-large

From:

microsoft <huggingface.co/microsoft>Newsgroups: alt.freeweights.textSubject:

Florence-2-large

Date: 2026-11-15Commit: 21a599dSigned: 2DC28CC7B26A283B ✓

Image text to text model with 777M parameters. Safetensors weights, 3.12 GB in all. Published by microsoft under MIT.

777M params

3.12GBFlorence-2-largesafetensorspermissive

FactsParams777MSize3.12 GBFormatssafetensorsTaskimage text to textLicenseMITFrom sourcepermissive

$ fw get microsoft/Florence-2-large
# manifest sha256 d48fb44d85a3… · 3.12 GB · 18 files
$ fw verify ./Florence-2-large microsoft/Florence-2-large

From sourceHosted (signed)microsoft/Florence-2-largeLast check: 2026-11-15 02:27 UTC

Fine-tune in Studio Get help

Get help: custom fine-tune or private deploy. We review your setup before we quote.

Runs with

AIBrix BentoML CTranslate2 ExLlamaV2 ExLlamaV3 exo fastllm FlashInfer KTransformers LitServe LMCache LMDeploy LocalAI mistral.rs MLC LLM Mooncake

SGLang Text Generation WebUI tinygrad

Transformers Triton Inference Server

vLLM XinferenceFine-tune with

Accelerate

Alignment Handbook

Axolotl Colossal-AI DeepSpeed EasyR1 FlashAttention H2O LLM Studio JAX Keras Liger Kernel LitGPT LLaMA-Factory Megatron-LM ms-swift

nanotron

OLMo-core OpenRLHF Oumi

PEFT PyTorch Lightning sd-scripts SimpleTuner TorchTitan torchtune Transformer Engine

TRL

Unsloth verlConvert withAQLM AutoRound Core ML Tools ExecuTorch GPTQModel

LLM Compressor mergekit Neural Compressor

Olive ONNX

Optimum Quanto

Optimum

safetensors TensorRT Model Optimizer torchaoEvaluate withAutoevals DeepEval

Evaluate HELM

LightEval lm-evaluation-harness OpenCompass Opik Ragas

simple-evals SWE-bench TruLens WeaveScan withAgentDojo AI Fairness 360 DeepTeam Detoxify Fairlearn garak Giskard Guardrails MCP Scanner ModelScan NeMo Guardrails picklescan promptfooWorks with

Agent Development Kit AG2 Google AI Edge Gallery Aider any-llm AnythingLLM BAML browser-use CAMEL

Hugging Face Chat UI Chatbox Cherry Studio Cline

Codex CLI Cognee CrewAI DeerFlow docTR DSPy EasyOCR Enchanted Gemini CLI GitHub MCP Server goose GPT Researcher Gradio Graphiti Guidance Hollama Instructor Kilo Code Langflow LangGraph LiteLLM LlamaIndex LLM LocalAGI LoLLMs WebUI MCP Python SDK Mem0 MetaGPT NextChat

Ollama JavaScript

Ollama Python

olmOCR

OpenAI Agents SDK opencode OpenHands oterm Outlines Page Assist

Playwright MCP Portkey AI Gateway PrivateGPT Qwen Code RWKV Runner

Semantic Kernel

smolagents Strands Agents Surya Vane XGrammar

Overview Model card Files Use it License Provenance Model tree

Overview

Size: 3.12 GB
Weights: 1.55 GB (safetensors)
Parameters: 777M
Formats: safetensors
Task: image text to text
License: MIT (permissive)
Files: 18, each with a sha256
Publisher: microsoft
Added: 2026-11-15
Section: alt.freeweights.text

Lineage

Original release from the publisher.

Where these bytes come from

Publisher at the pinned commit→You

Sources are tried in order. A file counts only when its size and sha256 match the manifest. Sources are checked when you ask for them, not on a timer.

Sources

Badge	Host	Last check	Result
From source	huggingface.co	2026-11-15 02:27 UTC	passed

Model card

From the publisher at commit 21a599d. Shown as plain text with links; images open at the publisher.

Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks#

Model Summary#

This is a continued pretrained version of Florence-2-large model with 4k context length, only 0.1B samples are used for continue pretraining, thus it might not be trained well. In addition, OCR task has been updated with line separator ('\n'). COCO OD AP 39.8

This Hub repository contains a HuggingFace's transformers implementation of Florence-2 model from Microsoft.

Florence-2 is an advanced vision foundation model that uses a prompt-based approach to handle a wide range of vision and vision-language tasks. Florence-2 can interpret simple text prompts to perform tasks like captioning, object detection, and segmentation. It leverages our FLD-5B dataset, containing 5.4 billion annotations across 126 million images, to master multi-task learning. The model's sequence-to-sequence architecture enables it to excel in both zero-shot and fine-tuned settings, proving to be a competitive vision foundation model.

Resources and Technical Documentation:

Model	Model size	Model Description
Florence-2-base[[HF]](https://huggingface.co/microsoft/Florence-2-base)	0.23B	Pretrained model with FLD-5B
Florence-2-large[[HF]](https://huggingface.co/microsoft/Florence-2-large)	0.77B	Pretrained model with FLD-5B
Florence-2-base-ft[[HF]](https://huggingface.co/microsoft/Florence-2-base-ft)	0.23B	Finetuned model on a colletion of downstream tasks
Florence-2-large-ft[[HF]](https://huggingface.co/microsoft/Florence-2-large-ft)	0.77B	Finetuned model on a colletion of downstream tasks

How to Get Started with the Model#

Use the code below to get started with the model. All models are trained with float16.

import requests

import torch
from PIL import Image
from transformers import AutoProcessor, AutoModelForCausalLM 


device = "cuda:0" if torch.cuda.is_available() else "cpu"
torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32

model = AutoModelForCausalLM.from_pretrained("microsoft/Florence-2-large", torch_dtype=torch_dtype, trust_remote_code=True).to(device)
processor = AutoProcessor.from_pretrained("microsoft/Florence-2-large", trust_remote_code=True)

prompt = "<OD>"

url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg?download=true"
image = Image.open(requests.get(url, stream=True).raw)

inputs = processor(text=prompt, images=image, return_tensors="pt").to(device, torch_dtype)

generated_ids = model.generate(
    input_ids=inputs["input_ids"],
    pixel_values=inputs["pixel_values"],
    max_new_tokens=4096,
    num_beams=3,
    do_sample=False
)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=False)[0]

parsed_answer = processor.post_process_generation(generated_text, task="<OD>", image_size=(image.width, image.height))

print(parsed_answer)

Tasks#

This model is capable of performing different tasks through changing the prompts.

First, let's define a function to run a prompt.

Click to expand

import requests

import torch
from PIL import Image
from transformers import AutoProcessor, AutoModelForCausalLM 

device = "cuda:0" if torch.cuda.is_available() else "cpu"
torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32

model = AutoModelForCausalLM.from_pretrained("microsoft/Florence-2-large", torch_dtype=torch_dtype, trust_remote_code=True).to(device)
processor = AutoProcessor.from_pretrained("microsoft/Florence-2-large", trust_remote_code=True)

url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg?download=true"
image = Image.open(requests.get(url, stream=True).raw)

def run_example(task_prompt, text_input=None):
    if text_input is None:
        prompt = task_prompt
    else:
        prompt = task_prompt + text_input
    inputs = processor(text=prompt, images=image, return_tensors="pt").to(device, torch_dtype)
    generated_ids = model.generate(
      input_ids=inputs["input_ids"],
      pixel_values=inputs["pixel_values"],
      max_new_tokens=1024,
      num_beams=3
    )
    generated_text = processor.batch_decode(generated_ids, skip_special_tokens=False)[0]

    parsed_answer = processor.post_process_generation(generated_text, task=task_prompt, image_size=(image.width, image.height))

    print(parsed_answer)

Here are the tasks Florence-2 could perform:

Click to expand

Caption#

prompt = "<CAPTION>"
run_example(prompt)

Detailed Caption#

prompt = "<DETAILED_CAPTION>"
run_example(prompt)

More Detailed Caption#

prompt = "<MORE_DETAILED_CAPTION>"
run_example(prompt)

Caption to Phrase Grounding#

caption to phrase grounding task requires additional text input, i.e. caption.

Caption to phrase grounding results format: {'\<CAPTION_TO_PHRASE_GROUNDING>': {'bboxes': [[x1, y1, x2, y2], ...], 'labels': ['', '', ...]}}

task_prompt = "<CAPTION_TO_PHRASE_GROUNDING>"
results = run_example(task_prompt, text_input="A green car parked in front of a yellow building.")

Object Detection#

OD results format: {'\': {'bboxes': [[x1, y1, x2, y2], ...], 'labels': ['label1', 'label2', ...]} }

prompt = "<OD>"
run_example(prompt)

Dense Region Caption#

Dense region caption results format: {'\<DENSE_REGION_CAPTION>' : {'bboxes': [[x1, y1, x2, y2], ...], 'labels': ['label1', 'label2', ...]} }

prompt = "<DENSE_REGION_CAPTION>"
run_example(prompt)

Region proposal#

Dense region caption results format: {'\<REGION_PROPOSAL>': {'bboxes': [[x1, y1, x2, y2], ...], 'labels': ['', '', ...]}}

prompt = "<REGION_PROPOSAL>"
run_example(prompt)

OCR#

prompt = "<OCR>"
run_example(prompt)

OCR with Region#

OCR with region output format: {'\<OCR_WITH_REGION>': {'quad_boxes': [[x1, y1, x2, y2, x3, y3, x4, y4], ...], 'labels': ['text1', ...]}}

prompt = "<OCR_WITH_REGION>"
run_example(prompt)

Output confidence score with Object Detection#


def run_example_with_score(task_prompt, text_input=None):
    if text_input is None:
        prompt = task_prompt
    else:
        prompt = task_prompt + text_input
    inputs = processor(text=prompt, images=image, return_tensors="pt").to(device, torch_dtype)
    generated_ids = model.generate(
      input_ids=inputs["input_ids"],
      pixel_values=inputs["pixel_values"],
      max_new_tokens=1024,
      num_beams=3,
      return_dict_in_generate=True,
      output_scores=True,
    )
    generated_text = processor.batch_decode(generated_ids.sequences, skip_special_tokens=False)[0]

    prediction, scores, beam_indices = generated_ids.sequences, generated_ids.scores, generated_ids.beam_indices
    transition_beam_scores = model.compute_transition_scores(
        sequences=prediction,
        scores=scores,
        beam_indices=beam_indices,
    )

    parsed_answer = processor.post_process_generation(sequence=generated_ids.sequences[0], 
        transition_beam_score=transition_beam_scores[0],
        task=task_prompt, image_size=(image.width, image.height)
    )

    print(parsed_answer)

prompt = "<OD>"
run_example_with_score(prompt)

for More detailed examples, please refer to notebook

Benchmarks#

Florence-2 Zero-shot performance#

The following table presents the zero-shot performance of generalist vision foundation models on image captioning and object detection evaluation tasks. These models have not been exposed to the training data of the evaluation tasks during their training phase.

Method	#params	COCO Cap. test CIDEr	NoCaps val CIDEr	TextCaps val CIDEr	COCO Det. val2017 mAP
Flamingo	80B	84.3	-	-	-
Florence-2-base	0.23B	133.0	118.7	70.1	34.7
Florence-2-large	0.77B	135.6	120.8	72.8	37.5

The following table continues the comparison with performance on other vision-language evaluation tasks.

Method	Flickr30k test R@1	Refcoco val Accuracy	Refcoco test-A Accuracy	Refcoco test-B Accuracy	Refcoco+ val Accuracy	Refcoco+ test-A Accuracy	Refcoco+ test-B Accuracy	Refcocog val Accuracy	Refcocog test Accuracy	Refcoco RES val mIoU
Kosmos-2	78.7	52.3	57.4	47.3	45.5	50.7	42.2	60.6	61.7	-
Florence-2-base	83.6	53.9	58.4	49.7	51.5	56.4	47.9	66.3	65.1	34.6
Florence-2-large	84.4	56.3	61.6	51.4	53.6	57.9	49.9	68.0	67.0	35.8

Florence-2 finetuned performance#

We finetune Florence-2 models with a collection of downstream tasks, resulting two generalist models Florence-2-base-ft and Florence-2-large-ft that can conduct a wide range of downstream tasks.

The table below compares the performance of specialist and generalist models on various captioning and Visual Question Answering (VQA) tasks. Specialist models are fine-tuned specifically for each task, whereas generalist models are fine-tuned in a task-agnostic manner across all tasks. The symbol "▲" indicates the usage of external OCR as input.

Method	# Params	COCO Caption Karpathy test CIDEr	NoCaps val CIDEr	TextCaps val CIDEr	VQAv2 test-dev Acc	TextVQA test-dev Acc	VizWiz VQA test-dev Acc
Specialist Models
CoCa	2.1B	143.6	122.4	-	82.3	-	-
BLIP-2	7.8B	144.5	121.6	-	82.2	-	-
GIT2	5.1B	145.0	126.9	148.6	81.7	67.3	71.0
Flamingo	80B	138.1	-	-	82.0	54.1	65.7
PaLI	17B	149.1	127.0	160.0▲	84.3	58.8 / 73.1▲	71.6 / 74.4▲
PaLI-X	55B	149.2	126.3	147.0 / 163.7▲	86.0	71.4 / 80.8▲	70.9 / 74.6▲
Generalist Models
Unified-IO	2.9B	-	100.0	-	77.9	-	57.4
Florence-2-base-ft	0.23B	140.0	116.7	143.9	79.7	63.6	63.6
Florence-2-large-ft	0.77B	143.3	124.9	151.1	81.7	73.5	72.6

Method	# Params	COCO Det. val2017 mAP	Flickr30k test R@1	RefCOCO val Accuracy	RefCOCO test-A Accuracy	RefCOCO test-B Accuracy	RefCOCO+ val Accuracy	RefCOCO+ test-A Accuracy	RefCOCO+ test-B Accuracy	RefCOCOg val Accuracy	RefCOCOg test Accuracy	RefCOCO RES val mIoU
Specialist Models
SeqTR	-	-	-	83.7	86.5	81.2	71.5	76.3	64.9	74.9	74.2	-
PolyFormer	-	-	-	90.4	92.9	87.2	85.0	89.8	78.0	85.8	85.9	76.9
UNINEXT	0.74B	60.6	-	92.6	94.3	91.5	85.2	89.6	79.8	88.7	89.4	-
Ferret	13B	-	-	89.5	92.4	84.4	82.8	88.1	75.2	85.8	86.3	-
Generalist Models
UniTAB	-	-	-	88.6	91.1	83.8	81.0	85.4	71.6	84.6	84.7	-
Florence-2-base-ft	0.23B	41.4	84.0	92.6	94.8	91.5	86.8	91.7	82.2	89.8	82.2	78.0
Florence-2-large-ft	0.77B	43.4	85.2	93.4	95.3	92.0	88.3	92.9	83.6	91.2	91.7	80.5

BibTex and citation info#

@article{xiao2023florence,
  title={Florence-2: Advancing a unified representation for a variety of vision tasks},
  author={Xiao, Bin and Wu, Haiping and Xu, Weijian and Dai, Xiyang and Hu, Houdong and Lu, Yumao and Zeng, Michael and Liu, Ce and Yuan, Lu},
  journal={arXiv preprint arXiv:2311.06242},
  year={2023}
}

Files

18 files · 3.12 GBWeights 1.55 GB (safetensors). Sizes and sha256 from the signed manifest.

Hide all 18 files with their sha256

File	Size	sha256
.gitattributes	1.56 KB	28aac17b0f70983ef58c18a7ee355503e454ce3178fce77a7b50503906df0635
CODE_OF_CONDUCT.md	444 B	9daeae709a0bd71bcfd1c96dc5822ecec5210327eff929da64b0ae7f8faf1444
LICENSE	1.14 KB	c2cfccb812fe482101a8f04597dfc5a9991a6b2748266c47ac91b6a5aae15383
README.md	16.3 KB	16212f1929e4992649d18b812e0ebf7ebbbaffd90a5ba23dfdc5c78df2f92ac2
SECURITY.md	2.66 KB	7b6976eec43edfa68b79a459dd089c56b7a395916dbf1a01bd11e6d86e12128f
SUPPORT.md	1.24 KB	149773411caf2c59cb1a6bfb2c6f070d1d054f1f018e7a80be319a8c384eeaf5
config.json	2.44 KB	6f8a8f92a74ce18b5c1e5646b4a8222477dd1ac49f31b953e75d6fc3e0f8583a
configuration_florence2.py	15.1 KB	de2e45a975b3582de05d2f4d963a3e9f9a3d20dccf78d28e0052932a0be93bdf
generation_config.json	51 B	30e9865458ecc8ee931eeeb43f44f1d169c5ab95be39e0072142a7a6b8f31990
model.safetensors	1.55 GB	4f38ce741c6b71188fe2b3419a55e11917a8a7b321ae2e63c61da0191b0ebad7
modeling_florence2.py	127 KB	5162bf465e61b6e29cc113a467630ec3cb56ed8e4d46eb6207157f10fb9b8a24
preprocessor_config.json	806 B	2f5921bbc53c7cc04251e1027b45b1cec726276be6db23d1bb40641bfbe2cf29
processing_florence2.py	48.7 KB	c655782a9e4347965c735ea54cbc4e98fdbc02155ffd1ce2ecd61f42c45eda28
pytorch_model.bin	1.56 GB	8b7d99c2ca930af3bcc4625df55c82b6bb372456280310b5189c519d6083a270
sample_inference.ipynb	5.71 MB	1dabc0f882f7f14752dec6c74404e516d99faf4db3e8bb93cd44a453d0981365
tokenizer.json	1.36 MB	847bbeab6174d66a88898f729d52fa8d355fafe1bea101cf960dd404581df70e
tokenizer_config.json	34 B	79ffcf43af8ebda99d165f61d243180da2e2639952e41e71e11611c18770489c
vocab.json	1.10 MB	394fdc63c71aabe0a9b97117f5d62fb5fcc4d59b2b3ea929a3929e6a53217b3c

VersionsEach time the pinned upstream commit or file list changed in this repo. Newest first.

Upstream commit	Date	Contents
21a599d414 pinned now	2026-11-15 02:27 UTC	18 files · 3.12 GB

Use it

$ fw get microsoft/Florence-2-large
# manifest sha256 d48fb44d85a3… · 3.12 GB · 18 files
$ fw verify ./Florence-2-large microsoft/Florence-2-large

fw.pyChecks the signature and every sha256, tries each source in order, resumes.

$ python3 fw.py get https://freeweights.allternit.com/manifests/microsoft/Florence-2-large.json --dir ./Florence-2-large
$ python3 fw.py verify ./Florence-2-large ./microsoft__Florence-2-large.json

hfThe Hugging Face CLI with one setting. Same repo id.

$ HF_ENDPOINT=https://freeweights.allternit.com hf download microsoft/Florence-2-large --revision 21a599d414c4d928c9032694c424fb94458e3594 --local-dir ./Florence-2-large

transformersLoad the weights in Python.

import os
os.environ["HF_ENDPOINT"] = "https://freeweights.allternit.com"
from transformers import AutoModelForImageTextToText, AutoProcessor
processor = AutoProcessor.from_pretrained("microsoft/Florence-2-large", revision="21a599d414c4d928c9032694c424fb94458e3594")
model = AutoModelForImageTextToText.from_pretrained("microsoft/Florence-2-large", revision="21a599d414c4d928c9032694c424fb94458e3594", device_map="auto")

vLLMServe an OpenAI-compatible API.

$ HF_ENDPOINT=https://freeweights.allternit.com vllm serve microsoft/Florence-2-large --revision 21a599d414c4d928c9032694c424fb94458e3594

Fine-tune in Studio

License

Name: MIT License
Id: MIT
Class: Permissive
Redistribution: yes
Listing: Hosted (signed)
Decided: License as declared by the publisher. No person read it. 2026-11-15
Text in: LICENSE
Text sha256: c2cfccb812fe482101a8f04597dfc5a9991a6b2748266c47ac91b6a5aae15383
Terms gate: No extra terms beyond the license.

Read the license text

Licenses are checked against the license table. A custom license passes only after a person reads it.

Provenance

From:

microsoft <huggingface.co/microsoft>Newsgroups: alt.freeweights.textSubject: Florence-2-largeDate: 2026-11-15Commit: 21a599dSigned: 2DC28CC7B26A283B ✓

Upstream: microsoft/Florence-2-large
Pinned commit: 21a599d414c4d928c9032694c424fb94458e3594
Manifest: Florence-2-large.json · .minisig
Manifest sha256: d48fb44d85a3b3980bc120fcf8bbf9bffef5d31e137f51bbcc95a6b25e8333a3
Signature: minisign, Ed25519, key 2DC28CC7B26A283B
Created: 2026-11-15 02:27 UTC
Generator: freeweights-manifest/0.1.0
Takedowns: None.

How to verify

$ curl -O https://freeweights.allternit.com/manifests/microsoft/Florence-2-large.json
$ curl -O https://freeweights.allternit.com/manifests/microsoft/Florence-2-large.json.minisig
$ minisign -Vm Florence-2-large.json -P RWQ7KGqyx4zCLXtUFOqvfv8L3g+8fpKJ1FsuxPyOYMF+4KufbUxKq1Oq
$ shasum -a 256 <file>   # compare with the Files tab

A manifest whose signature fails must be rejected. A file whose size or sha256 does not match must be discarded. Verify a download

Model tree

1 model in the catalog name this one as their parent.

Quantizations: 1

Quantizations 1

onnx-community/Florence-2-large14.8 GB

Get it

Download

$ HF_ENDPOINT=https://freeweights.allternit.com hf download microsoft/Florence-2-large

Open in Freeweights Studio

Pull, verify and run it in the Allternit app on Desktop, web or phone.

Signed manifest

minisign, key 2DC28CC7B26A283B. Florence-2-large.json. Provenance. Takedown history: none.

Source

huggingface.co/microsoft/Florence-2-large

Variants

1 quantized or adapter build of Florence-2-large. Each has its own license check and page.

SubjectSizeParamsLicenseDate

Florence-2-largeHosted (signed) From: onnx-community in alt.freeweights.text14.8 GB—Permissive2026-11-15

Related models

All of alt.freeweights.text →

From: laion <huggingface.co/laion>Subject: a2-rl-e2egit_large-50-32B

From: laion <huggingface.co/laion>Subject: a3-rl-DCAgent_exp_rpt_e2egit-large-15-8B

From: laion <huggingface.co/laion>Subject: a3-rl-DCAgent_exp_rpt_e2egit-v2-10-8B

From: laion <huggingface.co/laion>Subject: a3-rl-DCAgent_exp_rpt_pymethods2test-large-80-8B

From: laion <huggingface.co/laion>Subject: a3-rl-DCAgent_exp_rpt_unitsyn-python-large-10-8B

From: laion <huggingface.co/laion>Subject: a3-rl-DCAgent_mix_h2_language_proportional-65-8B