alt.freeweights.text / Florence-2-large-ft

From:

microsoft <huggingface.co/microsoft>Newsgroups: alt.freeweights.textSubject:

Florence-2-large-ft

Date: 2026-11-15Commit: 4a12a2bSigned: 2DC28CC7B26A283B ✓

Image text to text model with 770M parameters. Safetensors weights, 3.09 GB in all. Published by microsoft under MIT.

770M params

3.09GBFlorence-2-large-ftsafetensorspermissive

FactsParams770MSize3.09 GBFormatssafetensorsTaskimage text to textLicenseMITFrom sourcepermissive

$ fw get microsoft/Florence-2-large-ft
# manifest sha256 66c7e61a04a6… · 3.09 GB · 17 files
$ fw verify ./Florence-2-large-ft microsoft/Florence-2-large-ft

From sourceHosted (signed)microsoft/Florence-2-large-ftLast check: 2026-11-15 02:36 UTC

Fine-tune in Studio Get help

Get help: custom fine-tune or private deploy. We review your setup before we quote.

Runs with

AIBrix BentoML CTranslate2 ExLlamaV2 ExLlamaV3 exo fastllm FlashInfer KTransformers LitServe LMCache LMDeploy LocalAI mistral.rs MLC LLM Mooncake

SGLang Text Generation WebUI tinygrad

Transformers Triton Inference Server

vLLM XinferenceFine-tune with

Accelerate

Alignment Handbook

Axolotl Colossal-AI DeepSpeed EasyR1 FlashAttention H2O LLM Studio JAX Keras Liger Kernel LitGPT LLaMA-Factory Megatron-LM ms-swift

nanotron

OLMo-core OpenRLHF Oumi

PEFT PyTorch Lightning sd-scripts SimpleTuner TorchTitan torchtune Transformer Engine

TRL

Unsloth verlConvert withAQLM AutoRound Core ML Tools ExecuTorch GPTQModel

LLM Compressor mergekit Neural Compressor

Olive ONNX

Optimum Quanto

Optimum

safetensors TensorRT Model Optimizer torchaoEvaluate withAutoevals DeepEval

Evaluate HELM

LightEval lm-evaluation-harness OpenCompass Opik Ragas

simple-evals SWE-bench TruLens WeaveScan withAgentDojo AI Fairness 360 DeepTeam Detoxify Fairlearn garak Giskard Guardrails MCP Scanner ModelScan NeMo Guardrails picklescan promptfooWorks with

Agent Development Kit AG2 Google AI Edge Gallery Aider any-llm AnythingLLM BAML browser-use CAMEL

Hugging Face Chat UI Chatbox Cherry Studio Cline

Codex CLI Cognee CrewAI DeerFlow docTR DSPy EasyOCR Enchanted Gemini CLI GitHub MCP Server goose GPT Researcher Gradio Graphiti Guidance Hollama Instructor Kilo Code Langflow LangGraph LiteLLM LlamaIndex LLM LocalAGI LoLLMs WebUI MCP Python SDK Mem0 MetaGPT NextChat

Ollama JavaScript

Ollama Python

olmOCR

OpenAI Agents SDK opencode OpenHands oterm Outlines Page Assist

Playwright MCP Portkey AI Gateway PrivateGPT Qwen Code RWKV Runner

Semantic Kernel

smolagents Strands Agents Surya Vane XGrammar

Overview Model card Files Use it License Provenance Model tree

Overview

Size: 3.09 GB
Weights: 1.54 GB (safetensors)
Parameters: 770M
Formats: safetensors
Task: image text to text
License: MIT (permissive)
Files: 17, each with a sha256
Publisher: microsoft
Added: 2026-11-15
Section: alt.freeweights.text

Lineage

Original release from the publisher.

Where these bytes come from

Publisher at the pinned commit→You

Sources are tried in order. A file counts only when its size and sha256 match the manifest. Sources are checked when you ask for them, not on a timer.

Sources

Badge	Host	Last check	Result
From source	huggingface.co	2026-11-15 02:36 UTC	passed

Model card

From the publisher at commit 4a12a2b. Shown as plain text with links; images open at the publisher.

Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks#

Model Summary#

This Hub repository contains a HuggingFace's transformers implementation of Florence-2 model from Microsoft.

Florence-2 is an advanced vision foundation model that uses a prompt-based approach to handle a wide range of vision and vision-language tasks. Florence-2 can interpret simple text prompts to perform tasks like captioning, object detection, and segmentation. It leverages our FLD-5B dataset, containing 5.4 billion annotations across 126 million images, to master multi-task learning. The model's sequence-to-sequence architecture enables it to excel in both zero-shot and fine-tuned settings, proving to be a competitive vision foundation model.

Resources and Technical Documentation:

Model	Model size	Model Description
Florence-2-base[[HF]](https://huggingface.co/microsoft/Florence-2-base)	0.23B	Pretrained model with FLD-5B
Florence-2-large[[HF]](https://huggingface.co/microsoft/Florence-2-large)	0.77B	Pretrained model with FLD-5B
Florence-2-base-ft[[HF]](https://huggingface.co/microsoft/Florence-2-base-ft)	0.23B	Finetuned model on a colletion of downstream tasks
Florence-2-large-ft[[HF]](https://huggingface.co/microsoft/Florence-2-large-ft)	0.77B	Finetuned model on a colletion of downstream tasks

How to Get Started with the Model#

Use the code below to get started with the model. All models are trained with float16.

import requests

from PIL import Image
from transformers import AutoProcessor, AutoModelForCausalLM 



device = "cuda:0" if torch.cuda.is_available() else "cpu"
torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32

model = AutoModelForCausalLM.from_pretrained("microsoft/Florence-2-large-ft", torch_dtype=torch_dtype, trust_remote_code=True).to(device)
processor = AutoProcessor.from_pretrained("microsoft/Florence-2-large-ft", trust_remote_code=True)

prompt = "<OD>"

url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg?download=true"
image = Image.open(requests.get(url, stream=True).raw)

inputs = processor(text=prompt, images=image, return_tensors="pt").to(device, torch_dtype)

generated_ids = model.generate(
    input_ids=inputs["input_ids"],
    pixel_values=inputs["pixel_values"],
    max_new_tokens=1024,
    do_sample=False,
    num_beams=3
)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=False)[0]

parsed_answer = processor.post_process_generation(generated_text, task="<OD>", image_size=(image.width, image.height))

print(parsed_answer)

Tasks#

This model is capable of performing different tasks through changing the prompts.

First, let's define a function to run a prompt.

Click to expand

import requests

from PIL import Image
from transformers import AutoProcessor, AutoModelForCausalLM 


device = "cuda:0" if torch.cuda.is_available() else "cpu"
torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32

model = AutoModelForCausalLM.from_pretrained("microsoft/Florence-2-large-ft", torch_dtype=torch_dtype, trust_remote_code=True).to(device)
processor = AutoProcessor.from_pretrained("microsoft/Florence-2-large-ft", trust_remote_code=True)

url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg?download=true"
image = Image.open(requests.get(url, stream=True).raw)

def run_example(task_prompt, text_input=None):
    if text_input is None:
        prompt = task_prompt
    else:
        prompt = task_prompt + text_input
    inputs = processor(text=prompt, images=image, return_tensors="pt").to(device, torch_dtype)
    generated_ids = model.generate(
      input_ids=inputs["input_ids"],
      pixel_values=inputs["pixel_values"],
      max_new_tokens=1024,
      num_beams=3
    )
    generated_text = processor.batch_decode(generated_ids, skip_special_tokens=False)[0]

    parsed_answer = processor.post_process_generation(generated_text, task=task_prompt, image_size=(image.width, image.height))

    print(parsed_answer)

Here are the tasks Florence-2 could perform:

Click to expand

Caption#

prompt = "<CAPTION>"
run_example(prompt)

Detailed Caption#

prompt = "<DETAILED_CAPTION>"
run_example(prompt)

More Detailed Caption#

prompt = "<MORE_DETAILED_CAPTION>"
run_example(prompt)

Caption to Phrase Grounding#

caption to phrase grounding task requires additional text input, i.e. caption.

Caption to phrase grounding results format: {'\<CAPTION_TO_PHRASE_GROUNDING>': {'bboxes': [[x1, y1, x2, y2], ...], 'labels': ['', '', ...]}}

task_prompt = "<CAPTION_TO_PHRASE_GROUNDING>"
results = run_example(task_prompt, text_input="A green car parked in front of a yellow building.")

Object Detection#

OD results format: {'\': {'bboxes': [[x1, y1, x2, y2], ...], 'labels': ['label1', 'label2', ...]} }

prompt = "<OD>"
run_example(prompt)

Dense Region Caption#

Dense region caption results format: {'\<DENSE_REGION_CAPTION>' : {'bboxes': [[x1, y1, x2, y2], ...], 'labels': ['label1', 'label2', ...]} }

prompt = "<DENSE_REGION_CAPTION>"
run_example(prompt)

Region proposal#

Dense region caption results format: {'\<REGION_PROPOSAL>': {'bboxes': [[x1, y1, x2, y2], ...], 'labels': ['', '', ...]}}

prompt = "<REGION_PROPOSAL>"
run_example(prompt)

OCR#

prompt = "<OCR>"
run_example(prompt)

OCR with Region#

OCR with region output format: {'\<OCR_WITH_REGION>': {'quad_boxes': [[x1, y1, x2, y2, x3, y3, x4, y4], ...], 'labels': ['text1', ...]}}

prompt = "<OCR_WITH_REGION>"
run_example(prompt)

for More detailed examples, please refer to notebook

Benchmarks#

Florence-2 Zero-shot performance#

The following table presents the zero-shot performance of generalist vision foundation models on image captioning and object detection evaluation tasks. These models have not been exposed to the training data of the evaluation tasks during their training phase.

Method	#params	COCO Cap. test CIDEr	NoCaps val CIDEr	TextCaps val CIDEr	COCO Det. val2017 mAP
Flamingo	80B	84.3	-	-	-
Florence-2-base	0.23B	133.0	118.7	70.1	34.7
Florence-2-large	0.77B	135.6	120.8	72.8	37.5

The following table continues the comparison with performance on other vision-language evaluation tasks.

Method	Flickr30k test R@1	Refcoco val Accuracy	Refcoco test-A Accuracy	Refcoco test-B Accuracy	Refcoco+ val Accuracy	Refcoco+ test-A Accuracy	Refcoco+ test-B Accuracy	Refcocog val Accuracy	Refcocog test Accuracy	Refcoco RES val mIoU
Kosmos-2	78.7	52.3	57.4	47.3	45.5	50.7	42.2	60.6	61.7	-
Florence-2-base	83.6	53.9	58.4	49.7	51.5	56.4	47.9	66.3	65.1	34.6
Florence-2-large	84.4	56.3	61.6	51.4	53.6	57.9	49.9	68.0	67.0	35.8

Florence-2 finetuned performance#

We finetune Florence-2 models with a collection of downstream tasks, resulting two generalist models Florence-2-base-ft and Florence-2-large-ft that can conduct a wide range of downstream tasks.

The table below compares the performance of specialist and generalist models on various captioning and Visual Question Answering (VQA) tasks. Specialist models are fine-tuned specifically for each task, whereas generalist models are fine-tuned in a task-agnostic manner across all tasks. The symbol "▲" indicates the usage of external OCR as input.

Method	# Params	COCO Caption Karpathy test CIDEr	NoCaps val CIDEr	TextCaps val CIDEr	VQAv2 test-dev Acc	TextVQA test-dev Acc	VizWiz VQA test-dev Acc
Specialist Models
CoCa	2.1B	143.6	122.4	-	82.3	-	-
BLIP-2	7.8B	144.5	121.6	-	82.2	-	-
GIT2	5.1B	145.0	126.9	148.6	81.7	67.3	71.0
Flamingo	80B	138.1	-	-	82.0	54.1	65.7
PaLI	17B	149.1	127.0	160.0▲	84.3	58.8 / 73.1▲	71.6 / 74.4▲
PaLI-X	55B	149.2	126.3	147.0 / 163.7▲	86.0	71.4 / 80.8▲	70.9 / 74.6▲
Generalist Models
Unified-IO	2.9B	-	100.0	-	77.9	-	57.4
Florence-2-base-ft	0.23B	140.0	116.7	143.9	79.7	63.6	63.6
Florence-2-large-ft	0.77B	143.3	124.9	151.1	81.7	73.5	72.6

Method	# Params	COCO Det. val2017 mAP	Flickr30k test R@1	RefCOCO val Accuracy	RefCOCO test-A Accuracy	RefCOCO test-B Accuracy	RefCOCO+ val Accuracy	RefCOCO+ test-A Accuracy	RefCOCO+ test-B Accuracy	RefCOCOg val Accuracy	RefCOCOg test Accuracy	RefCOCO RES val mIoU
Specialist Models
SeqTR	-	-	-	83.7	86.5	81.2	71.5	76.3	64.9	74.9	74.2	-
PolyFormer	-	-	-	90.4	92.9	87.2	85.0	89.8	78.0	85.8	85.9	76.9
UNINEXT	0.74B	60.6	-	92.6	94.3	91.5	85.2	89.6	79.8	88.7	89.4	-
Ferret	13B	-	-	89.5	92.4	84.4	82.8	88.1	75.2	85.8	86.3	-
Generalist Models
UniTAB	-	-	-	88.6	91.1	83.8	81.0	85.4	71.6	84.6	84.7	-
Florence-2-base-ft	0.23B	41.4	84.0	92.6	94.8	91.5	86.8	91.7	82.2	89.8	82.2	78.0
Florence-2-large-ft	0.77B	43.4	85.2	93.4	95.3	92.0	88.3	92.9	83.6	91.2	91.7	80.5

BibTex and citation info#

@article{xiao2023florence,
  title={Florence-2: Advancing a unified representation for a variety of vision tasks},
  author={Xiao, Bin and Wu, Haiping and Xu, Weijian and Dai, Xiyang and Hu, Houdong and Lu, Yumao and Zeng, Michael and Liu, Ce and Yuan, Lu},
  journal={arXiv preprint arXiv:2311.06242},
  year={2023}
}

Files

17 files · 3.09 GBWeights 1.54 GB (safetensors). Sizes and sha256 from the signed manifest.

Hide all 17 files with their sha256

File	Size	sha256
.gitattributes	1.56 KB	28aac17b0f70983ef58c18a7ee355503e454ce3178fce77a7b50503906df0635
CODE_OF_CONDUCT.md	444 B	9daeae709a0bd71bcfd1c96dc5822ecec5210327eff929da64b0ae7f8faf1444
LICENSE	1.14 KB	c2cfccb812fe482101a8f04597dfc5a9991a6b2748266c47ac91b6a5aae15383
README.md	14.8 KB	a7ce927349dee88fbe0ca4c9944d38b88c7025b3731ae15f0a2f1b1c42aa49cc
SECURITY.md	2.66 KB	7b6976eec43edfa68b79a459dd089c56b7a395916dbf1a01bd11e6d86e12128f
SUPPORT.md	1.24 KB	149773411caf2c59cb1a6bfb2c6f070d1d054f1f018e7a80be319a8c384eeaf5
config.json	2.44 KB	fa081841369aa9c6e42faf5c52368d673b561e2c5f8fa03d1256e7408cb4130e
configuration_florence2.py	15.1 KB	653bafddc9651eaff1583a16db4a2bb27d33ec7d541dfab7201aaa4ecaa1cfbf
generation_config.json	51 B	30e9865458ecc8ee931eeeb43f44f1d169c5ab95be39e0072142a7a6b8f31990
model.safetensors	1.54 GB	8b4e610c952eef90a836c56cda0f398a672a3a6ca7b4d96b0e09a86dee42e2c3
modeling_florence2.py	127 KB	5bb7aa72c6ba62e96e1bbae6bc1aaf7b4e8e28cdfc62e670de3d5b67eeab1fdf
preprocessor_config.json	806 B	2f5921bbc53c7cc04251e1027b45b1cec726276be6db23d1bb40641bfbe2cf29
processing_florence2.py	46.4 KB	4bd7158536cbf1c7891fc8efd94437d79fd09f07f539c7398fab8a885d7d8bca
pytorch_model.bin	1.54 GB	4188ce5a9364c358c3e28eb3e4155985867ef70bb3ae628f1f289d6a3538310a
tokenizer.json	1.36 MB	847bbeab6174d66a88898f729d52fa8d355fafe1bea101cf960dd404581df70e
tokenizer_config.json	34 B	79ffcf43af8ebda99d165f61d243180da2e2639952e41e71e11611c18770489c
vocab.json	1.10 MB	394fdc63c71aabe0a9b97117f5d62fb5fcc4d59b2b3ea929a3929e6a53217b3c

VersionsEach time the pinned upstream commit or file list changed in this repo. Newest first.

Upstream commit	Date	Contents
4a12a2b54b pinned now	2026-11-15 02:36 UTC	17 files · 3.09 GB

Use it

$ fw get microsoft/Florence-2-large-ft
# manifest sha256 66c7e61a04a6… · 3.09 GB · 17 files
$ fw verify ./Florence-2-large-ft microsoft/Florence-2-large-ft

fw.pyChecks the signature and every sha256, tries each source in order, resumes.

$ python3 fw.py get https://freeweights.allternit.com/manifests/microsoft/Florence-2-large-ft.json --dir ./Florence-2-large-ft
$ python3 fw.py verify ./Florence-2-large-ft ./microsoft__Florence-2-large-ft.json

hfThe Hugging Face CLI with one setting. Same repo id.

$ HF_ENDPOINT=https://freeweights.allternit.com hf download microsoft/Florence-2-large-ft --revision 4a12a2b54b7016a48a22037fbd62da90cd566f2a --local-dir ./Florence-2-large-ft

transformersLoad the weights in Python.

import os
os.environ["HF_ENDPOINT"] = "https://freeweights.allternit.com"
from transformers import AutoModelForImageTextToText, AutoProcessor
processor = AutoProcessor.from_pretrained("microsoft/Florence-2-large-ft", revision="4a12a2b54b7016a48a22037fbd62da90cd566f2a")
model = AutoModelForImageTextToText.from_pretrained("microsoft/Florence-2-large-ft", revision="4a12a2b54b7016a48a22037fbd62da90cd566f2a", device_map="auto")

vLLMServe an OpenAI-compatible API.

$ HF_ENDPOINT=https://freeweights.allternit.com vllm serve microsoft/Florence-2-large-ft --revision 4a12a2b54b7016a48a22037fbd62da90cd566f2a

Fine-tune in Studio

License

Name: MIT License
Id: MIT
Class: Permissive
Redistribution: yes
Listing: Hosted (signed)
Decided: License as declared by the publisher. No person read it. 2026-11-15
Text in: LICENSE
Text sha256: c2cfccb812fe482101a8f04597dfc5a9991a6b2748266c47ac91b6a5aae15383
Terms gate: No extra terms beyond the license.

Read the license text

Licenses are checked against the license table. A custom license passes only after a person reads it.

Provenance

From:

microsoft <huggingface.co/microsoft>Newsgroups: alt.freeweights.textSubject: Florence-2-large-ftDate: 2026-11-15Commit: 4a12a2bSigned: 2DC28CC7B26A283B ✓

Upstream: microsoft/Florence-2-large-ft
Pinned commit: 4a12a2b54b7016a48a22037fbd62da90cd566f2a
Manifest: Florence-2-large-ft.json · .minisig
Manifest sha256: 66c7e61a04a644b513a32dc125587a0dc8882d0328a0d0ef46eb345c4652df0b
Signature: minisign, Ed25519, key 2DC28CC7B26A283B
Created: 2026-11-15 02:36 UTC
Generator: freeweights-manifest/0.1.0
Takedowns: None.

How to verify

$ curl -O https://freeweights.allternit.com/manifests/microsoft/Florence-2-large-ft.json
$ curl -O https://freeweights.allternit.com/manifests/microsoft/Florence-2-large-ft.json.minisig
$ minisign -Vm Florence-2-large-ft.json -P RWQ7KGqyx4zCLXtUFOqvfv8L3g+8fpKJ1FsuxPyOYMF+4KufbUxKq1Oq
$ shasum -a 256 <file>   # compare with the Files tab

A manifest whose signature fails must be rejected. A file whose size or sha256 does not match must be discarded. Verify a download

Model tree

1 model in the catalog name this one as their parent.

Quantizations: 1

Quantizations 1

onnx-community/Florence-2-large-ft14.8 GB

Get it

Download

$ HF_ENDPOINT=https://freeweights.allternit.com hf download microsoft/Florence-2-large-ft

Open in Freeweights Studio

Pull, verify and run it in the Allternit app on Desktop, web or phone.

Signed manifest

minisign, key 2DC28CC7B26A283B. Florence-2-large-ft.json. Provenance. Takedown history: none.

Source

huggingface.co/microsoft/Florence-2-large-ft

Variants

1 quantized or adapter build of Florence-2-large-ft. Each has its own license check and page.

SubjectSizeParamsLicenseDate

Florence-2-large-ftHosted (signed) From: onnx-community in alt.freeweights.text14.8 GB—Permissive2026-11-15

Related models

All of alt.freeweights.text →

From: laion <huggingface.co/laion>Subject: a2-rl-e2egit_large-50-32B

From: laion <huggingface.co/laion>Subject: a3-rl-DCAgent_exp_rpt_e2egit-large-15-8B

From: laion <huggingface.co/laion>Subject: a3-rl-DCAgent_exp_rpt_e2egit-v2-10-8B

From: laion <huggingface.co/laion>Subject: a3-rl-DCAgent_exp_rpt_pymethods2test-large-80-8B

From: laion <huggingface.co/laion>Subject: a3-rl-DCAgent_exp_rpt_unitsyn-python-large-10-8B

From: laion <huggingface.co/laion>Subject: a3-rl-DCAgent_mix_h2_language_proportional-65-8B