alt.freeweights.text / Florence-2-base

From:

microsoft <huggingface.co/microsoft>Newsgroups: alt.freeweights.textSubject:

Florence-2-base

Date: 2026-11-15Commit: 5ca5edfSigned: 2DC28CC7B26A283B ✓

Image text to text model with 232M parameters. Safetensors weights, 930 MB in all. Published by microsoft under MIT.

232M params

930MBFlorence-2-basesafetensorspermissive

FactsParams232MSize930 MBFormatssafetensorsTaskimage text to textLicenseMITFrom sourcepermissive

$ fw get microsoft/Florence-2-base
# manifest sha256 77742c6f0a3d… · 930 MB · 16 files
$ fw verify ./Florence-2-base microsoft/Florence-2-base

From sourceHosted (signed)microsoft/Florence-2-baseLast check: 2026-11-15 02:25 UTC

Fine-tune in Studio Get help

Get help: custom fine-tune or private deploy. We review your setup before we quote.

Runs with

AIBrix BentoML CTranslate2 ExLlamaV2 ExLlamaV3 exo fastllm FlashInfer KTransformers LitServe LMCache LMDeploy LocalAI mistral.rs MLC LLM Mooncake

SGLang Text Generation WebUI tinygrad

Transformers Triton Inference Server

vLLM XinferenceFine-tune with

Accelerate

Alignment Handbook

Axolotl Colossal-AI DeepSpeed EasyR1 FlashAttention H2O LLM Studio JAX Keras Liger Kernel LitGPT LLaMA-Factory Megatron-LM ms-swift

nanotron

OLMo-core OpenRLHF Oumi

PEFT PyTorch Lightning sd-scripts SimpleTuner TorchTitan torchtune Transformer Engine

TRL

Unsloth verlConvert withAQLM AutoRound Core ML Tools ExecuTorch GPTQModel

LLM Compressor mergekit Neural Compressor

Olive ONNX

Optimum Quanto

Optimum

safetensors TensorRT Model Optimizer torchaoEvaluate withAutoevals DeepEval

Evaluate HELM

LightEval lm-evaluation-harness OpenCompass Opik Ragas

simple-evals SWE-bench TruLens WeaveScan withAgentDojo AI Fairness 360 DeepTeam Detoxify Fairlearn garak Giskard Guardrails MCP Scanner ModelScan NeMo Guardrails picklescan promptfooWorks with

Agent Development Kit AG2 Google AI Edge Gallery Aider any-llm AnythingLLM BAML browser-use CAMEL

Hugging Face Chat UI Chatbox Cherry Studio Cline

Codex CLI Cognee CrewAI DeerFlow docTR DSPy EasyOCR Enchanted Gemini CLI GitHub MCP Server goose GPT Researcher Gradio Graphiti Guidance Hollama Instructor Kilo Code Langflow LangGraph LiteLLM LlamaIndex LLM LocalAGI LoLLMs WebUI MCP Python SDK Mem0 MetaGPT NextChat

Ollama JavaScript

Ollama Python

olmOCR

OpenAI Agents SDK opencode OpenHands oterm Outlines Page Assist

Playwright MCP Portkey AI Gateway PrivateGPT Qwen Code RWKV Runner

Semantic Kernel

smolagents Strands Agents Surya Vane XGrammar

Overview Model card Files Use it License Provenance Model tree

Overview

Size: 930 MB
Weights: 463 MB (safetensors)
Parameters: 232M
Formats: safetensors
Task: image text to text
License: MIT (permissive)
Files: 16, each with a sha256
Publisher: microsoft
Added: 2026-11-15
Section: alt.freeweights.text

Lineage

Original release from the publisher.

Where these bytes come from

Publisher at the pinned commit→You

Sources are tried in order. A file counts only when its size and sha256 match the manifest. Sources are checked when you ask for them, not on a timer.

Sources

Badge	Host	Last check	Result
From source	huggingface.co	2026-11-15 02:25 UTC	passed

Model card

From the publisher at commit 5ca5edf. Shown as plain text with links; images open at the publisher.

Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks#

Model Summary#

This Hub repository contains a HuggingFace's transformers implementation of Florence-2 model from Microsoft.

Florence-2 is an advanced vision foundation model that uses a prompt-based approach to handle a wide range of vision and vision-language tasks. Florence-2 can interpret simple text prompts to perform tasks like captioning, object detection, and segmentation. It leverages our FLD-5B dataset, containing 5.4 billion annotations across 126 million images, to master multi-task learning. The model's sequence-to-sequence architecture enables it to excel in both zero-shot and fine-tuned settings, proving to be a competitive vision foundation model.

Resources and Technical Documentation:

Model	Model size	Model Description
Florence-2-base[[HF]](https://huggingface.co/microsoft/Florence-2-base)	0.23B	Pretrained model with FLD-5B
Florence-2-large[[HF]](https://huggingface.co/microsoft/Florence-2-large)	0.77B	Pretrained model with FLD-5B
Florence-2-base-ft[[HF]](https://huggingface.co/microsoft/Florence-2-base-ft)	0.23B	Finetuned model on a colletion of downstream tasks
Florence-2-large-ft[[HF]](https://huggingface.co/microsoft/Florence-2-large-ft)	0.77B	Finetuned model on a colletion of downstream tasks

How to Get Started with the Model#

Use the code below to get started with the model. All models are trained with float16.

import requests

from PIL import Image
from transformers import AutoProcessor, AutoModelForCausalLM 

device = "cuda:0" if torch.cuda.is_available() else "cpu"
torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32

model = AutoModelForCausalLM.from_pretrained("microsoft/Florence-2-base", torch_dtype=torch_dtype, trust_remote_code=True).to(device)
processor = AutoProcessor.from_pretrained("microsoft/Florence-2-base", trust_remote_code=True)

prompt = "<OD>"

url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg?download=true"
image = Image.open(requests.get(url, stream=True).raw)

inputs = processor(text=prompt, images=image, return_tensors="pt").to(device, torch_dtype)

generated_ids = model.generate(
    input_ids=inputs["input_ids"],
    pixel_values=inputs["pixel_values"],
    max_new_tokens=1024,
    do_sample=False,
    num_beams=3,
)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=False)[0]

parsed_answer = processor.post_process_generation(generated_text, task="<OD>", image_size=(image.width, image.height))

print(parsed_answer)

Tasks#

This model is capable of performing different tasks through changing the prompts.

First, let's define a function to run a prompt.

Click to expand

import requests

from PIL import Image
from transformers import AutoProcessor, AutoModelForCausalLM 

device = "cuda:0" if torch.cuda.is_available() else "cpu"
torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32

model = AutoModelForCausalLM.from_pretrained("microsoft/Florence-2-base", torch_dtype=torch_dtype, trust_remote_code=True).to(device)
processor = AutoProcessor.from_pretrained("microsoft/Florence-2-base", trust_remote_code=True)

url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg?download=true"
image = Image.open(requests.get(url, stream=True).raw)

def run_example(task_prompt, text_input=None):
    if text_input is None:
        prompt = task_prompt
    else:
        prompt = task_prompt + text_input
    inputs = processor(text=prompt, images=image, return_tensors="pt").to(device, torch_dtype)
    generated_ids = model.generate(
      input_ids=inputs["input_ids"],
      pixel_values=inputs["pixel_values"],
      max_new_tokens=1024,
      num_beams=3
    )
    generated_text = processor.batch_decode(generated_ids, skip_special_tokens=False)[0]

    parsed_answer = processor.post_process_generation(generated_text, task=task_prompt, image_size=(image.width, image.height))

    print(parsed_answer)

Here are the tasks Florence-2 could perform:

Click to expand

Caption#

prompt = "<CAPTION>"
run_example(prompt)

Detailed Caption#

prompt = "<DETAILED_CAPTION>"
run_example(prompt)

More Detailed Caption#

prompt = "<MORE_DETAILED_CAPTION>"
run_example(prompt)

Caption to Phrase Grounding#

caption to phrase grounding task requires additional text input, i.e. caption.

Caption to phrase grounding results format: {'\<CAPTION_TO_PHRASE_GROUNDING>': {'bboxes': [[x1, y1, x2, y2], ...], 'labels': ['', '', ...]}}

task_prompt = "<CAPTION_TO_PHRASE_GROUNDING>"
results = run_example(task_prompt, text_input="A green car parked in front of a yellow building.")

Object Detection#

OD results format: {'\': {'bboxes': [[x1, y1, x2, y2], ...], 'labels': ['label1', 'label2', ...]} }

prompt = "<OD>"
run_example(prompt)

Dense Region Caption#

Dense region caption results format: {'\<DENSE_REGION_CAPTION>' : {'bboxes': [[x1, y1, x2, y2], ...], 'labels': ['label1', 'label2', ...]} }

prompt = "<DENSE_REGION_CAPTION>"
run_example(prompt)

Region proposal#

Dense region caption results format: {'\<REGION_PROPOSAL>': {'bboxes': [[x1, y1, x2, y2], ...], 'labels': ['', '', ...]}}

prompt = "<REGION_PROPOSAL>"
run_example(prompt)

OCR#

prompt = "<OCR>"
run_example(prompt)

OCR with Region#

OCR with region output format: {'\<OCR_WITH_REGION>': {'quad_boxes': [[x1, y1, x2, y2, x3, y3, x4, y4], ...], 'labels': ['text1', ...]}}

prompt = "<OCR_WITH_REGION>"
run_example(prompt)

for More detailed examples, please refer to notebook

Benchmarks#

Florence-2 Zero-shot performance#

The following table presents the zero-shot performance of generalist vision foundation models on image captioning and object detection evaluation tasks. These models have not been exposed to the training data of the evaluation tasks during their training phase.

Method	#params	COCO Cap. test CIDEr	NoCaps val CIDEr	TextCaps val CIDEr	COCO Det. val2017 mAP
Flamingo	80B	84.3	-	-	-
Florence-2-base	0.23B	133.0	118.7	70.1	34.7
Florence-2-large	0.77B	135.6	120.8	72.8	37.5

The following table continues the comparison with performance on other vision-language evaluation tasks.

Method	Flickr30k test R@1	Refcoco val Accuracy	Refcoco test-A Accuracy	Refcoco test-B Accuracy	Refcoco+ val Accuracy	Refcoco+ test-A Accuracy	Refcoco+ test-B Accuracy	Refcocog val Accuracy	Refcocog test Accuracy	Refcoco RES val mIoU
Kosmos-2	78.7	52.3	57.4	47.3	45.5	50.7	42.2	60.6	61.7	-
Florence-2-base	83.6	53.9	58.4	49.7	51.5	56.4	47.9	66.3	65.1	34.6
Florence-2-large	84.4	56.3	61.6	51.4	53.6	57.9	49.9	68.0	67.0	35.8

Florence-2 finetuned performance#

We finetune Florence-2 models with a collection of downstream tasks, resulting two generalist models Florence-2-base-ft and Florence-2-large-ft that can conduct a wide range of downstream tasks.

The table below compares the performance of specialist and generalist models on various captioning and Visual Question Answering (VQA) tasks. Specialist models are fine-tuned specifically for each task, whereas generalist models are fine-tuned in a task-agnostic manner across all tasks. The symbol "▲" indicates the usage of external OCR as input.

Method	# Params	COCO Caption Karpathy test CIDEr	NoCaps val CIDEr	TextCaps val CIDEr	VQAv2 test-dev Acc	TextVQA test-dev Acc	VizWiz VQA test-dev Acc
Specialist Models
CoCa	2.1B	143.6	122.4	-	82.3	-	-
BLIP-2	7.8B	144.5	121.6	-	82.2	-	-
GIT2	5.1B	145.0	126.9	148.6	81.7	67.3	71.0
Flamingo	80B	138.1	-	-	82.0	54.1	65.7
PaLI	17B	149.1	127.0	160.0▲	84.3	58.8 / 73.1▲	71.6 / 74.4▲
PaLI-X	55B	149.2	126.3	147.0 / 163.7▲	86.0	71.4 / 80.8▲	70.9 / 74.6▲
Generalist Models
Unified-IO	2.9B	-	100.0	-	77.9	-	57.4
Florence-2-base-ft	0.23B	140.0	116.7	143.9	79.7	63.6	63.6
Florence-2-large-ft	0.77B	143.3	124.9	151.1	81.7	73.5	72.6

Method	# Params	COCO Det. val2017 mAP	Flickr30k test R@1	RefCOCO val Accuracy	RefCOCO test-A Accuracy	RefCOCO test-B Accuracy	RefCOCO+ val Accuracy	RefCOCO+ test-A Accuracy	RefCOCO+ test-B Accuracy	RefCOCOg val Accuracy	RefCOCOg test Accuracy	RefCOCO RES val mIoU
Specialist Models
SeqTR	-	-	-	83.7	86.5	81.2	71.5	76.3	64.9	74.9	74.2	-
PolyFormer	-	-	-	90.4	92.9	87.2	85.0	89.8	78.0	85.8	85.9	76.9
UNINEXT	0.74B	60.6	-	92.6	94.3	91.5	85.2	89.6	79.8	88.7	89.4	-
Ferret	13B	-	-	89.5	92.4	84.4	82.8	88.1	75.2	85.8	86.3	-
Generalist Models
UniTAB	-	-	-	88.6	91.1	83.8	81.0	85.4	71.6	84.6	84.7	-
Florence-2-base-ft	0.23B	41.4	84.0	92.6	94.8	91.5	86.8	91.7	82.2	89.8	82.2	78.0
Florence-2-large-ft	0.77B	43.4	85.2	93.4	95.3	92.0	88.3	92.9	83.6	91.2	91.7	80.5

BibTex and citation info#

@article{xiao2023florence,
  title={Florence-2: Advancing a unified representation for a variety of vision tasks},
  author={Xiao, Bin and Wu, Haiping and Xu, Weijian and Dai, Xiyang and Hu, Houdong and Lu, Yumao and Zeng, Michael and Liu, Ce and Yuan, Lu},
  journal={arXiv preprint arXiv:2311.06242},
  year={2023}
}

Files

16 files · 930 MBWeights 463 MB (safetensors). Sizes and sha256 from the signed manifest.

Hide all 16 files with their sha256

File	Size	sha256
.gitattributes	1.56 KB	28aac17b0f70983ef58c18a7ee355503e454ce3178fce77a7b50503906df0635
CODE_OF_CONDUCT.md	444 B	9daeae709a0bd71bcfd1c96dc5822ecec5210327eff929da64b0ae7f8faf1444
LICENSE	1.14 KB	c2cfccb812fe482101a8f04597dfc5a9991a6b2748266c47ac91b6a5aae15383
README.md	14.8 KB	e49de5527bd39688745ce20388bf3f31fdb891abc0e6950a3f64630114769d17
SECURITY.md	2.66 KB	7b6976eec43edfa68b79a459dd089c56b7a395916dbf1a01bd11e6d86e12128f
SUPPORT.md	1.24 KB	149773411caf2c59cb1a6bfb2c6f070d1d054f1f018e7a80be319a8c384eeaf5
config.json	2.43 KB	c666d0fe0172d46e115e8fba6cd93cd83714575b33a73005cab8d24ce2a3aa8f
configuration_florence2.py	15.1 KB	de2e45a975b3582de05d2f4d963a3e9f9a3d20dccf78d28e0052932a0be93bdf
model.safetensors	463 MB	03075d2d2d2bbd3e180b9ba0afae4aa8563226e2d32911656966e05b2f2ee060
modeling_florence2.py	127 KB	5162bf465e61b6e29cc113a467630ec3cb56ed8e4d46eb6207157f10fb9b8a24
preprocessor_config.json	806 B	2f5921bbc53c7cc04251e1027b45b1cec726276be6db23d1bb40641bfbe2cf29
processing_florence2.py	48.7 KB	f146023a507c009f425a49ee39aa037f4f25c64e14336e3e4f3f1d7377a68e98
pytorch_model.bin	464 MB	b480ac374593b0dcb18ffa63b23213734e04cd43eab0d620d23e39708d4a4a7e
tokenizer.json	1.36 MB	847bbeab6174d66a88898f729d52fa8d355fafe1bea101cf960dd404581df70e
tokenizer_config.json	34 B	79ffcf43af8ebda99d165f61d243180da2e2639952e41e71e11611c18770489c
vocab.json	1.10 MB	394fdc63c71aabe0a9b97117f5d62fb5fcc4d59b2b3ea929a3929e6a53217b3c

VersionsEach time the pinned upstream commit or file list changed in this repo. Newest first.

Upstream commit	Date	Contents
5ca5edf5bd pinned now	2026-11-15 02:25 UTC	16 files · 930 MB

Use it

$ fw get microsoft/Florence-2-base
# manifest sha256 77742c6f0a3d… · 930 MB · 16 files
$ fw verify ./Florence-2-base microsoft/Florence-2-base

fw.pyChecks the signature and every sha256, tries each source in order, resumes.

$ python3 fw.py get https://freeweights.allternit.com/manifests/microsoft/Florence-2-base.json --dir ./Florence-2-base
$ python3 fw.py verify ./Florence-2-base ./microsoft__Florence-2-base.json

hfThe Hugging Face CLI with one setting. Same repo id.

$ HF_ENDPOINT=https://freeweights.allternit.com hf download microsoft/Florence-2-base --revision 5ca5edf5bd017b9919c05d08aebef5e4c7ac3bac --local-dir ./Florence-2-base

transformersLoad the weights in Python.

import os
os.environ["HF_ENDPOINT"] = "https://freeweights.allternit.com"
from transformers import AutoModelForImageTextToText, AutoProcessor
processor = AutoProcessor.from_pretrained("microsoft/Florence-2-base", revision="5ca5edf5bd017b9919c05d08aebef5e4c7ac3bac")
model = AutoModelForImageTextToText.from_pretrained("microsoft/Florence-2-base", revision="5ca5edf5bd017b9919c05d08aebef5e4c7ac3bac", device_map="auto")

vLLMServe an OpenAI-compatible API.

$ HF_ENDPOINT=https://freeweights.allternit.com vllm serve microsoft/Florence-2-base --revision 5ca5edf5bd017b9919c05d08aebef5e4c7ac3bac

Fine-tune in Studio

License

Name: MIT License
Id: MIT
Class: Permissive
Redistribution: yes
Listing: Hosted (signed)
Decided: License as declared by the publisher. No person read it. 2026-11-15
Text in: LICENSE
Text sha256: c2cfccb812fe482101a8f04597dfc5a9991a6b2748266c47ac91b6a5aae15383
Terms gate: No extra terms beyond the license.

Read the license text

Licenses are checked against the license table. A custom license passes only after a person reads it.

Provenance

From:

microsoft <huggingface.co/microsoft>Newsgroups: alt.freeweights.textSubject: Florence-2-baseDate: 2026-11-15Commit: 5ca5edfSigned: 2DC28CC7B26A283B ✓

Upstream: microsoft/Florence-2-base
Pinned commit: 5ca5edf5bd017b9919c05d08aebef5e4c7ac3bac
Manifest: Florence-2-base.json · .minisig
Manifest sha256: 77742c6f0a3db10ca7ad200fc8f1c52844d3c8a7f9c1ba391b0777f71d39656d
Signature: minisign, Ed25519, key 2DC28CC7B26A283B
Created: 2026-11-15 02:25 UTC
Generator: freeweights-manifest/0.1.0
Takedowns: None.

How to verify

$ curl -O https://freeweights.allternit.com/manifests/microsoft/Florence-2-base.json
$ curl -O https://freeweights.allternit.com/manifests/microsoft/Florence-2-base.json.minisig
$ minisign -Vm Florence-2-base.json -P RWQ7KGqyx4zCLXtUFOqvfv8L3g+8fpKJ1FsuxPyOYMF+4KufbUxKq1Oq
$ shasum -a 256 <file>   # compare with the Files tab

A manifest whose signature fails must be rejected. A file whose size or sha256 does not match must be discarded. Verify a download

Model tree

1 model in the catalog name this one as their parent.

Quantizations: 1

Quantizations 1

onnx-community/Florence-2-base5.37 GB

Get it

Download

$ HF_ENDPOINT=https://freeweights.allternit.com hf download microsoft/Florence-2-base

Open in Freeweights Studio

Pull, verify and run it in the Allternit app on Desktop, web or phone.

Signed manifest

minisign, key 2DC28CC7B26A283B. Florence-2-base.json. Provenance. Takedown history: none.

Source

huggingface.co/microsoft/Florence-2-base

Variants

1 quantized or adapter build of Florence-2-base. Each has its own license check and page.

SubjectSizeParamsLicenseDate

Florence-2-baseHosted (signed) From: onnx-community in alt.freeweights.text5.37 GB—Permissive2026-11-15

Related models

All of alt.freeweights.text →

From: laion <huggingface.co/laion>Subject: a2-rl-e2egit_large-50-32B

From: laion <huggingface.co/laion>Subject: a3-rl-DCAgent_exp_rpt_e2egit-large-15-8B

From: laion <huggingface.co/laion>Subject: a3-rl-DCAgent_exp_rpt_e2egit-v2-10-8B

From: laion <huggingface.co/laion>Subject: a3-rl-DCAgent_exp_rpt_pymethods2test-large-80-8B

From: laion <huggingface.co/laion>Subject: a3-rl-DCAgent_exp_rpt_unitsyn-python-large-10-8B

From: laion <huggingface.co/laion>Subject: a3-rl-DCAgent_mix_h2_language_proportional-65-8B