UEmbed-9B

UEmbed: Unified Sparse and Dense Multimodal Embeddings#

Website arXiv GitHub Image: License: CC-BY-4.0

UEmbed is a decoder-only multimodal embedding model that produces both dense embeddings and SPLADE-style sparse lexical embeddings from a single causal forward pass. It supports text, image, video, and mixed-modal inputs for retrieval, multimodal search, and visual-document retrieval.

News 🔥#

• [2026-08-17]: Thanks to the ms-swift team, our models now support sft with ms-swift.
• [2026-08-15]: UEmbed achieves state-of-the-art results on the text and agent tracks of MMEB-v3, and ranks second only to the Qwen3-VL-Embedding series among open-source models on MMEB-v2.
• [2026-08-13]: We updated the inference code — UEmbed now loads natively with transformers (no trust_remote_code or processor patching needed), and we added a vLLM backend for high-throughput dense and sparse embedding inference. See the runnable examples: examples/transformers_example.py and examples/vllm_example.py.

Model Family#

Highlights#

• Unified dense and sparse retrieval: one checkpoint returns normalized dense vectors and sparse lexical vectors.
• Multimodal inputs: text, images, videos, and mixed inputs are represented in the same retrieval space.
• Sparse interpretability: sparse activations correspond to vocabulary terms and can be used with inverted indexes.
• Causal-model serving compatibility: the sparse design keeps the decoder-only backbone, no conversion to a bidirectional encoder.

Architecture#

Usage#

Requires a recent transformers build with Qwen3.5/Qwen3-VL support:

pip install "transformers>=5.4.0" torch qwen-vl-utils tokenizers huggingface-hub pillow numpy

Download the complete model repository, since sparse inference requires both sparse_info.json and sparse_weights.pt in the local model directory:

huggingface-cli download Alibaba-NLP/UEmbed-2B --local-dir ./models/UEmbed-2B

Inference code is provided in the GitHub repository. Set pooling="last.normal" for dense embeddings or pooling="splade.last" for sparse embeddings.

import torch
from src.models.qwen35_embedding import Qwen35Embedder

model = Qwen35Embedder(
    model_name_or_path="./models/UEmbed-2B",
    torch_dtype=torch.bfloat16,
    # flash_attention_2 for better acceleration and memory saving
    attn_implementation="flash_attention_2",
)

inputs = [{
    "text": "A woman playing with her dog on a beach at sunset.",
    "instruction": "Retrieve images or text relevant to the user's query.",
}, {
    "text": "A woman shares a joyful moment with her golden retriever on a sun-drenched beach at sunset, as the dog offers its paw in a heartwarming display of companionship and trust."
}, {
    "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg"
}, {
    "text": "A woman shares a joyful moment with her golden retriever on a sun-drenched beach at sunset, as the dog offers its paw in a heartwarming display of companionship and trust.",
    "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg"
}]

embeddings = model.process(inputs)
print(embeddings @ embeddings.T)

Input Format#

Qwen35Embedder.process accepts a list of dictionaries with the following fields:

Training Data#

UEmbed is trained on 3.94M public samples:

• E5 training data for broad text retrieval coverage.
• M3 training data, using the MLDR subset.
• MMEB training sets for multimodal query-document pairs.

For multimodal data, hard negatives are mined with Qwen3-VL-Embedding-8B as the teacher retriever.

Citation#

If you use UEmbed, please cite the paper:

@misc{uembed2026,
      title={UEmbed: Unified Sparse and Dense Multimodal Embeddings}, 
      author={Tingyu Song and Mingxin Li and Yanzhao Zhang and Dingkun Long and Pengjun Xie and Zhijie Nie and Yilun Zhao and Shu Wu},
      year={2026},
      eprint={2608.02583},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2608.02583}, 
}

Acknowledgements#

Thanks to the Qwen3-VL-Embedding repo for the evaluation framework.