We are excited to introduce the gte-modernbert series of models, which are built upon the latest modernBERT pre-trained encoder-only foundation models. The gte-modernbert series models include both text embedding models and rerank models.
The gte-modernbert models demonstrates competitive performance in several text embedding and text retrieval evaluation tasks when compared to similar-scale models from the current open-source community. This includes assessments such as MTEB, LoCO, and COIR evaluation.
Tip. For transformers and sentence-transformers, if your GPU supports it, the efficient Flash Attention 2 will be used automatically if you have flash_attn installed. It is not mandatory.
pip install flash_attn
Use with transformers
# Requires transformers>=4.48.0
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer
input_texts = [
"what is the capital of China?",
"how to implement quick sort in python?",
"Beijing",
"sorting algorithms"
]
model_path = "Alibaba-NLP/gte-modernbert-base"
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModel.from_pretrained(model_path)
# Tokenize the input texts
batch_dict = tokenizer(input_texts, max_length=8192, padding=True, truncation=True, return_tensors='pt')
outputs = model(**batch_dict)
embeddings = outputs.last_hidden_state[:, 0]
# (Optionally) normalize embeddings
embeddings = F.normalize(embeddings, p=2, dim=1)
scores = (embeddings[:1] @ embeddings[1:].T) * 100
print(scores.tolist())
# [[42.89073944091797, 71.30911254882812, 33.664554595947266]]
Use with sentence-transformers:
# Requires transformers>=4.48.0
from sentence_transformers import SentenceTransformer
from sentence_transformers.util import cos_sim
input_texts = [
"what is the capital of China?",
"how to implement quick sort in python?",
"Beijing",
"sorting algorithms"
]
model = SentenceTransformer("Alibaba-NLP/gte-modernbert-base")
embeddings = model.encode(input_texts)
print(embeddings.shape)
# (4, 768)
similarities = cos_sim(embeddings[0], embeddings[1:])
print(similarities)
# tensor([[0.4289, 0.7131, 0.3366]])
Use with transformers.js:
// npm i @huggingface/transformers
import { pipeline, matmul } from "@huggingface/transformers";
// Create a feature extraction pipeline
const extractor = await pipeline(
"feature-extraction",
"Alibaba-NLP/gte-modernbert-base",
{ dtype: "fp32" }, // Supported options: "fp32", "fp16", "q8", "q4", "q4f16"
);
// Embed queries and documents
const embeddings = await extractor(
[
"what is the capital of China?",
"how to implement quick sort in python?",
"Beijing",
"sorting algorithms",
],
{ pooling: "cls", normalize: true },
);
// Compute similarity scores
const similarities = (await matmul(embeddings.slice([0, 1]), embeddings.slice([1, null]).transpose(1, 0))).mul(100);
console.log(similarities.tolist()); // [[42.89077377319336, 71.30916595458984, 33.66455841064453]]
The results of other models are retrieved from MTEB leaderboard. Given that all models in the gte-modernbert series have a size of less than 1B parameters, we focused exclusively on the results of models under 1B from the MTEB leaderboard.
We have open positions for Research Interns and Full-Time Researchers to join our team at Tongyi Lab. We are seeking passionate individuals with expertise in representation learning, LLM-driven information retrieval, Retrieval-Augmented Generation (RAG), and agent-based systems. Our team is located in the vibrant cities of Beijing and Hangzhou. If you are driven by curiosity and eager to make a meaningful impact through your work, we would love to hear from you. Please submit your resume along with a brief introduction to dingkun.ldk@alibaba-inc.com.
If you find our paper or models helpful, feel free to give us a cite.
@inproceedings{zhang2024mgte,
title={mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval},
author={Zhang, Xin and Zhang, Yanzhao and Long, Dingkun and Xie, Wen and Dai, Ziqi and Tang, Jialong and Lin, Huan and Yang, Baosong and Xie, Pengjun and Huang, Fei and others},
booktitle={Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track},
pages={1393--1412},
year={2024}
}
@article{li2023towards,
title={Towards general text embeddings with multi-stage contrastive learning},
author={Li, Zehan and Zhang, Xin and Zhang, Yanzhao and Long, Dingkun and Xie, Pengjun and Zhang, Meishan},
journal={arXiv preprint arXiv:2308.03281},
year={2023}
}
Files
19 files · 2.53 GBWeights 298 MB (safetensors). Sizes and sha256 from the signed manifest.
import os
os.environ["HF_ENDPOINT"] = "https://freeweights.allternit.com"
from transformers import AutoModel, AutoTokenizer
tok = AutoTokenizer.from_pretrained("Alibaba-NLP/gte-modernbert-base", revision="e7f32e3c00f91d699e8c43b53106206bcc72bb22")
model = AutoModel.from_pretrained("Alibaba-NLP/gte-modernbert-base", revision="e7f32e3c00f91d699e8c43b53106206bcc72bb22", device_map="auto")
sentence-transformersEmbed text with sentence-transformers.
import os
os.environ["HF_ENDPOINT"] = "https://freeweights.allternit.com"
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("Alibaba-NLP/gte-modernbert-base", revision="e7f32e3c00f91d699e8c43b53106206bcc72bb22")
vectors = model.encode(["Hello"])
unsloth 
mlx-community