We are excited to introduce the gte-modernbert series of models, which are built upon the latest modernBERT pre-trained encoder-only foundation models. The gte-modernbert series models include both text embedding models and rerank models.
The gte-modernbert models demonstrates competitive performance in several text embedding and text retrieval evaluation tasks when compared to similar-scale models from the current open-source community. This includes assessments such as MTEB, LoCO, and COIR evaluation.
Tip. For transformers and sentence-transformers, if your GPU supports it, the efficient Flash Attention 2 will be used automatically if you have flash_attn installed. It is not mandatory.
pip install flash_attn
Use with transformers
# Requires transformers>=4.48.0
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
model_name_or_path = "Alibaba-NLP/gte-reranker-modernbert-base"
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)
model = AutoModelForSequenceClassification.from_pretrained(
model_name_or_path,
torch_dtype=torch.float16,
)
model.eval()
pairs = [
["what is the capital of China?", "Beijing"],
["how to implement quick sort in python?", "Introduction of quick sort"],
["how to implement quick sort in python?", "The weather is nice today"],
]
with torch.no_grad():
inputs = tokenizer(pairs, padding=True, truncation=True, return_tensors='pt', max_length=512)
scores = model(**inputs, return_dict=True).logits.view(-1, ).float()
print(scores)
# tensor([ 2.1387, 2.4609, -1.6729])
Use with sentence-transformers:
Before you start, install the sentence-transformers libraries:
pip install sentence-transformers
# Requires transformers>=4.48.0
from sentence_transformers import CrossEncoder
model = CrossEncoder(
"Alibaba-NLP/gte-reranker-modernbert-base",
automodel_args={"torch_dtype": "auto"},
)
pairs = [
["what is the capital of China?", "Beijing"],
["how to implement quick sort in python?","Introduction of quick sort"],
["how to implement quick sort in python?", "The weather is nice today"],
]
scores = model.predict(pairs)
print(scores)
# [0.8945664 0.9213594 0.15742092]
# NOTE: Sentence Transformers calls Softmax over the outputs by default, hence the scores are in [0, 1] range.
Use with transformers.js
import {
AutoTokenizer,
AutoModelForSequenceClassification,
} from "@huggingface/transformers";
const model_id = "Alibaba-NLP/gte-reranker-modernbert-base";
const model = await AutoModelForSequenceClassification.from_pretrained(
model_id,
{ dtype: "fp32" }, // Supported options: "fp32", "fp16", "q8", "q4", "q4f16"
);
const tokenizer = await AutoTokenizer.from_pretrained(model_id);
const pairs = [
["what is the capital of China?", "Beijing"],
["how to implement quick sort in python?", "Introduction of quick sort"],
["how to implement quick sort in python?", "The weather is nice today"],
];
const inputs = tokenizer(
pairs.map((x) => x[0]),
{
text_pair: pairs.map((x) => x[1]),
padding: true,
truncation: true,
},
);
const { logits } = await model(inputs);
console.log(logits.tolist()); // [[2.138258218765259], [2.4609625339508057], [-1.6775450706481934]]
The results of other models are retrieved from MTEB leaderboard. Given that all models in the gte-modernbert series have a size of less than 1B parameters, we focused exclusively on the results of models under 1B from the MTEB leaderboard.
We have open positions for Research Interns and Full-Time Researchers to join our team at Tongyi Lab. We are seeking passionate individuals with expertise in representation learning, LLM-driven information retrieval, Retrieval-Augmented Generation (RAG), and agent-based systems. Our team is located in the vibrant cities of Beijing and Hangzhou. If you are driven by curiosity and eager to make a meaningful impact through your work, we would love to hear from you. Please submit your resume along with a brief introduction to dingkun.ldk@alibaba-inc.com.
If you find our paper or models helpful, feel free to give us a cite.
@inproceedings{zhang2024mgte,
title={mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval},
author={Zhang, Xin and Zhang, Yanzhao and Long, Dingkun and Xie, Wen and Dai, Ziqi and Tang, Jialong and Lin, Huan and Yang, Baosong and Xie, Pengjun and Huang, Fei and others},
booktitle={Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track},
pages={1393--1412},
year={2024}
}
@article{li2023towards,
title={Towards general text embeddings with multi-stage contrastive learning},
author={Li, Zehan and Zhang, Xin and Zhang, Yanzhao and Long, Dingkun and Xie, Pengjun and Zhang, Meishan},
journal={arXiv preprint arXiv:2308.03281},
year={2023}
}
Files
15 files · 2.54 GBWeights 598 MB (safetensors). Sizes and sha256 from the signed manifest.
unsloth 
mlx-community