The merged model scores above the average of its two parents on 7 of the 11 NanoBEIR datasets, including 2 where it beats both. It achieves a mean nDCG@10 of 0.6981, compared with 0.6705 for BGE-reranker-large and 0.6929 for BGE-reranker-v2-m3, using the same inference settings for all three models.
On the seven datasets evaluated after candidate selection, the merge scores 0.6919, compared with 0.6911 for BGE-reranker-large and 0.6876 for M3. This additional-seven gain is small. Its paired query bootstrap intervals against both parents include zero.
from sentence_transformers import CrossEncoder
model = CrossEncoder("cross-encoder/merged-bge-reranker-large")
queries = ['Which planet is known as the Red Planet?']
documents = [
"Venus is often called Earth's twin because of its similar size and proximity.",
'Mars, known for its reddish appearance, is often referred to as the Red Planet.',
'Saturn, famous for its rings, is sometimes mistaken for the Red Planet.',
]
scores = model.predict([(queries[0], document) for document in documents])
print(scores)
# [-0.08445622 3.0177891 1.9747379 ]
The complete XLM-RoBERTa backbone and classification head were merged in fp32 with equal weights, without any additional training. M3's position embedding table and matching buffers were first cropped from 8194 to 514 rows, covering the 512-token joint input limit. This crop preserves M3's logits at that input limit. The source M3 model supports 8192-token inputs. Both parents were evaluated at the same 512-token limit as the merge.
This is all it took:
# pip install "sentence-transformers[merge]"
import torch
from sentence_transformers import CrossEncoder
large = CrossEncoder("BAAI/bge-reranker-large")
m3 = CrossEncoder("BAAI/bge-reranker-v2-m3")
large.max_seq_length = m3.max_seq_length = 512
embeddings = m3.model.roberta.embeddings
embeddings.position_embeddings = torch.nn.Embedding.from_pretrained(
embeddings.position_embeddings.weight[:514].clone(), freeze=False, padding_idx=1,
)
for name in ("position_ids", "token_type_ids"):
setattr(embeddings, name, getattr(embeddings, name)[:, :514].clone())
m3.config.max_position_embeddings = 514
model = CrossEncoder.merge([large, m3], method="linear", parameters={"weight": [0.5, 0.5]}, dtype="float32")
model.activation_fn = torch.nn.Identity()
model.save_pretrained("merged-bge-reranker-large")
@misc{li2023making,
title={Making Large Language Models A Better Foundation For Dense Retrieval},
author={Chaofan Li and Zheng Liu and Shitao Xiao and Yingxia Shao},
year={2023},
eprint={2312.15503},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
@misc{chen2024bge,
title={BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation},
author={Jianlv Chen and Shitao Xiao and Peitian Zhang and Kun Luo and Defu Lian and Zheng Liu},
year={2024},
eprint={2402.03216},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
@inproceedings{goddard-etal-2024-arcees,
title = "Arcee{'}s {M}erge{K}it: A Toolkit for Merging Large Language Models",
author = "Goddard, Charles and Siriwardhana, Shamane and Ehghaghi, Malikeh and Meyers, Luke and Karpukhin, Vladimir and Benedict, Brian and McQuade, Mark and Solawetz, Jacob",
editor = "Dernoncourt, Franck and Preo{\c{t}}iuc-Pietro, Daniel and Shimorina, Anastasia",
booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track",
month = nov,
year = "2024",
address = "Miami, Florida, US",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.emnlp-industry.36",
doi = "10.18653/v1/2024.emnlp-industry.36",
pages = "477--485",
}
Files
9 files · 2.26 GBWeights 2.24 GB (safetensors). Sizes and sha256 from the signed manifest.
unsloth 
mlx-community