Ruri v3 is a general-purpose Japanese text embedding model built on top of ModernBERT-Ja. Ruri v3 offers several key technical advantages:
State-of-the-art performance for Japanese text embedding tasks.
Supports sequence lengths up to 8192 tokens
Previous versions of Ruri (v1, v2) were limited to 512.
Expanded vocabulary of 100K tokens, compared to 32K in v1 and v2
The larger vocabulary make input sequences shorter, improving efficiency.
Integrated FlashAttention, following ModernBERT's architecture
Enables faster inference and fine-tuning.
Tokenizer based solely on SentencePiece
Unlike previous versions, which relied on Japanese-specific BERT tokenizers and required pre-tokenized input, Ruri v3 performs tokenization with SentencePiece only—no external word segmentation tool is required.
Additionally, if your GPUs support Flash Attention 2, we recommend using our models with Flash Attention 2.
pip install flash-attn --no-build-isolation
Then you can load this model and run inference.
import torch
import torch.nn.functional as F
from sentence_transformers import SentenceTransformer
# Download from the 🤗 Hub
device = "cuda" if torch.cuda.is_available() else "cpu"
model = SentenceTransformer("cl-nagoya/ruri-v3-310m", device=device)
# Ruri v3 employs a 1+3 prefix scheme to distinguish between different types of text inputs:
# "" (empty string) is used for encoding semantic meaning.
# "トピック: " is used for classification, clustering, and encoding topical information.
# "検索クエリ: " is used for queries in retrieval tasks.
# "検索文書: " is used for documents to be retrieved.
sentences = [
"川べりでサーフボードを持った人たちがいます",
"サーファーたちが川べりに立っています",
"トピック: 瑠璃色のサーファー",
"検索クエリ: 瑠璃色はどんな色？",
"検索文書: 瑠璃色（るりいろ）は、紫みを帯びた濃い青。名は、半貴石の瑠璃（ラピスラズリ、英: lapis lazuli）による。JIS慣用色名では「こい紫みの青」（略号 dp-pB）と定義している[1][2]。",
]
embeddings = model.encode(sentences, convert_to_tensor=True)
print(embeddings.size())
# [5, 768]
similarities = F.cosine_similarity(embeddings.unsqueeze(0), embeddings.unsqueeze(1), dim=2)
print(similarities)
# [[1.0000, 0.9603, 0.8157, 0.7074, 0.6916],
# [0.9603, 1.0000, 0.8192, 0.7014, 0.6819],
# [0.8157, 0.8192, 1.0000, 0.8701, 0.8470],
# [0.7074, 0.7014, 0.8701, 1.0000, 0.9746],
# [0.6916, 0.6819, 0.8470, 0.9746, 1.0000]]
@misc{
Ruri,
title={{Ruri: Japanese General Text Embeddings}},
author={Hayato Tsukagoshi and Ryohei Sasano},
year={2024},
eprint={2409.07737},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2409.07737},
}
import os
os.environ["HF_ENDPOINT"] = "https://freeweights.allternit.com"
from transformers import AutoModel, AutoTokenizer
tok = AutoTokenizer.from_pretrained("cl-nagoya/ruri-v3-310m", revision="18b60fb8c2b9df296fb4212bb7d23ef94e579cd3")
model = AutoModel.from_pretrained("cl-nagoya/ruri-v3-310m", revision="18b60fb8c2b9df296fb4212bb7d23ef94e579cd3", device_map="auto")
sentence-transformersEmbed text with sentence-transformers.
import os
os.environ["HF_ENDPOINT"] = "https://freeweights.allternit.com"
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("cl-nagoya/ruri-v3-310m", revision="18b60fb8c2b9df296fb4212bb7d23ef94e579cd3")
vectors = model.encode(["Hello"])
unsloth 
mlx-community