This is a pytorch version of the universal-sentence-encoder-cmlm/multilingual-base-br model. It can be used to map 109 languages to a shared vector space. As the model is based LaBSE, it perform quite comparable on downstream tasks.
from sentence_transformers import SentenceTransformer
sentences = ["This is an example sentence", "Each sentence is converted"]
model = SentenceTransformer('sentence-transformers/use-cmlm-multilingual')
embeddings = model.encode(sentences)
print(embeddings)
import os
os.environ["HF_ENDPOINT"] = "https://freeweights.allternit.com"
from transformers import AutoModel, AutoTokenizer
tok = AutoTokenizer.from_pretrained("sentence-transformers/use-cmlm-multilingual", revision="736a29561b4df04fc2d03bc7f6a574e4d0081b56")
model = AutoModel.from_pretrained("sentence-transformers/use-cmlm-multilingual", revision="736a29561b4df04fc2d03bc7f6a574e4d0081b56", device_map="auto")
sentence-transformersEmbed text with sentence-transformers.
import os
os.environ["HF_ENDPOINT"] = "https://freeweights.allternit.com"
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("sentence-transformers/use-cmlm-multilingual", revision="736a29561b4df04fc2d03bc7f6a574e4d0081b56")
vectors = model.encode(["Hello"])