voiceclap-large-v2

VoiceCLAP-Large-v2 (PCL)#

A rank-16 LoRA finetune of LCO-Embedding-Omni-7B (Qwen2.5-Omni thinker) trained with InfoNCE + Prototypical Contrastive loss (PCL) on the VoiceCLAP 9-corpus mix with MOSS-Audio k=2 sampled captions. Successor to laion/voiceclap-large — better on every benchmark we measure.

What's new vs voiceclap-large#

1. PCL auxiliary loss (weight 0.1): 39 learned emotion prototypes; cross-entropy of audio embeddings vs prototypes on pseudo-labeled clips.
2. z-scored pseudo-labels: emolia's emotion_annotation scalars, argmax over per-emotion z-scores vs corpus base rates (raw argmax is degenerate — high-base-rate dimensions win ~99% of clips). z ≥ 1.5 labels ~80-98% of emolia across all 39 emotion classes.
3. LoRA rank 16 (α=32) — rank shown equivalent to r=32 in a controlled A/B.

Evaluation (VoiceNet benchmark, human-annotated)#

Same-commit comparison on the VoiceNet harness:

Controlled A/B vs its exact no-PCL twin (identical data/recipe, ep1):

Ensemble notes: averaging this model's similarities with gijs/voiceclap-lco-7b-lora and the k=10 MOSS variant sets the current VoiceNet records (Emo bal@pp 0.7102; Ext bal@pp 0.6883).

Training recipe#

Quick start#

import torch
from sentence_transformers import SentenceTransformer

model = SentenceTransformer(
    "laion/voiceclap-large-v2",
    trust_remote_code=True,
    model_kwargs={"torch_dtype": torch.bfloat16},
)
audio_emb = model.encode("clip.flac")
text_emb  = model.encode("A person speaking with quiet pride in their voice")
score     = (audio_emb @ text_emb.T).item()

License#

cc-by-4.0

Confirmed by the rights holder (LAION) on 2026-09-09: the weights of this model are released under CC BY 4.0. This licence statement is the rights holder's determination for the released weights; see the training-data table above for the corpora used and their own licences. laion/voiceclap-commercial is offered for users who require a model whose training lineage contains no non-commercially licensed corpus at all.