moss-tts-local-transformer-4.55b-voice-acting-v2

MOSS-TTS-Local 4.55B — Voice Acting v2 (48 kHz local transformer)#

An expressive voice-acting model: fantasy characters (orc / dragon / fairy / goblin), shouting, whispering, and vocal bursts (laughs, gasps, sighs, giggles, moans) driven by natural-language director instructions, in English and German.

Authors: Christoph Schuhmann and LAION.

What is new in v2#

This checkpoint is the latest and best in the line. It was obtained by taking the previous release — laion/moss-tts-local-transformer-4.55b-voice-acting — and running further supervised fine-tuning on additional synthetic voice-acting data. The extra data was self-generated with the model family and filtered with a Best-of-N reward (a lightweight VoiceCLAP-based genuineness + vocal-burst-blend + character reward that keeps only the strongest of many candidate takes per prompt), so the model learns from its own highest-quality performances — more consistent character delivery, more natural vocal bursts, and cleaner emotional range. It remains a full 4.55-billion-parameter local-transformer MOSS variant at 48 kHz and is a drop-in replacement for the previous release (same architecture, tokenizer, and prompt format).

Its sibling is the 8-billion-parameter delay-architecture model at 24 kHz: laion/moss-tts-v1.5-8b-voice-acting.

Off-the-shelf: this repository contains the fully merged weights (bf16). Load and generate; no adapter handling needed.

Further fine-tuning (v2)#

• Additional data: a large corpus of synthetically generated, Best-of-N-filtered voice-acting performances, self-distilled from the model family — structured natural-language director instructions paired with spoken lines, covering fantasy characters, strong emotions, and vocal bursts (laughs, gasps, sighs, moans) in English and German. No third-party dataset names are disclosed.
• Objective / filtering: candidates were ranked by a cheap automatic reward (VoiceCLAP-commercial embedding → genuineness + vocal-burst-blend + character probes) and only the top takes were kept, so the model is trained on its own strongest performances rather than raw generations.
• Setup: 8×GPU DDP, fp32 master weights + 8-bit optimizer with bf16 compute (avoids the update-stagnation that pure-bf16 parameter fine-tuning suffers), gradient checkpointing, per-sample caption augmentation (randomly dropping the general voice description, the inline stage directions, or a fraction of the inline cues/pauses so the model learns to improvise them) and reference-audio dropout, linear LR schedule with warmup, channel-wise RVQ loss weighting. Released weights are the best checkpoint by cumulative genuineness + vocal-burst-blend on a held-out validation set.

Requirements#

pip install "transformers>=4.45" torch torchaudio safetensors huggingface_hub

The MOSS v2 audio codec (OpenMOSS-Team/MOSS-Audio-Tokenizer-v2) is pulled automatically. ~10 GB VRAM (bf16 + codec).

Prompt format — GENERAL: / SCRIPT: + plain text#

The model was trained on two fields per example:

• instruction = the performance direction, written as two labelled blocks:
  • GENERAL: — the overall voice: age / gender / timbre / mood / delivery. For the most natural result, also state that it is a pristine, high-quality studio recording with no background noise and a genuine, spontaneous delivery ("like a real person in a real moment, not acted").
  • SCRIPT: — the spoken lines again, annotated in position with (delivery cues), vocal bursts such as (a soft laugh) / (gasp) / (sigh), and [pause] markers. The emotional arc can change line-by-line here — this is where you direct the performance.
• text = just the plain spoken words — no cues, no brackets.

The words appear in both fields (plain in text, cue-annotated inside the SCRIPT block) — this is intentional and matches the training data. These two strings map directly onto generate(text=..., instruction=...) in the inference example below.

Exact template#

instruction =
GENERAL: <one or two sentences: voice / age / gender / timbre / mood / delivery arc; note "pristine high-quality studio recording, genuine and spontaneous">
SCRIPT:
(cue, cue, ...) first line of dialogue [pause] rest of the line
(cue, cue, ...) second line of dialogue
...

text =
<exactly the same spoken words, plain — no cues, no (), no []>

Example (an emotional arc: warm → shocked → furious)#

instruction =
GENERAL: A young man's voice — bright, warm and overjoyed, then it curdles into shocked disbelief and finally erupts into loud, aggressive, disgusted ranting. Completely natural and spontaneous, like a real person in a real moment, not acted. Pristine high-quality studio recording, no background noise.
SCRIPT:
(lighting up, warm, breathless with delight) Oh my god— hey! [pause] Is that really you?
(beaming, affectionate, a soft happy laugh) I can't believe it, look at you!
(the smile fading, confused, voice tightening) ...Wait. What is that?
(erupting, very loud, aggressive, furious and betrayed) You actually did it?! After everything I did for you?!
(ranting fast, seething, disgusted) God, it makes me sick. Get out of my sight.

text =
Oh my god— hey! Is that really you? I can't believe it, look at you! ...Wait. What is that? You actually did it?! After everything I did for you?! God, it makes me sick. Get out of my sight.

Tips: keep each (cue) short and place it right before the words it affects; put [pause] where you want a real beat; describe loud/aggressive/whispered etc. explicitly in the cue to push intensity. For a live, click-free streaming demo of this exact format, see laion/moss-voice-acting-playground.

Inference (complete)#

import torch, torchaudio
from transformers import AutoProcessor, AutoModel

REPO = "laion/moss-tts-local-transformer-4.55b-voice-acting-v2"
DEVICE = "cuda"

proc = AutoProcessor.from_pretrained(REPO, trust_remote_code=True,
                                     codec_path="OpenMOSS-Team/MOSS-Audio-Tokenizer-v2")
proc.audio_tokenizer = proc.audio_tokenizer.to(DEVICE).eval()
model = AutoModel.from_pretrained(REPO, trust_remote_code=True,
                                  dtype=torch.bfloat16,
                                  attn_implementation="sdpa").to(DEVICE).eval()

@torch.no_grad()
def generate(text, instruction, language="English", reference_codes=None, n=3):
    kw = dict(text=text, instruction=instruction, language=language)
    if reference_codes is not None:
        kw["reference"] = [torch.tensor(reference_codes, dtype=torch.long)]
    best = None
    for seed in range(n):
        torch.manual_seed(seed)
        batch = proc([[proc.build_user_message(**kw)]], mode="generation")
        out = model.generate(input_ids=batch["input_ids"].to(DEVICE),
                             attention_mask=batch["attention_mask"].to(DEVICE),
                             max_new_tokens=1200, do_sample=True,
                             audio_temperature=1.0, audio_top_p=0.95,
                             audio_top_k=25, audio_repetition_penalty=1.1)
        msg = proc.decode(out)[0]
        if not msg.audio_codes_list:
            continue
        wav = msg.audio_codes_list[0]                    # waveform @ 48 kHz
        if best is None or wav.shape[-1] > best.shape[-1]:
            best = wav
    return best

wav = generate(
    text="You dare enter my domain? Then you shall not leave alive!",
    instruction="As a mighty orc warrior with a deep, guttural, growling voice, "
                "bellow these words with brutal force.",
    language="English")
torchaudio.save("orc.wav", wav.cpu().float(), 48000)

Tips (measured on the model family): batch by repeating input_ids n× and calling generate once (batch 64 → ~0.87 s/clip on an A100); use SDPA (flash-attn 2.x is incompatible with this remote-code attention); without reference audio use audio_temperature=0.8; pass tokens ≈ words × 6 (12.5 Hz codec frames) via build_user_message to avoid rushed pacing. Generating N candidates and keeping the best (as in generate(..., n=...) above) is how the v2 training data itself was produced and is the recommended way to get top-quality takes.

Optional: clone a reference voice#

Pass 12-codebook MOSS codes of a reference clip to steer timbre; keep the instruction for the delivery (an empty instruction gives the highest speaker similarity):

ref_codes = proc.encode_audios_from_path(["my_reference.wav"], n_vq=12)[0]
wav = generate(text="...", instruction="", language="English", reference_codes=ref_codes)

⚡ Serving with SGLang-Omni (fast inference)#

For interactive or bulk dataset generation, serve this model with SGLang-Omni — an OpenAI-compatible speech server with continuous batching + CUDA graphs (measured on the model family at ~35× realtime for offline data-gen on 1× A100-80GB).

uv pip install sglang-omni soundfile
hf download laion/moss-tts-local-transformer-4.55b-voice-acting-v2

# NOTE the `env -u LD_LIBRARY_PATH` — without it the FIRST request SIGABRTs with
# `undefined symbol: cudnnGetLibConfig` (a leaked cuDNN path in the stage subprocess).
env -u LD_LIBRARY_PATH CUDA_VISIBLE_DEVICES=0 \
  sgl-omni serve \
    --model-path laion/moss-tts-local-transformer-4.55b-voice-acting-v2 \
    --config examples/configs/moss_tts_local.yaml --port 8000

curl -X POST http://localhost:8000/v1/audio/speech -H "Content-Type: application/json" \
  -d '{"input": "You dare enter my lair?!", "instructions": "A furious ancient dragon, low and booming."}' \
  --output dragon.wav

Notes#

• The text_lm_head / audio_lm_heads "MISSING" note on load is benign weight-tying.
• Same architecture, tokenizer and prompt format as the previous release — you can swap the repo id in existing code with no other changes.
• License: Apache-2.0.

Previous version & background#

The v1 release, plus the full experiment write-up (temperature/best-of-N studies, reward variants, the character-archetype prompt studies and prompting playbook, fast-inference notes and audio demos), lives here: laion/moss-tts-local-transformer-4.55b-voice-acting · experiment overview · GitHub.