moss-tts-local-transformer-4.55b-voice-acting-v2-sft3

MOSS voice-acting v2 — SFT round 3#

A supervised fine-tune of laion/moss-tts-local-transformer-4.55b-voice-acting-v2 that adds explicit control over timing — per-sentence durations, pauses, and vocal bursts with their lengths — and inline delivery directions that say how to perform each line.

4.13 B trainable parameters: a ~4 B semantic transformer (36 layers), a ~550 M local "talker" transformer, and 12 audio LM heads over a 12-codebook audio tokenizer at 12.5 frames per second (one frame = 80 ms).

What this round changed, measured#

Trained on 398,282 rows selected as the strongest examples of each of 40 emotions and each VoiceNet dimension, 2 epochs, 712 steps on 32 nodes. Evaluated by generating 320 clips and scoring them, not by validation loss — this project has repeatedly seen training metrics point the wrong way.

Round 2 had accidentally dropped the delivery directions its predecessor was trained with, and a model that has never seen a direction falls apart when given one — word error rate 0.48–0.51 on such prompts, for every round-2 model. Round 3 trained them back in. Timing control is solved.

Emotional intensity is not. Asked for percentile 0.90–0.98 of a named emotion, this model reaches about 0.35. Several objectives were tried against that — GRPO with a group-relative reward, DPO with contrastive pairs, DPO with symmetric instruction-conditioned pairs — and none moved it. The emotion adapters below are the only thing that has, and even they are not selective enough to merge in blindly. This is documented honestly in the technical report.

Inference#

import torch, torchaudio
from transformers import AutoProcessor, AutoModel

BASE = "laion/moss-tts-local-transformer-4.55b-voice-acting-v2-sft3"

proc = AutoProcessor.from_pretrained(BASE, trust_remote_code=True)
model = AutoModel.from_pretrained(BASE, trust_remote_code=True,
                                  dtype=torch.bfloat16, attn_implementation="sdpa").cuda().eval()

prompt = open("prompt.txt").read()          # the <user_inst> block, see "How to prompt" below
um = {"role": "user", "content": prompt, "audio_codes_list": []}
b = proc([[um]], mode="generation")

with torch.no_grad():
    out = model.generate(input_ids=b["input_ids"].cuda(),
                         attention_mask=b["attention_mask"].cuda(),
                         max_new_frames=340, do_sample=True,
                         audio_temperature=1.0, audio_top_p=0.95, audio_top_k=50,
                         audio_repetition_penalty=1.0)

# codes -> waveform. Use the processor's own decoder: calling the audio tokenizer directly, or
# reshaping its output, yields a two-channel result that flattens into audio at HALF SPEED and
# still sounds like speech. This project lost a whole corpus to that once.
wav = proc.decode_audio_codes([out_codes], return_stereo=False)[0].reshape(-1).float().cpu()
torchaudio.save("out.flac", wav[None], int(proc.model_config.sampling_rate), format="flac")

Loading adapters#

from peft import PeftModel

# one adapter
model = PeftModel.from_pretrained(model, "laion/moss-va-sft3-dpo-lora")

# several, each with its own weight -- the usual case: identity from a voice adapter,
# affect from an emotion adapter, general quality from the DPO adapter.
#
# NOTE: `add_weighted_adapter(..., combination_type="linear")` does NOT work here. It raises
# `ValueError: All adapters must have the same r value`, because the DPO adapter is rank 64 and
# the voice / emotion adapters are rank 16. Activate them together instead and scale each one.
model = PeftModel.from_pretrained(model, "<dpo adapter path>", adapter_name="dpo")
model.load_adapter("<voice adapter path>", adapter_name="voice")
model.load_adapter("<emotion adapter path>", adapter_name="emo")

names = ["dpo", "voice", "emo"]
weights = {"dpo": 1.0, "voice": 1.0, "emo": 1.5}     # 1.5 for emotion is the measured optimum
model.base_model.set_adapter(names)                  # the TUNER takes a list; PeftModel does not
model.active_adapter = names[0]                      # must stay a str or generate() indexes a list

for mod in model.modules():
    sc = getattr(mod, "scaling", None)
    if isinstance(sc, dict):
        if not hasattr(mod, "_base_scaling"):
            mod._base_scaling = dict(sc)
        for k in sc:
            if k in weights:
                sc[k] = mod._base_scaling[k] * weights[k]

Scaling an adapter without re-merging#

A LoRA layer computes h + scaling · B(A(x)), so multiplying the stored scaling is the merge weight — exact and reversible:

def set_lora_scale(model, w):
    for mod in model.modules():
        sc = getattr(mod, "scaling", None)
        if isinstance(sc, dict):
            if not hasattr(mod, "_base_scaling"):
                mod._base_scaling = dict(sc)
            for k in sc:
                sc[k] = mod._base_scaling[k] * w

How to prompt this model#

Every request is one <user_inst> block. The fields are fixed — none may be added or removed:

<user_inst>
- Reference(s):
{None | Speaker: <name> | <|audio|>}
- Instruction:
{GENERAL: ... and/or SCRIPT: ...}
- Tokens:
{target length in audio frames}
- Quality:
None
- Sound Event:
None
- Ambient Sound:
None
- Language:
{English | German}
- Text:
{the same script as under SCRIPT:, character for character}
</user_inst>

GENERAL: — who is speaking#

Prose describing the voice and the clip: age and gender, energy and pace, tension, timbre, clarity, pitch range, breath, affect, which emotions are audible, style, recording quality.

GENERAL: A young adult masculine voice; delivery is normally alert, brisk, neutral tension;
timbre is neutral-toned, fairly smooth; average clarity, wide pitch range, light breath;
affect is mildly positive, slightly dominant; reads as bitterness, contempt; 9.5s, EN.

The phrase reads as … is where the emotion names live.

SCRIPT: — what to say, when, and how#

Four kinds of tag, told apart by their brackets:

The disambiguation rule in one line: square bracket = a number of seconds; round bracket with a number = a vocal burst; round bracket without a number = a delivery direction. That is the only thing separating a burst from a direction, which is why directions never carry a number.

A complete example:

<user_inst>
- Reference(s):
None
- Instruction:
GENERAL: A young adult feminine voice, warm and conversational; reads as amusement; 6.0s, EN.
SCRIPT:
[0.4 seconds pause] (intensely amused: letting it out / not hiding it, warm and open,
unguarded; bright, relaxed) [2.4 seconds duration] You are not going to believe this.
[0.3 seconds pause] (breathy giggle, 0.4 seconds) [0.2 seconds pause] (still intensely amused)
[2.3 seconds duration] He actually wore it to the wedding.
- Tokens:
75
- Quality:
None
- Sound Event:
None
- Ambient Sound:
None
- Language:
English
- Text:
[0.4 seconds pause] (intensely amused: letting it out / not hiding it, warm and open,
unguarded; bright, relaxed) [2.4 seconds duration] You are not going to believe this.
[0.3 seconds pause] (breathy giggle, 0.4 seconds) [0.2 seconds pause] (still intensely amused)
[2.3 seconds duration] He actually wore it to the wedding.
</user_inst>

0.4 + 2.4 + 0.3 + 0.4 + 0.2 + 2.3 = 6.0 s = 75 frames. If the numbers do not add up to the token budget the model has to choose which to honour, and length control is the thing it honours best.

Segmentation rules the training data followed#

• split at sentence ends (. ! ? …) and at every vocal burst;
• any segment still longer than 12 s is split again at its largest internal gap;
• a duration is measured from the first word onset to the last word offset of that segment, so two sentences of 12 s and 8 s produce [12.0 seconds duration] and [8.0 seconds duration], never a single [20.0 seconds duration];
• gaps below 0.20 s are folded into the neighbouring speech instead of being printed, so the printed numbers still add up;
• a burst that overlaps speech prints only the part that does not overlap.

Vocal-burst labels that actually occur in training#

low mumble, ahem, contented sigh, surprised gasp, chuckle, breathy giggle, childlike giggle, wistful sigh, exhausted groan, sharp inhale, resonant hum, scream, yawn, deep breath, soft hum, exasperated sigh, cackle, shriek, coughing, mournful wail, growl, purr.

Realistic lengths: median 0.28 s, 10th percentile 0.14 s, 90th percentile 0.48 s, longest observed 2.46 s. A sigh requested at 3 s is outside anything in the data.

Intensity bands#

Delivery directions carry an intensity adverb drawn from the percentile band of the requested emotion. The same cutoffs are used by the training data, the reward and the evaluation:

The family#

Citation and provenance#

Derived from MOSS-TTSD / MOSS local-transformer 1.5. Training data: LAION voice profiles plus public real-speech corpora (EmoLia, Kartoffelphon, MLS). Released CC-BY-4.0.

Recipes, prompting and the rest of the stack#

This is the base model the adapter sets below are trained against. What you can ask it for, and at what weight, is written down rather than left to trial and error.

Where the recipes live#

Writing the cue#

Brackets carry meaning in this format, and two rules catch nearly everyone out.

Cues are always written in English, even when the spoken line is German. This is how the training corpus is written — German rows read Das zerreißt einen einfach, weißt du? (relief sigh) — so a German cue is out of distribution.

Never write a number inside a bracket. The durations are computed for you and inserted afterwards: [N.N seconds duration] before each speech segment, [N.N seconds pause] for gaps, (label, N.N seconds) for each burst, summed at 12.5 frames per second.