moss-tts-local-transformer-4.55b-voice-acting-v2-sft-dpo

MOSS Voice-Acting 4.55B — SFT + DPO (full-parameter)#

Full-parameter preference tuning of the SFT model. All 4.13 B parameters were updated in both stages — this is not an adapter and not a merge of one. It is the SFT + DPO full track on the sample grid.

By Christoph Schuhmann and LAION.

Two training stages on top of laion/moss-tts-local-transformer-4.55b-voice-acting-v2:

1. SFT — all 4.13 B parameters, 3 epochs over 3,147,802 annotated utterances (…-v2-sft).
2. DPO — full-parameter preference tuning of that checkpoint on 7,410,723 preference pairs. A rank-64 LoRA alternative trained on the same data and objective is published separately as …-v2-dpo-lora.

In informal listening this is the best of the three (base / SFT / SFT+DPO) on most prompts.

What is and is not established#

The run stopped early, and that is worth knowing before you use this. It reached step 123 of 988 — one eighth of the planned epoch — wrote its checkpoint, and then died: an ALLREDUCE of 24,916,480 elements timed out after 600 s across 256 ranks immediately following that save, and NCCL aborted the job (EXIT=134). The checkpoint itself is complete and loads and generates normally; there is simply no later one. Full-parameter DPO here shards its Adam state with ZeroRedundancyOptimizer (a second frozen reference copy plus fp32 master weights, gradients and moments would otherwise need 74.3 GB before a single activation), and a ZeRO-sharded optimizer state is not written by save_state, so the run was never resumable either.

The DPO caveat, stated plainly. On this preference set, DPO over-optimises: preference accuracy climbs to 0.994 while reward(chosen) — the implicit reward on the preferred sequence — falls to −2.39, meaning the policy makes the good audio less likely too, only less so than the bad. This checkpoint's single evaluation reads val_loss = 0.9394, preference accuracy 0.8965, reward(chosen) = −0.264, reward(rejected) = −13.549. It is flagged unhealthy by that rule. Stopping at step 123 is, by accident, close to where the LoRA variant's checkpoint selection independently landed (step 246 of

1. — in the LoRA run reward(chosen) fell monotonically from −0.111 to −2.385 over the

following 615 steps, so a completed run would very likely have been worse, not better.

Preference accuracy is not a health metric here. It was highest exactly where the model was most degraded.

Training summary#

Stage 1 — SFT#

Every micro-batch was exactly half voice profiles / half real speech — the sampler alternates strictly rather than mixing in expectation (verified 0.5000 / 0.5000 on 20 real batches). Measured 7.75 distinct voices per 8-sample batch, no voice above 25 % of a batch.

Stage 2 — DPO (full-parameter)#

Length normalisation is load-bearing: 85.6 % of the raw preference pairs differ from the chosen sequence mainly in LENGTH, and an unnormalised summed log-probability reached 1.000 preference accuracy at step 102 of 1194 by counting tokens. The length families were also downsampled to length_keep = 0.16, which keeps all 1,064,594 emotion pairs and raises emotion from 14.4 % to ≈51 % of the mix.

The anchor weight 0.013 is not a guess: -logp is ≈54 nats per position here, because each supervised position carries 13 channels (12 codebooks + the binary stop head). A first attempt at 0.25 put the anchor at ≈13.6 against a DPO term of 0.69 and turned the run into an SFT pass.

Prompt format#

Identical to the SFT model's — the DPO stage used the same prompt library (format hash 3d8a696ccec4a98f), which is what lets the two stages compose at all.

<user_inst>
- Reference(s):
{<|audio|>  |  Speaker: <name>  |  None}
- Instruction:
{caption}
- Tokens:
{target frame count, 12.5 fps}
- Quality:
None
- Sound Event:
None
- Ambient Sound:
None
- Language:
{English | German}
- Text:
{[7.3 seconds duration] }{transcript}
</user_inst>

The duration tag is spoken duration — last word offset minus first word onset — not clip duration. Clip duration overstates it by ~1.05 s on voice profiles and ~0.40 s on real speech because of leading and trailing silence; tag with clip duration and the model will pad.

Reference audio is passed as codec codes, [T, 12] int64, in audio_codes_list. No audio tokenizer is needed at inference for conditioning.

Usage#

import torch
from transformers import AutoProcessor, AutoModel

REPO  = "laion/moss-tts-local-transformer-4.55b-voice-acting-v2-sft-dpo"
CODEC = "OpenMOSS-Team/MOSS-Audio-Tokenizer-v2"

proc  = AutoProcessor.from_pretrained(REPO, trust_remote_code=True, codec_path=CODEC)
model = AutoModel.from_pretrained(REPO, trust_remote_code=True,
                                  dtype=torch.bfloat16,
                                  attn_implementation="sdpa").cuda().eval()

msg = proc.build_user_message(
    text="[4.2 seconds duration] I really did not see that coming.",
    instruction="GENERAL: A young adult feminine voice; delivery is bright and quick; "
                "affect is positive, animated.\nSCRIPT:\n\"I really did not see that coming.\"",
    tokens=53,                      # 4.2 s x 12.5 fps
    language="English")

batch = proc([[msg]], mode="generation")
out = model.generate(input_ids=batch["input_ids"].cuda(),
                     attention_mask=batch["attention_mask"].cuda(),
                     max_new_frames=400, do_sample=True,
                     audio_temperature=1.0, audio_top_p=0.95, audio_top_k=50,
                     audio_repetition_penalty=1.0)

wav = proc.decode(out)[0].audio_codes_list[0]        # 48 kHz waveform
import soundfile as sf
sf.write("out.wav", wav.float().cpu().numpy(), int(proc.model_config.sampling_rate))

Cloning a voice from a reference clip#

codes = proc.audio_tokenizer.encode(wav48k)          # [T, 12] int64 at 12.5 fps
msg = {"role": "user",
       "content": prompt_text_with_audio_placeholder, # "<|audio|>" in the Reference(s) slot
       "audio_codes_list": [codes]}

Keep the reference under ~25 s; the training pipeline capped references at 320 frames.

Naming a voice instead#

Put Speaker: <name> in the Reference(s) slot. The 500 profile names the model was trained on are listed with laion/moss-voice-profile-loras-500; a name outside that set is out of distribution.

Training data#

Annotation source: laion/laion-voice-profiles-annotated.

If you want to merge the LoRA variant instead — do not use merge_and_unload()#

Measured, because it is a silent failure. The 12 audio_lm_heads are weight-tied to the audio embeddings — tie_weights() assigns head.weight = embedding.weight, the same tensor object. The rank-64 adapter targets those heads, so folding W ← W + BA writes the head update into the embeddings as well:

audio_lm_heads.0.weight    max|ΔW| = 6.103515625e-05     (the merge)
audio_embeddings.0.weight  max|ΔW| = 6.103515625e-05     (same tensor)
transformer.embed_tokens.weight  max|ΔW| = 0.0           (not a LoRA target, correct)

The result loads, generates, and sounds like speech — and is a different function from the adapter: on the same input, the final hidden state differs with rms 0.107 against a signal rms of 2.518, 4.3 %. Use the adapter unmerged, or use this full-parameter model.

Known faults, and what is being done about them#

Both are audible and both have an identified cause:

• Vocal bursts run too long. The training prompts named a burst ((sigh)) without ever saying how long it should last, so nothing constrains its length. Additionally, 39.4 % of the parenthetical cues in the voice-profile corpus are synthesis directions the burst detector never confirmed — a cue with no corresponding audio.
• Occasional unprompted content. Better than the base model and better than SFT alone, but present. Nothing in the prompt states a per-utterance time budget.

A follow-up round replaces the single clip-level duration tag with a full timed script — a per-sentence [12.0 seconds duration], an explicit [0.6 seconds pause] for every silence over 200 ms, and (contented sigh, 0.4 seconds) for every burst — built from word-level timestamps and detected burst spans only.

Other limitations#

• English and German only.
• Validation loss is not audio quality. No MOS study, intelligibility benchmark or speaker-similarity re-measurement has been run.
• The SFT stage saw the synthetic half ≈3.93× and the real half ≈2.42×, a consequence of balancing every batch across two unequal datasets.
• The DPO stage covered 12.5 % of one epoch. Whatever it learned, it learned from 251,904 preference pairs, not 7.41 M.
• Inherits the corpora's biases in speaker demographics, recording conditions and emotion labels.
• The 12 audio LM heads are weight-tied to the audio embeddings, as in the base model.

Intended use#

Research on expressive and controllable speech synthesis. Not validated for, and should not be used for, generating speech attributed to a real person without their consent, or for any application where a synthetic voice could be mistaken for a specific individual.

Licence and attribution#

Released under CC-BY-4.0 by Christoph Schuhmann and LAION.

Derived from laion/moss-tts-local-transformer-4.55b-voice-acting-v2 (Apache-2.0), whose notice is preserved. The MOSS-TTS architecture and the MOSS-Audio-Tokenizer-v2 codec originate with the OpenMOSS team.

@misc{schuhmann2026mossva_sftdpo,
  title  = {MOSS Voice-Acting 4.55B --- SFT + DPO (full-parameter)},
  author = {Schuhmann, Christoph and {LAION}},
  year   = {2026},
  url    = {https://huggingface.co/laion/moss-tts-local-transformer-4.55b-voice-acting-v2-sft-dpo},
  note   = {Full fine-tune on 3.15M utterances plus full-parameter DPO; CC-BY-4.0}
}