moss-tts-local-transformer-4.55b-voice-acting-v2-sft

MOSS Voice-Acting 4.55B — SFT#

A full fine-tune of laion/moss-tts-local-transformer-4.55b-voice-acting-v2 on 3,147,802 annotated utterances — half synthetic voice profiles, half real speech in English and German — for three epochs.

By Christoph Schuhmann and LAION.

Held-out validation loss fell at every one of the twelve evaluations, on both constituent datasets independently: 4.7076 → 4.6314.

What changed relative to the base model#

This is not an adapter. All 4.13 B parameters were trained.

Learning rate, and why 7e-6. The reference trainer for this model family defaults to 1e-5, but that default is for a small fine-tune at an effective batch of ~32. Here an already-specialised checkpoint is fully fine-tuned over 9.4 M sample-passes at a global batch of 4,096. A long full-fine-tune schedule on a good checkpoint argues for a lower rate; a large batch (less gradient noise) argues for a higher one. 7e-6 is 0.7× the reference default.

Validation#

Held out by blake2b(uid) % 1000 == 0 — deterministic, disjoint from training, and covering both datasets. Loss is reported per dataset so a regression on one cannot hide behind a gain on the other.

Monotone throughout, no divergence between the two datasets, no sign of overfitting at three epochs. Voice profiles sit ~0.39 nats below real speech, which is expected — synthetic single-voice material is more predictable than recordings made in the wild.

Listen before you trust the number: the validation sample grid pairs the base model, this fine-tune and the ground truth on the same held-out prompts.

Prompt format#

The model expects the MOSS <user_inst> block. The exact prompt surface it was trained on matters — the training mix deliberately varied it, and departing from these conventions will cost you quality.

<user_inst>
- Reference(s):
{<|audio|>  |  Speaker: <name>  |  None}
- Instruction:
{caption}
- Tokens:
{target frame count, 12.5 fps}
- Quality:
None
- Sound Event:
None
- Ambient Sound:
None
- Language:
{English | German}
- Text:
{[7.3 seconds duration] }{transcript}
</user_inst>

What the training mix contained, and in what proportion:

The duration tag is spoken duration — last word offset minus first word onset — not clip duration. Clip duration overstates it by ~1.05 s on voice profiles and ~0.40 s on real speech because of leading and trailing silence. If you tag with clip duration the model will pad.

Reference audio is passed as codec codes, [T, 12] int64, directly in audio_codes_list; no audio tokenizer is needed at inference for conditioning.

Usage#

import torch
from transformers import AutoProcessor, AutoModel

REPO = "laion/moss-tts-local-transformer-4.55b-voice-acting-v2-sft"
proc = AutoProcessor.from_pretrained(REPO, trust_remote_code=True,
                                     codec_path="OpenMOSS-Team/MOSS-Audio-Tokenizer-v2")
model = AutoModel.from_pretrained(REPO, trust_remote_code=True,
                                  dtype=torch.bfloat16, attn_implementation="sdpa").cuda().eval()

msg = proc.build_user_message(
    text="[4.2 seconds duration] I really did not see that coming.",
    instruction="GENERAL: A young adult feminine voice; delivery is bright and quick; "
                "affect is positive, animated.\nSCRIPT:\n\"I really did not see that coming.\"",
    tokens=53, language="English")
batch = proc([[msg]], mode="generation")
out = model.generate(input_ids=batch["input_ids"].cuda(),
                     attention_mask=batch["attention_mask"].cuda(),
                     max_new_frames=400, do_sample=True,
                     audio_temperature=1.0, audio_top_p=0.95, audio_top_k=50)
wav = proc.decode(out)[0].audio_codes_list[0]   # 48 kHz

tokens is the target length in codec frames at 12.5 fps — 4.2 s ≈ 53 frames.

Training data#

Every micro-batch was exactly half of each dataset — the sampler alternates strictly between the two pools rather than mixing in expectation. Verified on 20 real batches: 0.5000 / 0.5000 on every one.

Because the two datasets are unequal in size and each batch is balanced, over three nominal epochs the voice-profile rows are seen ≈3.93 times and the real-speech rows ≈2.42 times.

The voice-profile source ships strictly one voice per file, so the corpus was globally shuffled before training; each shard ends up containing all 500 voices. Measured on real batches: 7.75 distinct voices per 8-sample batch, and no single voice ever exceeded 25 % of a batch.

Limitations#

• Two languages only. English and German. Anything else is out of distribution.
• Validation loss is not audio quality. It measures next-token likelihood on held-out codec tokens. No listening test, MOS study, or intelligibility benchmark has been run. Judge the sample grid with your own ears.
• The reference audio must be the same voice you want. Speaker similarity was not re-measured after fine-tuning.
• Three epochs on a balanced mix over-represents the synthetic half. If your use is real-recorded speech, that skew is worth knowing.
• Inherits whatever biases exist in the underlying corpora — speaker demographics, recording conditions and emotion labelling are not uniformly distributed.
• The 12 audio LM heads are weight-tied to the audio embeddings, as in the base model.

Intended use#

Research on expressive and controllable speech synthesis. Not validated for, and should not be used for, generating speech attributed to a real person without their consent, or for any application where a synthetic voice could be mistaken for a specific individual.

Licence and attribution#

Released under CC-BY-4.0 by Christoph Schuhmann and LAION.

The base model laion/moss-tts-local-transformer-4.55b-voice-acting-v2 is Apache-2.0, and that notice is preserved: this work is a derivative of it. The MOSS-TTS architecture and the MOSS-Audio-Tokenizer-v2 codec originate with the OpenMOSS team.

@misc{schuhmann2026mossva_sft,
  title  = {MOSS Voice-Acting 4.55B --- SFT},
  author = {Schuhmann, Christoph and {LAION}},
  year   = {2026},
  url    = {https://huggingface.co/laion/moss-tts-local-transformer-4.55b-voice-acting-v2-sft},
  note   = {Full fine-tune on 3.15M annotated utterances; CC-BY-4.0}
}