Audio8-TTS-Realistic-Air

Image: Audio8-TTS-Realistic-Air — More realistic speech. More languages. Still 0.6B.

GitHub License Demo arXiv

Playground

Languages · Quick start · Serving · Evaluation

Audio8-TTS-Realistic-Air builds on Audio8-TTS-Preview-0.6b, delivering substantially improved speech realism while retaining its compact 0.6B architecture and zero-shot voice cloning capabilities.

This release supports 23 languages, adding 12 languages compared with Preview. It uses the same Transformers inference interface as Preview. This repository includes the complete model checkpoint, its 44.1 kHz neural audio codec, tokenizer, processor, and custom Transformers code.

Try speech generation and voice cloning in the interactive Playground. Hosted on Hugging Face ZeroGPU; availability depends on the queue and your daily GPU allowance.

Highlights#

• Improved realism: more natural prosody and expressive delivery, building on Preview.
• Broader multilingual coverage: 23 languages, including 12 additions to Preview's 11 supported languages.
• Compact model: 601 million main-model parameters, with the audio codec bundled separately.
• Flexible generation: zero-shot voice cloning from a reference recording and transcript, or speech generation without a reference.
• GPU serving: an SGLang Omni adapter with paged attention, dynamic batching, and an OpenAI-compatible speech API.

Supported Languages#

Note: Language quality varies; see the per-language evaluation below. Benchmark coverage includes some additional languages and is distinct from the supported-language list.

Model Details#

Audio8 TTS uses a DualAR architecture inspired by Fish Audio S2 Pro. The slow AR transformer predicts one semantic token for each audio frame. The fast AR transformer predicts the frame's codec codebooks, conditioned on the slow hidden state and preceding codebooks.

The bundled codec handles both reference-audio encoding and waveform decoding, so no additional codec checkpoint is required. The 0.6B size refers to the main DualAR model; the codec is excluded from that parameter count.

Installation#

Python 3.10 or newer and a CUDA-capable GPU are recommended.

pip install "torch>=2.5.0" "torchaudio>=2.5.0" \
  "transformers>=4.57.0,<5" "soundfile>=0.12" "safetensors>=0.4"

The Transformers release package was smoke-tested with PyTorch 2.8.0 and Transformers 4.57.3 on an NVIDIA H20 using BF16. Checks covered model loading, English and Chinese generation with and without reference audio, and generation in Arabic, Cantonese, Portuguese, and Russian. These are runtime checks, separate from the benchmark results below.

Usage#

The model uses custom Transformers code. Review the files in this repository, then load it with trust_remote_code=True.

Zero-shot voice cloning#

Use a clean reference recording. Its transcript must match the spoken content exactly.

import soundfile as sf
import torch
from transformers import AutoModel, AutoProcessor

model_id = "Edge0/Audio8-TTS-Realistic-Air"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if device == "cuda" else torch.float32

processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModel.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=dtype,
).eval().to(device)

inputs = processor(
    text=["Welcome to Audio8 TTS. It is wonderful to hear from you."],
    reference_audio=["reference.wav"],
    reference_text=["The exact transcript of the reference recording."],
    return_tensors="pt",
)
inputs = {name: value.to(device) for name, value in inputs.items()}

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=1024,
        temperature=0.8,
        top_p=0.95,
        top_k=50,
        do_sample=True,
        return_dict_in_generate=True,
    )
    waveforms, waveform_lengths = model.decode_audio(output.codes)

audio = waveforms[0, : int(waveform_lengths[0])].float().cpu().numpy()
sf.write("output.wav", audio, model.config.codec_sample_rate)

Generation without a reference#

Replace the processor call above with:

inputs = processor(
    text=["This utterance does not use a reference voice."],
    return_tensors="pt",
)

Then move the inputs to the model device and run the same generation and decoding steps. For command-line inference, batching, and supervised fine-tuning, see the Audio8 TTS repository, using Edge0/Audio8-TTS-Realistic-Air as the model path.

High-throughput serving: SGLang Omni#

The SGLang Omni adapter provides GPU serving for Audio8 TTS. It is installed as an independent model plugin and does not overwrite SGLang Omni core files.

Use the 0.6B configuration, sglang_omni/configs/audio8_tts_0_6b.yaml, and set the model path to Edge0/Audio8-TTS-Realistic-Air or its downloaded snapshot. The adapter supports generation without a reference and zero-shot voice cloning. See the deployment guide for compatible dependency versions, installation, server configuration, and API examples.

Validation note: Audio8-TTS-Realistic-Air was tested on an NVIDIA H20 in BF16 for generation with and without a reference through the non-streaming speech API, with Torch compilation and CUDA graphs disabled. These checks validate runtime compatibility; throughput and streaming were not benchmarked.

Evaluation#

Error rates are percentages (lower is better). Speaker similarity (SIM) is on a 0–1 scale (higher is better). These metrics measure intelligibility and speaker similarity; they do not directly measure perceived speech realism.

Benchmark summary#

Audio8-TTS-Preview and Audio8-TTS-Realistic-Air, both 0.6B:

Note: Values are unweighted language macro averages. Preview results are calculated from its published model card, with SIM converted from percentages to a 0–1 scale. The two releases were evaluated separately. A dash means the corresponding result was not reported on Preview's card.

Seed-TTS averages English WER and Chinese CER, excluding hard Chinese. The shared CV3 subset is Chinese, English, Japanese, Korean, German, French, Spanish, and Italian; the 9-language set also includes Russian. CV3 averages exclude hard subsets and the 13 additional languages. MiniMax averages all 24 evaluated languages, including languages outside the supported list.

Reference comparison#

The following error rates come from the sources used in our technical report. Different evaluation pipelines and text normalization make this a reference comparison rather than a strictly matched ranking. A dash means that a result is unavailable in the comparison, not that the model lacks support.

Sources: Qwen Seed-TTS results are from the Qwen3-TTS report, linked from the official Qwen3-TTS repository; Qwen's CV3/MiniMax results, all Higgs Audio v3 results, and MOSS-TTS-v1.5's CV3/MiniMax results are from Boson AI's evaluation. Fish S2 Pro's MiniMax result is from the Fish Audio S2 report, linked from the official Fish Speech repository; its Seed-TTS/CV3 results and Higgs Audio v2's results are our reevaluations. MOSS-TTS-v1.5's Seed-TTS result and the CosyVoice3/VoxCPM2 results are compiled in the VoxCPM repository. Cross-project SIM values are omitted because speaker encoders and evaluation conventions differ.

Parameter conventions: Audio8's 0.6B excludes its separate codec; Qwen and CosyVoice use release labels; Higgs Audio v3 uses its nominal 4B release label. The remaining sizes are rounded checkpoint counts used in our report (VoxCPM2 excludes its separate AudioVAE). These counts use different conventions across systems.

Seed-TTS detail#

Note: Preview values are from its published Seed-TTS table, with SIM converted to a 0–1 scale.

Multilingual detail#

CV3: 22 languages and hard subsets

Hard Chinese CER is 10.665% and hard English WER is 4.736%. These hard subsets are excluded from the 9-language summary. Each of the 22 regular language rows contains 500 evaluation items. Chinese, Japanese, and Korean use CER; the other regular language rows use WER.

MiniMax: all 24 evaluated languages

Each language has 100 evaluation items. Rows marked † are evaluated for completeness and are outside this release's supported-language list. The metric column preserves the evaluation suite's language-specific WER/CER choice. Cantonese is included in the 24-language average.

Compared with Preview#

Realistic Air expands Preview's 11-language coverage to 23 languages, adding Arabic, Finnish, Hindi, Indonesian, Malay, Norwegian, Portuguese, Russian, Swedish, Thai, Turkish, and Vietnamese.

Limitations and Responsible Use#

• Quality varies by language and evaluation set. The detailed results include difficult cases, including Cantonese and some newly supported languages; the support list is not a claim of equal quality across languages.
• Noisy reference audio or mismatched transcripts can reduce stability and speaker similarity. The reference audio, transcript, and target text share a finite context budget; split long passages into manageable utterances.
• Speech may contain pronunciation errors, repetitions, omissions, or early stopping. Listen to generated audio before using it in a finished product.
• Obtain permission to clone a voice and disclose synthetic audio where appropriate.

License and Acknowledgements#

The code and model weights are released under the Apache License 2.0. See NOTICE for attribution details.

We thank the Fish Audio team for publishing the DualAR architecture used in Fish Audio S2 Pro.