phonon-31m-en-ONNX

This is a Transformers.js-ready ONNX conversion of the original Hugging Face model gradium/phonon-31m-en. The original model card content follows below.

Usage with Transformers.js#

Native Phonon support (the text-to-speech pipeline below) comes with the Transformers.js release that includes huggingface/transformers.js#1811. On Transformers.js 4.2.0, use the snippet under "Streaming without the pipeline".

import { pipeline } from "@huggingface/transformers";

const tts = await pipeline("text-to-speech", "onnx-community/phonon-31m-en-ONNX", { dtype: "q4" });
// Any voice from "Available voices" below, as a URL, a Float32Array or a Tensor.
const speaker_embeddings = "https://huggingface.co/onnx-community/phonon-31m-en-ONNX/resolve/main/voices/LMDIYE0Oa377Qh6Z.bin";
const audio = await tts("Hello from Phonon!", { speaker_embeddings });
await audio.save("phonon.wav");

A voice URL is downloaded on every call. To speak several texts, load the voice once and pass the Float32Array instead. To receive audio chunk by chunk (each chunk is one or more whole sentences, up to about 50 tokens), call the model directly:

import { AutoProcessor, GradiumPhononTTSForConditionalGeneration, Tensor } from "@huggingface/transformers";

const id = "onnx-community/phonon-31m-en-ONNX";
const processor = await AutoProcessor.from_pretrained(id);
const model = await GradiumPhononTTSForConditionalGeneration.from_pretrained(id, { dtype: "q4" });
const data = new Float32Array(await (await fetch(`https://huggingface.co/${id}/resolve/main/voices/default.bin`)).arrayBuffer());
const speaker_embeddings = new Tensor("float32", data, [1, 512, data.length / 512]);
const chunks = await processor("Hello from Phonon. This is the second sentence.");
for await (const pcm of model.generate_speech({ chunks, speaker_embeddings })) {
  console.log(pcm.length); // mono 24 kHz samples, one text chunk at a time
}

How it works#

Phonon runs as two graphs. flow_lm is a causal transformer that is prompted with a voice and then the text, and emits one 32-dim latent per 80 ms frame, plus an end-of-speech logit. mimi_decoder turns latents into 1920 samples of 24 kHz audio each. Both graphs expose all their state (key/value caches, convolution context) as past* inputs and present* outputs; feed every output back under its past name on the next call.

The decoder takes any number of latents per call. The pipeline and generate_speech run the flow model to the end of a text chunk and then decode all of its latents in one call from an empty decoder state. The streaming snippet below decodes each latent as soon as it exists, so playback can start after the first 80 ms frame. Both give the same number of samples and the same audio up to rounding: above 120 dB SNR for fp32 and q4, and at least 45 dB for q4f16.

Both graphs use the same dtype. The decoder's file suffix names the variant it belongs to, not its precision: the decoder is not 4-bit quantized in any variant, because 4-bit weights audibly degrade its audio. mimi_decoder_q4.onnx is a copy of the fp32 mimi_decoder.onnx, and mimi_decoder_q4f16.onnx (with mimi_decoder_q4f16.onnx_data) is the fp16 decoder. mimi_decoder_fp16.onnx is the same fp16 graph under its older name, kept so existing code keeps working.

All three run several times faster than real time in the browser. Seconds of compute per second of audio on an Apple M3 Max in Chrome, decoding per chunk (as the pipeline does) and per frame (as the streaming snippet does):

Use q4f16 or q4 on WebGPU and fp32 on WASM (or q4 for a smaller download). WASM runs 4-bit matrix products slower than full precision for a model this size. On WebGPU, keep the state that is fed back into the graphs on the GPU, as the snippet below does (preferredOutputLocation); copying it to the CPU after every frame made WebGPU about three times slower. The figures were measured with this repository's runtime on the same graphs, not with the Transformers.js pipeline itself. fp32 output matches the reference Rust runtime (ptts) sample for sample (above 70 dB SNR at temperature 0). The quantized variants sample slightly different but equally intelligible speech: Whisper transcribed all of them identically to fp32 on a 12-sentence test set.

Streaming without the pipeline#

This snippet runs on Transformers.js 4.2.0 and yields audio frame by frame.

import { AutoConfig, AutoTokenizer, PreTrainedModel, Tensor } from "@huggingface/transformers";

const repo = "onnx-community/phonon-31m-en-ONNX";
const device = "webgpu"; // or "wasm"
const dtype = device === "webgpu" ? "q4f16" : "fp32"; // both graphs use the same one
const config = await AutoConfig.from_pretrained(repo);
const cfg = config.phonon;

// On WebGPU, keep every output that only goes back into a graph (caches, latent) on the GPU.
const onGpu = (names) => (device === "webgpu" ? { session_options: { preferredOutputLocation: Object.fromEntries(names.map((n) => [n, "gpu-buffer"])) } } : {});
const caches = ({ num_layers }) => Array.from({ length: num_layers }, (_, i) => [`present.${i}.key`, `present.${i}.value`]).flat();
const convStates = Object.keys(cfg.mimi_decoder.conv_states).map((name) => name.replace("past_", "present_"));
const tokenizer = await AutoTokenizer.from_pretrained(repo);
const flow = await PreTrainedModel.from_pretrained(repo, { config, model_file_name: "flow_lm", dtype, device, ...onGpu(["latent_out", ...caches(cfg.flow_lm)]) });
const mimi = await PreTrainedModel.from_pretrained(repo, { config, model_file_name: "mimi_decoder", dtype, device, ...onGpu([...caches(cfg.mimi_decoder), ...convStates]) });

// The tokenizer is Unigram with byte fallback, which Transformers.js does not apply
// yet: expand pieces missing from the vocabulary into their <0xXX> byte tokens.
const unigram = tokenizer._tokenizer.model;
if (!unigram.byte_fallback) {
  const encodePieces = unigram.encode.bind(unigram);
  unigram.encode = (tokens) => encodePieces(tokens).flatMap((piece) => {
    if (unigram.tokens_to_ids.has(piece)) return [piece];
    const bytes = Array.from(new TextEncoder().encode(piece), (b) => `<0x${b.toString(16).toUpperCase().padStart(2, "0")}>`);
    return bytes.every((b) => unigram.tokens_to_ids.has(b)) ? bytes : [piece];
  });
}

const f32 = (dims, data) => new Tensor("float32", data ?? new Float32Array(dims.reduce((a, b) => a * b, 1)), dims);
const i64 = (values, dims) => new Tensor("int64", BigInt64Array.from(values, BigInt), dims);
const emptyCache = ({ num_layers, num_heads, head_dim }) => Object.fromEntries(
  Array.from({ length: num_layers }, (_, i) => ["key", "value"].map((k) => [`past_key_values.${i}.${k}`, f32([1, num_heads, 0, head_dim])])).flat());
const nextState = (outputs) => Object.fromEntries(Object.entries(outputs)
  .filter(([name]) => name.startsWith("present"))
  .map(([name, value]) => [name.replace("present.", "past_key_values.").replace("present_", "past_"), value]));
// GPU tensors are not garbage collected: free each state once the next one replaces it.
const release = (...states) => states.forEach((state) => Object.values(state).forEach((t) => t.location === "gpu-buffer" && t.dispose()));

// A voice is raw little-endian float32 data in [1, 512, frames] order. voices/ has all of
// them (see "Available voices" below); voices/default.bin is the original model's default voice.
const voiceData = new Float32Array(await (await fetch(`https://huggingface.co/${repo}/resolve/main/voices/default.bin`)).arrayBuffer());
const voice = f32([1, cfg.voice_dim, voiceData.length / cfg.voice_dim], voiceData);

const runFlow = (past, { text = [], voiceLatents = f32([1, cfg.voice_dim, 0]), latent = f32([1, 0, cfg.latent_dim]), noise = f32([1, cfg.latent_dim]) }) =>
  flow({ text_ids: i64(text, [1, text.length]), voice_latents: voiceLatents, latent, noise, num_speakers: i64([1], [1]), duration_delta: f32([1]), ...past });

// Prompt the voice once; every sentence starts from this state.
const voiceState = nextState(await runFlow(emptyCache(cfg.flow_lm), { voiceLatents: voice }));

// Yields one Float32Array of 24 kHz audio per 80 ms frame. Keep each call to about one
// sentence (at most 50 tokens) and call it per sentence for longer text.
async function* speak(sentence, temperature = cfg.temperature) {
  let text = sentence.trim().replace(/\s+/g, " ");
  text = text[0].toUpperCase() + text.slice(1) + (/[\p{L}\p{N}]$/u.test(text) ? "." : "");
  const ids = tokenizer.encode(text, { add_special_tokens: false });
  let state = nextState(await runFlow(voiceState, { text: ids }));
  let codec = { past_offset: i64([0], [1]), ...emptyCache(cfg.mimi_decoder) };
  for (const [name, dims] of Object.entries(cfg.mimi_decoder.conv_states)) codec[name] = f32(dims);
  let latent = f32([1, 1, cfg.latent_dim], new Float32Array(cfg.latent_dim).fill(NaN)); // NaN = start
  const randn = () => Math.sqrt(-2 * Math.log(1 - Math.random())) * Math.cos(2 * Math.PI * Math.random());
  let countdown = null;
  try {
    for (let frame = 0; frame < Math.ceil((ids.length / 3 + 2) * cfg.frame_rate); frame++) {
      const noise = f32([1, cfg.latent_dim], Float32Array.from({ length: cfg.latent_dim }, () => randn() * Math.sqrt(temperature)));
      const step = await runFlow(state, { latent, noise });
      release(state, { latent });
      [state, latent] = [nextState(step), step.latent_out];
      const decoded = await mimi({ latent, ...codec });
      release(codec);
      codec = nextState(decoded);
      yield decoded.audio.data;
      // Stop a few frames after the end-of-speech logit crosses the threshold.
      if (countdown === null && step.eos_logit.data[0] > cfg.eos_threshold) countdown = text.split(" ").length <= 4 ? 3 : 1;
      if (countdown !== null && countdown-- === 0) break;
    }
  } finally {
    release(state, codec, { latent });
  }
}

const chunks = [];
for await (const pcm of speak("Hello from Phonon, running in Transformers.js.")) chunks.push(pcm); // or play each chunk as it arrives
const audio = new Float32Array(chunks.reduce((n, c) => n + c.length, 0));
chunks.reduce((offset, c) => (audio.set(c, offset), offset + c.length), 0);

Notes:

• temperature (default 0.3) scales the sampling noise; 0 gives deterministic output.
• Each call to speak resets the codec, so successive sentences join cleanly. The voice prompt is computed once and reused, not recomputed per sentence.
• This model has no audio encoder, so it cannot clone a voice from a recording. It uses precomputed voice embeddings: the 102 voices listed under "Available voices" below.
• The tokenizer prepares text the way the Python runtime does: it collapses whitespace, capitalizes the first letter, adds a final period, and folds typographic quotes, dashes and accented letters outside the model's alphabet. The Rust and Python runtimes also spell out currency, e-mail addresses and other symbols before tokenizing; this repo does not. Digits, times, percentages and phone numbers are read correctly without it, but €5.50 comes out in dollars and the @ of an address is skipped. Spell such symbols out ("5 euros 50", "at") for best results.
• num_speakers (index into ["0", "1", "2"], default 1, the value the original runtime uses) and duration_delta (default 0.0) are the checkpoint's conditioning inputs.
• Output is raw PCM. The Python processor's trailing-silence post-processing is not applied.

Export details#

The graphs were exported from gradium/phonon-31m-en at revision 52433c46975bfa13144c56204c9f4ee1c5b148aa, which config.json also records under phonon.source. All graphs have producer pytorch 2.13.0, IR version 8 and ONNX opset 18. The 4-bit flow graphs also import the com.microsoft opset (version 1) for MatMulNBits.

Validation scope#

• fp32 output was compared numerically with the reference Rust runtime (above 70 dB SNR at temperature 0). The quantized variants were checked by transcription, not sample for sample.
• The speed figures above are from Chrome on an Apple M3 Max, using WASM and WebGPU. Other browsers, devices and WebNN have not been measured.
• WebGPU fp16 needs the shader-f16 feature. Where it is missing, use fp32 or q4.

Available voices#

Each file in voices/ is one voice: raw little-endian float32 data in [1, 512, frames] order (here 125 frames, 256 000 bytes). Pass its URL or a Float32Array of it as the voice. voices/default.bin is the voice the original runtime uses by default.

All 102 voices of gradium/phonon-voices (Apache 2.0, revision 0ee17205), converted to the raw format above. The voice's language is the language it was designed in; this model always speaks English. Audio samples of every voice are on gradium/phonon-voices. voices/voices.jsonl lists the same voices with their metadata.

English (US)#

English (UK)#

English characters#

French#

German#

Spanish (Spain)#

Spanish (Mexico)#

Portuguese (Brazil)#

Original model card#

phonon-31m English#

High-quality Text-to-Speech, on your device.

phonon-31m is a text-to-speech model from the Phonon family. It turns text into natural 24 kHz speech in English. It streams audio while it is still generating, and it is small enough to run in a browser tab, on a phone, or on a laptop CPU.

Phonon builds on Pocket TTS and the Mimi codec, developed by Kyutai.

Try it in your browser · All Phonon models · Runtime on GitHub · Python · Browser · Discord

Model details#

• Model type: text-to-speech. A flow-matching transformer predicts audio latents from the text one 80 ms frame at a time, and the Mimi codec's decoder turns them into audio.
• Parameters: 31M (excluding the audio tokenizer)
• Weights file: model.q8.gguf, 64.5 MB (8-bit quantized)
• Languages: English
• Output: mono, 24 kHz
• License: Apache 2.0

The Phonon family#

All of them run on the same runtime and use the same file layout, so switching models means changing the repo ID.

Quickstart#

Every runtime names this model as hf://gradium/phonon-31m-en, and takes lang, which selects how text is normalized (numbers, symbols, dates). This model speaks English, so use en.

Command line#

uvx ptts --lang en --quant q8 \
  --voice en_us_Easton "Hello from Phonon." -o speech.wav

uvx runs the Python package's command without installing it. The standalone ptts binary from GitHub Releases or Homebrew (brew install gradium-ai/tap/ptts) takes the same flags.

Python#

pip install ptts

import ptts

tts = ptts.TTS("hf://gradium/phonon-31m-en", lang="en", quant="q8")
tts.save("speech.wav", "Hello from Phonon.", voice="en_us_Easton")

# Or stream: chunks arrive while the rest is still being generated.
with tts.stream("Speech can start playing right away.", voice="en_us_Easton") as audio:
    for pcm in audio:
        ...  # float32 chunks at audio.sample_rate

Other platforms#

Voices#

Phonon comes with a catalog of ~100 voices available in the gradium/phonon-voices repo. Instant voice cloning and voice design are available through the Gradium studio.

Benchmarks#

Phonon was compared with other open text-to-speech models on three measures, per language:

• Elo: human preference from blind pairwise listening tests (higher is better).
• Hard set LLM WER: word error rate on hard sentences full of structured data (numbers, dates, codes, units), scored with an LLM-assisted ASR comparison (lower is better).
• Seed-TTS LLM WER: the same metric on the Seed-TTS evaluation sentences (lower is better).

In every language, a Phonon model is first on all three measures.

English (US) (Elo from 468 raters, 8,290 votes)

French (Elo from 172 raters, 2,270 votes)

German (Elo from 168 raters, 2,220 votes)

Spanish (Spain) (Elo from 201 raters, 2,290 votes)

Portuguese (Brazil) (Elo from 171 raters, 2,000 votes)

Intended use#

Phonon is made for applications that speak on the user's device: offline and private assistants, screen readers and other accessibility tools, games, and interactive apps that cannot wait for a cloud round trip. It also runs as a local server with an OpenAI-compatible API and as a voice for Home Assistant.

Limitations#

• The model speaks only the languages above. Text in other languages is read with the wrong pronunciation.
• Text is normalized before synthesis (numbers, currency, dates, emails and so on), and that step needs the right language. Choose lang to match the text.
• Very long inputs are split into sentences and spoken one after the other, so prosody does not carry across the whole text.

Responsible use#

Do not use Phonon to impersonate a real person, to deceive listeners about who is speaking, or to clone a voice without the speaker's consent. Tell listeners when speech is synthetic where that matters to them.

License#

The weights, tokenizer and voices in this repository are released under Apache 2.0, which allows commercial use. The runtime is a separate project, licensed MIT OR Apache-2.0.