Sources are tried in order. A file counts only when its size and sha256 match the manifest. Sources are checked when you ask for them, not on a timer.
Sources
Badge
Host
Last check
Result
From source
huggingface.co
2026-11-15 05:25 UTC
passed
Model card
From the publisher at commit 657f5d5. Shown as plain text with links; images open at the publisher.
MOSS Audio — Whole-audio voice captioning trained on Gemini captions#
Describe audible voice characteristics, delivery, emotion, nonverbal vocal events and the recording environment in English. This repository contains a trained, separate LoRA adapter for the pinned MOSS-Audio-8B-Instruct model. It is a text-generating caption model, distinct from the encoder-only Humaneness Ears score and localization models.
Base: OpenMOSS-Team/MOSS-Audio-8B-Instruct, revision d4dc5a6d8cd79b43dcd82884c75e303b1ecd016d. Reviewed official source, revision 66326e6e0db34f036c86a76ba005efa4830c69dd. MOSS combines a pretrained audio encoder, an audio-to-language adapter, DeepStack audio injections and a Qwen3 language transformer. Approximate audio token rate is 12.5 Hz. The exact instantiated base has 9,052,463,488 parameters; the repository's 8B name refers to its language-model size class.
Only rank-32 LoRA weights inside the language transformer's attention and MLP projections are trained. Base language weights, the audio encoder, the audio adapter and DeepStack mergers are frozen. The actual total parameter count including LoRA is 9,139,757,440; trainable adapter parameters are 87,293,952. This adaptation does not train a new audio encoder or a burst localization head.
The Gemini annotation collection joins a balanced S1 selection, LAION's Got Talent, balanced 40-emotion snippets and a voice annotation POC. Splits are disjoint by exact MP3 SHA256. Burst examples share full recordings within a split and use one training example per event. The labels are machine-generated Gemini descriptions, not independently adjudicated human ground truth. Some S1 clips may have been seen in earlier Whisper score-model pretraining; they are excluded from this caption adaptation's training split.
Whole-voice supervision is the voice_caption. Burst supervision is an event's detailed caption. Audio, the task prompt and, for burst captioning, numeric interval endpoints are the only inputs. No transcript, class label, reference caption or Gemini score vector is included in the input. Every event has both start and end seconds.
mean next-token cross entropy, assistant caption tokens only
Best checkpoint
checkpoint-1826
Best validation loss
1.290871
Validation loss is teacher-caption token prediction loss. It measures fit to Gemini text and is not a human caption-quality score or an event detection metric. The best epoch checkpoint is selected by this loss. Full optimizer/scheduler/RNG checkpoints are saved each epoch and every 200 updates in the training run; this lightweight public release contains the best adapter and training history, not the large optimizer states.
Gradient checkpointing is disabled because the upstream DeepStack path uses temporary forward hooks. Frozen audio modules remain in evaluation mode. model_accepts_loss_kwargs=False ensures correct accumulation scaling: the official MOSS loss does not consume Trainer's num_items_in_batch. Transformers 4.57.1 is pinned, matching the upstream recommendation; a narrow encoder output compatibility hook is included.
The download helper pins the base revision and verifies all four weight shard hashes. A BF16 CUDA GPU is recommended; CPU inference uses float32 and is much slower and more memory-intensive. Complete clips of up to 30 seconds match the adaptation distribution. Generation is greedy and deterministic with a configurable token limit. The burst adapter requires externally supplied timestamps; pair it with a locator to describe automatically detected intervals.
export_sft.py exports the published annotation TARs to this exact SFT contract. Where source audio is not redistributed, supply an --audio-map mapping original MP3 hashes to local TAR members. The exporter refuses a partial dataset by default and never silently invents missing audio.
The full executed trainer, task recipe, data reader, masking and official source files are included. The upstream manifest verifies vendored source hashes. See training_run.json, training_results.json, training/trainer_state.json, data_audit.json and base_weights_manifest.json for evidence and exact revisions.
These captions can describe delivery and audible affect, but they are probabilistic interpretations and may omit or hallucinate details. They do not establish a speaker's internal emotional state, identity or diagnosis. They do not replace the separate numeric emotion, VoiceNet, DNSMOS, AudioBox or Orange embedding heads: those scores are not training targets for these two text adapters. The listening demo uses 50 newly generated Gemini TTS acting performances, distinct from the Gemini-caption adaptation corpus. Humaneness Ears Medium proposes the burst intervals; the caption adapter receives the full recording and both interval endpoints. These intervals are unvalidated predictions. No Gemini reference captions are displayed. This is a qualitative listening grid rather than an independent human-rated benchmark. The untuned baseline uses the same audio and prompt with LoRA disabled. Recordings longer than 30 seconds are supplied in full but extend beyond the adaptation clip-length range.
Original adapter weights and documentation: CC BY 4.0, LAION. Pinned MOSS base weights and vendored OpenMOSS source retain Apache 2.0, with attribution and license preserved in upstream/LICENSE. Audio sources retain their source terms. Dataset provenance and annotation methods are documented in the linked dataset card.
