sound-effect-captioning-whisper

LAION Sound-Effect Captioning Whisper#

A Whisper-Small-sized audio-captioning model that writes rich, natural-language descriptions of general-purpose sound effects, ambiences, vocal bursts, and music snippets. Given ≤ 30 s of audio it produces a paragraph-length caption describing the content, timbre, and likely source of the sound.

This is the final stage of a multi-step training lineage that stacks emotional speech captioning, generative-audio pre-training, and sound-effect fine-tuning on top of OpenAI's Whisper-Small.

TL;DR#

• Architecture: WhisperForConditionalGeneration · Whisper-Small · 241.7 M params
• Input: mono audio, 16 kHz, up to 30 s
• Output: free-form English caption describing the sound
• Best validation loss (held-out val set, ~500 samples across 5 source datasets): 1.431 — down from 1.494 after stage 2, 1.689 after stage 1, and from ~2.0 before any sound-effect training
• Training compute: 8 × H100 · DDP via torchrun · ~9 h wall-clock end-to-end across three fine-tuning stages

Model genealogy#

This checkpoint is the end of a multi-step lineage; every stage starts from the previous one.

1. OpenAI Whisper-Small – ASR pre-training on 680 k hours of labelled speech.
2. laion/BUD-E-Whisper – LAION's emotion-aware speech captioning fine-tune. BUD-E-Whisper was trained on LAION's Got Talent plus ~5 k hours of public vlogs, with emotion scores generated by Gemini Flash 2.0 (40 emotion dimensions + 15 auxiliary dimensions such as age, arousal, valence, harshness and vocal-burst cues), templated into captions and then paraphrased for semantic richness. The resulting model can caption how a voice sounds, not just what it says.
3. laion/captioning-whisper-proof_of_concept (checkpoint-35000) – a continued pre-training run on a broad mix of non-speech audio with Gemini-generated captions:
   • Re-captioned AudioSet (mitermix/audioset-with-grounded-captions)
   • Re-captioned Freesound (laion/freesound-commercially-permissive-subset-with-captions)
   • TangoFlux-generated sound events – synthetic sound effects produced with TangoFlux and re-captioned
   • LAION AI-music snippets – machine-generated music clips with Gemini-style captions This stage teaches the decoder the "sound-effect paragraph" writing style and expands its vocabulary from speech events to environmental, mechanical, musical and abstract audio events.
4. This checkpoint – Sound-Effect Captioning Whisper – three additional fine-tuning stages on a fresh local mix of five public sound-effect / vocal-burst datasets (details below). Stage 1 = 1 epoch @ 5e-6 linear, Stage 2 = 3 epochs @ 1e-5 cosine (warmup 2 %), Stage 3 = 5 more epochs @ 5e-6 cosine (warmup 5 %). Each stage starts from the best checkpoint of the previous one, and the validation split is held fixed across all three so val-loss numbers are directly comparable.

Training data (this model)#

All datasets were downloaded, normalised to {mp3, caption} pairs, and pooled into a single flat directory. Per-dataset counts after filtering:

From each prefix, 100 random samples were held out as a stable validation split (~500 total), leaving ~915.6 k for training.

Training setup#

The stage-3 loss descended monotonically at every eval step — it still had not plateaued at the end of epoch 5, so a further continuation run could likely shave off more.

Stage-3 eval curve (lower is better):

For reference, the earlier stage-2 run's eval curve on the same held-out set:

The final checkpoint uploaded here (model.safetensors) is the stage-3 best weights from step 56 250, restored via load_best_model_at_end=True at the end of stage-3 training.

Usage#

import torch, torchaudio
from transformers import WhisperProcessor, WhisperForConditionalGeneration

REPO = "laion/sound-effect-captioning-whisper"
processor = WhisperProcessor.from_pretrained("openai/whisper-small")
model     = WhisperForConditionalGeneration.from_pretrained(REPO).eval().to("cuda")
model.generation_config.forced_decoder_ids = None

# Load any audio ≤ 30 s
wav, sr = torchaudio.load("your_sound.mp3")
if wav.shape[0] > 1:                 # → mono
    wav = wav.mean(dim=0, keepdim=True)
if sr != 16_000:                     # → 16 kHz
    wav = torchaudio.functional.resample(wav, sr, 16_000)
arr = wav.squeeze(0).numpy()[: 16_000 * 30]

feats = processor.feature_extractor(arr, sampling_rate=16_000,
                                    return_tensors="pt").input_features.to("cuda")
with torch.no_grad():
    ids = model.generate(feats, max_new_tokens=400, num_beams=1, do_sample=False)
print(processor.batch_decode(ids, skip_special_tokens=True)[0].strip())

Notes:

• Use the base openai/whisper-small processor (this repo only ships tokenizer.json + preprocessor_config.json, which are the Whisper-Small defaults – the processor class needs the bundled chat_template.json / special-tokens files that live on the base repo).
• forced_decoder_ids=None is required, otherwise Whisper's transcription language prompt clamps the decoder into ASR mode and suppresses captioning.
• Greedy decoding works well; beam=4 trades a bit of repetition for a small quality bump but is not necessary.

Example predictions#

All four clips below are drawn from the fixed held-out validation split (these exact files were never seen during training). They were captioned with this released checkpoint (stage-3 final-best, step 56 250, val loss 1.4314).

Example 1 — Vocal burst · groan#

Reference: This file features a low, rumbling vocalization, possibly a groan. There's a sense of strain in the sound, and it's drawn out, not abrupt. The recording appears to be of moderate quality, and no other sound events are discernible. The vocalization does not seem to be musical or speech-related.

Prediction: The audio features a low, guttural vocalization, possibly a groan or a moan. The sound is drawn out and has a somewhat strained quality, suggesting discomfort or effort. The recording is clear, with no noticeable background noise. The vocalization is the primary focus of the audio. The speaker sounds like an adult male.

Example 2 — Sound effect · sonar-style beeps#

Reference: Repetitive, high-pitched electronic beeps with a reverberant quality, resembling sonar or a digital alert. This sound could indicate a warning, a scanning process (like sonar), or a futuristic interface sound in a sci-fi context.

Prediction: The audio features a distinct, high-pitched, and sustained electronic tone that rapidly sweeps upwards in frequency, creating a sweeping effect. This is followed by a series of shorter, distinct electronic beeps or pulses, which then transition into a continuous, high-pitched, and somewhat distorted electronic tone. This sounds like a synthesized sound effect, possibly used in electronic music, video games, or as a sound design element to create a sense of tension, transition, or an otherworldly atmosphere.

Example 3 — Music · upbeat electronic#

Reference: The audio features a high-energy, upbeat piece of music. The melody is carried by a synthesizer, with a driving beat provided by electronic drums. A heavy bassline is prominent. There are sections with and without a strong beat. This recording is a song likely meant for dancing, especially as there is a synth melody.

Prediction: The audio features a fast-paced, energetic electronic dance track. A prominent, repetitive synth melody is layered over a driving beat. The overall sound is bright and upbeat. This is a clip from an electronic dance music track, likely intended for dancing or club environments. The hint confirms the presence of music.

Example 4 — Machinery · mechanical whine (TangoFlux-generated)#

Reference: The audio features a continuous, high-pitched whirring sound, which is then joined by a distinct, rhythmic clanking or grinding noise. This clanking/grinding sound is accompanied by a squealing sound that rises in pitch and then quickly descends. The sounds are consistent and repetitive. This sound is indicative of heavy machinery in operation, possibly a large engine or industrial equipment, given the combination of continuous whirring and repetitive clanking/grinding sounds, along with the squealing of moving parts.

Prediction: The audio features a continuous, high-pitched mechanical whine, characteristic of a jet engine. This sound is accompanied by a distinct, rhythmic clanking or grinding noise, suggesting the movement of mechanical parts. The overall sound is loud and sustained. This soundscape is consistent with the operation of a large industrial fan or a powerful engine, possibly from a jet engine or a large vehicle. The combination of whine, grinding, and clanking suggests the movement of heavy machinery or a large mechanical system.

Intended use#

• Automatic tagging / describing of sound-effect libraries and stock-audio collections.
• Audio-understanding component in agents that need to talk about what they hear.
• Research baseline for open-vocabulary audio captioning on top of Whisper.

Limitations#

• The model has no speech-to-text output: the base ASR behaviour of Whisper is intentionally suppressed, and transcribing spoken content is out of scope.
• Trained on ≤ 30 s clips; longer inputs are truncated. For long recordings, chunk them yourself and run the model per chunk.
• Captioning style is strongly biased toward the Gemini-style "the audio features …" phrasing inherited from the training data.
• Some mechanical / music / everyday-sound confusions remain; for difficult clips the model tends to fall back to generic genre templates ("electronic dance track", "jet engine", etc.).
• Training data is predominantly English and primarily captures the tastes of the AudioSet + Freesound + LAION-curated distributions.

Acknowledgements#

• OpenAI Whisper for the base model.
• LAION for curating the BUD-E-Whisper emotion dataset, the captioning-whisper proof-of-concept, the freesound / vocal-burst / in-the-wild / generated sound-event corpora, and the AI-music caption snippets used in the intermediate pre-training stage.
• TangoFlux for generating the synthetic sound-event data that fed the intermediate pre-training step.
• Gemini Flash 2.0 for the underlying caption generation and paraphrasing across multiple datasets.
• mitermix/audioset-with-grounded-captions for the grounded AudioSet captions.

License#

Released under Apache-2.0, following the upstream Whisper license. The audio clips in examples/ are samples from the five training datasets and retain their respective upstream licences (see the dataset pages linked above).