vocalburst-captioning-whisper

vocalburst-captioning-whisper#

A fine-tune of laion/sound-effect-captioning-whisper (Whisper-small encoder–decoder) specialised for captioning vocal bursts — laughs, sighs, screams, coughs, crying, panting, throat-clearing, etc. Given a short audio clip it generates a free-text description of the non-speech vocalisation.

🔗 Ensemble: use together with the detector laion/vocalburst-locator — that model finds where the vocal bursts are (timestamps); this model captions each detected segment. See the threshold study below.

Training#

• Data: laion/improved_synthetic_vocal_burts (15,480 train / 200 val). Target = the Flash 2.5 Annotation.caption field (the long descriptive caption).
• Recipe (best of a 5-epoch × 2-LR sweep): peak LR 1e-5, 3 epochs, cosine decay with 10 % warmup, bf16, batch 8, AdamW. Labels masked with -100; forced_decoder_ids = None.
• Selection: every ~2000 seen samples the model captioned the 200 val clips and was scored by cosine( caption, audio ) using laion/voiceclap-small-v2; the best checkpoint by this clap_sim was kept.

Results (val clap_sim — higher = caption matches audio better)#

Full sweep (all beat the 0.190 baseline; lr 1e-5 dominates, 3 epochs is the peak):

Example predictions on 100 val clips (audio + greedy & temperature-0.5 captions) are in vb_finetune_predictions.html. Mean clap_sim over those 100: greedy 0.248, temperature-0.5 0.236.

Vocal-burst captioning ensemble & detection-threshold study#

This detector is designed to be used as an ensemble with the fine-tuned captioner laion/vocalburst-locator: the locator finds where vocal bursts occur (start/end timestamps); each detected segment is then cut and described by the captioner (this locator). Together they turn raw audio into timestamped, captioned vocal-burst events that feed the LAION Universal Audio Annotation Pipeline.

How the study was run#

We swept the detector's confidence threshold from 0.85 to 0.92 (1% steps) on 150 audio samples (clean-speech false-positive checks + clips with inserted bursts + isolated bursts), with merge_gap = 0.3 s, min_dur = 0.5 s. For every (sample × threshold) the detector's segments were captioned by laion/vocalburst-captioning-whisper and the audio + (start, end, caption) list was sent to Gemini 3.1 Pro, which rated three axes 0–5 (5 = perfect): caption quality, timestamp accuracy, and completeness (do the detections cover ALL real vocal bursts, penalizing both misses and false positives). That is 1,200 independent LLM judgments; overall = mean of the three axes.

Results — average Gemini-3.1-Pro scores per threshold (ranked)#

Findings: scores are tightly clustered across 0.85–0.92 (the detections change little in that band); threshold ≈ 0.88 is the sweet spot (best overall). Timestamp accuracy is consistently strong (~4.0), caption quality is moderate (~3.2), and completeness is the weakest axis (~3.0–3.15) — it degrades at the highest thresholds (0.91–0.92) as real bursts start being missed. Recommended operating point for the ensemble: threshold 0.85–0.89 (default 0.88).

📊 Full interactive report (stats table + audio players + predictions + per-clip Gemini scores for the top-3 thresholds): vocalburst_threshold_report.html.

Usage#

import torch, librosa
from transformers import WhisperProcessor, WhisperForConditionalGeneration

proc = WhisperProcessor.from_pretrained("openai/whisper-small")
model = WhisperForConditionalGeneration.from_pretrained("laion/vocalburst-captioning-whisper").eval().cuda()
model.generation_config.forced_decoder_ids = None

wav = librosa.load("burst.wav", sr=16000, mono=True)[0]
feat = proc.feature_extractor(wav, sampling_rate=16000, return_tensors="pt").input_features.cuda()
ids = model.generate(feat, max_new_tokens=256)   # greedy; add do_sample=True, temperature=0.5 to sample
print(proc.batch_decode(ids, skip_special_tokens=True)[0])