music-whisper

Music Whisper (Whisper Small Fine-tune)#

This is a fine-tuned version of OpenAI Whisper Small specifically trained for Music Captioning. Its mean-pooled encoder state also serves as the 768-d audio fingerprint of LAION-Tunes.

Companion release of the NeurIPS 2026 Evaluations & Datasets Track paper LAION-Tunes: An Open 1.4M-Track Dataset and Perceptual Benchmark for AI-Generated Music (Kaczmarczyk, Ahmed, Friedrich, Erickson, Sharoni, Herremans, Schuhmann). Code: LAION-AI/tunes. Demo: Tunes-Search.

Model Details#

• Base Model: OpenAI Whisper Small
• Task: Music Captioning (generating textual descriptions of audio tracks)

Training#

Knowledge distillation from Gemini Flash 2.0. The teacher produced six-field structured captions (Overall Sound, Instrumentation, Tempo, Genre, Mood, Vocals) for 2.91M Suno clips of 3 to 30 s from laion/captioned-ai-music-snippets, cut from nyuuzyou/suno.

Reference Samples#

This repository includes an HTML file with embedded reference samples that you can download and view to hear the audio alongside the generated captions.

Example Predictions#

Below are examples of captions generated by this model (Ground Truth/Prediction comparison):

Example 1#

"The listener hears a track that immediately establishes itself with a distorted guitar riff, setting a raw and edgy tone. A driving drum beat soon joins, characterized by a prominent snare drum and a heavy kick drum, solidifying the rhythmic foundation. The tempo is approximately 140 beats per minute, contributing to the track's energetic feel. A male vocalist then begins to sing, employing a slightly strained vocal delivery. The vocal profile is that of an adult male, with a timbre and quality described as slightly raspy and aggressive. There are no non-lyrical vocal sounds present. The genre of the music is identified as either Alternative Rock or Hard Rock. The overall mood of the piece is energetic, rebellious, and carries a slight sense of angst. The combination of distorted guitars, the driving drum beat, and the strained vocal delivery are all characteristic elements of these genres. The production quality is relatively clean, yet it retains a raw edge, which enhances the overall intensity of the track. This track would be well-suited for a high-energy action scene, a rock concert, or a rebellious youth anthem."

Example 2#

"The listener hears a piece of music primarily characterized by a slow tempo and a melancholic melody. The primary instrumentation consists of a piano, which carries the main melodic line, and a string section that provides a subtle harmonic foundation. The piano's timbre is described as warm and slightly muffled, contributing to the overall sonic texture. The string section is played with a gentle touch, adding a layer of emotional depth to the piece. The tempo is slow, contributing to the overall mood. The genre of the music is identified as either Classical or Film Score. The mood is predominantly melancholic, reflective, and peaceful. The combination of the piano's melody, the string section's presence, and the string section's delicate sound creates a sense of intimacy and vulnerability. This musical composition would be well-suited for a dramatic scene in a film, a quiet moment of reflection, or as background music for a relaxing activity."

License#

Apache 2.0.

Citation#

@inproceedings{kaczmarczyk2026laiontunes,
  title     = {{LAION-Tunes}: An Open 1.4{M}-Track Dataset and Perceptual Benchmark for {AI}-Generated Music},
  author    = {Kaczmarczyk, Robert and Ahmed, Tawsif and Friedrich, Felix and Erickson, Aidan C. and Sharoni, Orian and Herremans, Dorien and Schuhmann, Christoph},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS), Evaluations and Datasets Track},
  year      = {2026}
}