musiclm

MusicLM is Google's text-to-music generation model that creates high-fidelity music from natural language descriptions, generating 24 kHz audio that captures the genre, mood, instrumentation, tempo, and stylistic qualities specified in text prompts. Introduced by Agostinelli et al. (2023), MusicLM frames music generation as a hierarchical sequence-to-sequence task, using a cascade of neural audio codec tokens at different granularities. The architecture combines three pre-trained models: MuLan (a music-text joint embedding model that aligns audio and text in a shared representation space, providing the semantic conditioning signal), SoundStream (Google's neural audio codec that compresses audio into discrete tokens at multiple levels of detail — semantic tokens capturing high-level musical structure and acoustic tokens encoding fine-grained audio details), and w2v-BERT (a self-supervised audio model providing intermediate semantic representations). Generation proceeds hierarchically: semantic tokens are generated first (capturing melody, rhythm, and overall structure), then acoustic tokens are generated conditioned on the semantic tokens (adding timbral detail, audio quality, and fine-grained sonic textures). This hierarchical decomposition allows the model to first establish musical coherence (getting the song structure right) before filling in audio details. MusicLM capabilities include: text-to-music generation (creating music matching textual descriptions like "a calming violin melody backed by a distorted guitar riff"), long-form generation (producing minutes-long coherent compositions), melody conditioning (generating music that follows a hummed or whistled melody while matching a text description's style), and sequential prompting (generating music that transitions between different text descriptions over time). MusicLM was trained on a 280K-hour music dataset and demonstrated that increased scale improves audio quality and text adherence. Google subsequently released MusicFX as a consumer product based on this research.