jukebox

Jukebox is OpenAI's music generation model that produces music with singing in raw audio form, generating complete songs with vocals, instruments, and lyrics at CD quality (44.1 kHz). Released in 2020, Jukebox was groundbreaking as the first model to generate convincing singing voices alongside instrumental accompaniment directly in the audio waveform, rather than relying on MIDI or symbolic representations. The architecture uses a hierarchical VQ-VAE (Vector Quantized Variational Autoencoder) with three levels of compression: the top level compresses audio by 128× (operating at ~344 tokens per second), the middle level by 32×, and the bottom level by 8×. Generation proceeds top-down: the top-level prior (a transformer) generates the most compressed representation capturing high-level musical structure (melody, harmony, song form), then upsampling priors progressively add detail at each level. This hierarchical approach addresses the fundamental challenge of music generation — a 4-minute song at 44.1 kHz contains over 10 million samples, far too many for direct autoregressive generation. Jukebox is conditioned on artist, genre, and lyrics metadata, enabling control over musical style. The model can generate in the style of specific artists (having learned from their discography), follow provided lyrics with approximate word-level alignment, and produce novel compositions that capture genre-specific characteristics. Training data comprised 1.2 million songs (600K in English). Limitations include: extremely slow generation (taking hours of GPU time per minute of audio due to the autoregressive sampling at each hierarchy level), limited coherence for long-form structure (songs may drift stylistically over several minutes), imperfect lyric alignment (vocals may be intelligible but don't precisely follow provided text), and the inability to generate to a specific tempo or key. Despite these limitations, Jukebox demonstrated that neural networks could learn the complex interplay between vocals and instruments directly from raw audio.