AudioLM generates coherent audio continuations by treating audio generation as language modeling. Core insight: Represent audio as discrete tokens (via codec), apply language model to predict next tokens. Generates semantically and acoustically consistent continuations. Architecture: Hierarchical token generation - first predict high-level semantic tokens (like w2v-BERT), then acoustic tokens (SoundStream). Two-stage: Semantic modeling captures content/meaning, acoustic modeling captures fine audio details. Training: Self-supervised on audio-only data, no text labels needed. Capabilities: Continue speech naturally (content + voice), continue music (melody + instruments), generate piano performances, maintain speaker identity. Key properties: Long-range coherence, natural prosody, voice consistency, music structure. Relationship to other models: Foundation for MusicLM (add text conditioning), similar principles in VALL-E, Bark. Sample quality: Remarkably natural continuations, difficult to distinguish from real audio. Limitations: Continuation only (not text-conditioned in base form), computationally intensive. Impact: Demonstrated audio can be modeled as language, opened path for transformer-based audio generation.
Explore 500+ Semiconductor & AI Topics
From EUV lithography to CUDA optimization — search the full knowledge base or chat with our AI assistant.