audiolm

AudioLM generates coherent audio continuations by treating audio generation as language modeling. **Core insight**: Represent audio as discrete tokens (via codec), apply language model to predict next tokens. Generates semantically and acoustically consistent continuations. **Architecture**: Hierarchical token generation - first predict high-level semantic tokens (like w2v-BERT), then acoustic tokens (SoundStream). **Two-stage**: Semantic modeling captures content/meaning, acoustic modeling captures fine audio details. **Training**: Self-supervised on audio-only data, no text labels needed. **Capabilities**: Continue speech naturally (content + voice), continue music (melody + instruments), generate piano performances, maintain speaker identity. **Key properties**: Long-range coherence, natural prosody, voice consistency, music structure. **Relationship to other models**: Foundation for MusicLM (add text conditioning), similar principles in VALL-E, Bark. **Sample quality**: Remarkably natural continuations, difficult to distinguish from real audio. **Limitations**: Continuation only (not text-conditioned in base form), computationally intensive. **Impact**: Demonstrated audio can be modeled as language, opened path for transformer-based audio generation.