Neural Audio and Music Generation is the application of generative AI to synthesize music, sound effects, and audio from text descriptions or other conditioning inputs — using architectures like autoregressive codec language models (MusicGen, MusicLM), audio diffusion models (Stable Audio, Riffusion), and hybrid approaches to generate coherent, musically structured audio that captures rhythm, melody, harmony, and timbre, representing a frontier where AI meets creative expression.

Audio Generation Architectures

Audio Representation for Generation

Raw audio: 44.1 kHz × 16 bits = 705,600 bits/second → too high-dimensional

Solution 1: Mel Spectrogram
  Time-frequency representation → treat as image → use image diffusion
  Resolution: ~86 frames/sec × 80⁠-128 mel bins

Solution 2: Neural Audio Codec (EnCodec, DAC)
  Compress audio into discrete tokens via VQ-VAE
  ~50-75 tokens/second × 4-8 codebook levels
  Enables: Language-model-style autoregressive generation

Solution 3: Latent audio representation
  VAE compresses spectrogram into continuous latent space
  Run diffusion in this compressed space (like Stable Diffusion for images)

MusicGen (Meta)

[Text: "upbeat electronic dance music with heavy bass"]
         ↓
[T5 text encoder] → text conditioning
         ↓
[Autoregressive transformer over EnCodec tokens]
  Generates codebook tokens level by level:
  Level 1 (coarse/semantic): Full autoregressive
  Levels 2-4 (fine/acoustic): Parallel or delayed pattern
         ↓
[EnCodec decoder] → waveform
         ↓
[30 seconds of generated music]

• Sizes: 300M, 1.5B, 3.3B parameters.
• Conditioning: Text, melody (humming → genre transfer), continuation.
• Open source (Meta), runs locally.

Stable Audio (Stability AI)

[Text + timing info] → [T5 encoder + timing embedder]
         ↓
[Latent diffusion model] (operates on latent audio spectrogram)
         ↓
[VAE decoder + HiFi-GAN vocoder] → high-quality waveform

• Generates: Up to 3 minutes of 44.1 kHz stereo audio.
• Timing conditioning: Control exact duration and structure.
• Applications: Music, sound effects, ambient audio.

Major Music AI Systems

Evaluation Challenges

Current Limitations

• Structure: Long-term musical structure (verse-chorus-bridge) still challenging.
• Lyrics: Coherent singing with understandable lyrics is emerging but imperfect.
• Style control: Fine-grained control over instrumentation and mixing is limited.
• Copyright: Legal questions around training on copyrighted music.

Neural audio generation is transforming music creation from a specialized skill to an accessible creative tool — by enabling anyone to describe the music they imagine and receive professional-quality audio in seconds, these systems are democratizing music production while opening new creative possibilities for composers, filmmakers, game developers, and content creators who need custom audio on demand.