Speech Language Models are the foundation models that process and generate speech directly as a native modality — either by tokenizing audio into discrete units that language models can process alongside text, or by operating on continuous audio representations, enabling unified models that can transcribe, translate, converse, and generate speech in a single architecture rather than cascading separate ASR → LLM → TTS systems.

Evolution of Speech AI

Era 1 (pre-2020): Separate ASR → NLU → TTS pipeline
  [Audio] → [ASR: DeepSpeech/wav2vec] → [Text] → [NLU] → [Text] → [TTS] → [Audio]
  Problem: Error propagation, high latency, loses prosody/emotion

Era 2 (2023+): Speech Language Models
  [Audio] → [Speech LM] → [Audio + Text]
  Unified model handles everything end-to-end

Key Systems

Audio Tokenization Approaches

Whisper Architecture

[Audio waveform] → [Mel spectrogram] → [Transformer Encoder]
                                              ↓
                                    [Transformer Decoder] → [Text tokens]

• Trained on 680,000 hours of labeled audio from the internet.
• Multitask: Transcription, translation, language identification, timestamp prediction.
• Robust: Works across accents, background noise, technical terminology.
• Sizes: Tiny (39M) to Large-v3 (1.5B parameters).

Neural Codec Language Models (VALL-E)

• Step 1: Encode speech with neural codec (EnCodec) → 8 codebooks of discrete tokens.
• Step 2: Train autoregressive LM on first codebook (semantic content).
• Step 3: Train non-autoregressive model for remaining codebooks (acoustic detail).
• Result: Given 3 seconds of someone's voice → generate arbitrary speech in that voice.
• Implication: Zero-shot voice cloning with natural prosody and emotion.

Full-Duplex Speech AI

• Traditional: Half-duplex — system listens OR speaks, never both.
• GPT-4o / Moshi: Full-duplex — can listen while speaking, handle interruptions.
• Architecture: Streaming input + streaming output simultaneously.
• Enables: Natural conversation flow, backchanneling ("mmhmm"), interruption handling.

Training Data Scale

Speech language models are the technology that will make AI conversational interfaces indistinguishable from human interaction — by processing speech as a native modality rather than converting to text as an intermediate step, these models preserve the full richness of spoken communication including tone, emotion, and timing, enabling real-time AI assistants that can truly converse rather than merely chat.