self supervised speech models

**Self-Supervised Speech Models** are **foundation models pretrained on large corpora of unlabeled audio that learn general-purpose speech representations through contrastive, predictive, or masked reconstruction objectives** — enabling state-of-the-art performance on downstream tasks including automatic speech recognition, speaker verification, emotion detection, and language identification with minimal labeled data. **Pretraining Paradigms:** - **Contrastive Learning (Wav2Vec 2.0)**: Mask portions of the latent speech representation, then train the model to identify the correct latent among distractors using a contrastive loss (InfoNCE), forcing the network to learn contextual speech features from the surrounding audio context - **Masked Prediction (HuBERT)**: Use offline clustering (k-means) on MFCC or earlier-iteration features to create pseudo-labels, then predict these discrete targets for masked frames — iteratively refining cluster quality as the model improves - **Auto-Regressive Prediction**: Predict future audio frames from past context, as in Autoregressive Predictive Coding (APC) and Contrastive Predictive Coding (CPC) - **Multi-Task Pretraining (Whisper)**: Train on 680,000 hours of weakly supervised audio-transcript pairs in a multitask format covering transcription, translation, language identification, and timestamp prediction - **Encoder-Decoder Pretraining (USM/AudioPaLM)**: Combine self-supervised encoder pretraining with supervised decoder fine-tuning across dozens of languages simultaneously **Architecture Details:** - **Feature Encoder**: A multi-layer 1D convolutional network converts raw 16kHz waveform into latent representations at 20ms frame resolution (50Hz) - **Contextualization**: A Transformer encoder (12–48 layers) processes the latent sequence to produce contextualized representations capturing long-range dependencies - **Quantization Module**: Wav2Vec 2.0 uses a Gumbel-softmax quantizer to discretize continuous latents into codebook entries for the contrastive objective - **Relative Positional Encoding**: Convolutional positional embeddings or rotary encoding provide sequence position information without fixed-length limitations - **Model Scales**: Range from Wav2Vec 2.0 Base (95M parameters) to Whisper Large-v3 (1.5B parameters) and USM (2B parameters) **Key Models and Capabilities:** - **Wav2Vec 2.0**: Demonstrated that with only 10 minutes of labeled speech, self-supervised pretraining achieves competitive ASR performance compared to fully supervised systems trained on 960 hours - **HuBERT**: Improved on Wav2Vec 2.0 by using offline discovered units as targets, achieving better downstream performance and generating more consistent representations - **WavLM**: Extended HuBERT with denoising objectives and additional data, excelling on the SUPERB benchmark across diverse speech processing tasks - **Whisper**: OpenAI's weakly supervised model trained on internet audio, providing robust zero-shot transcription across 99 languages with punctuation and formatting - **SeamlessM4T**: Meta's multimodal translation model handling speech-to-speech, speech-to-text, and text-to-speech translation across nearly 100 languages **Fine-Tuning and Downstream Tasks:** - **ASR (Automatic Speech Recognition)**: Add a CTC or attention-based decoder head on top of pretrained representations and fine-tune with labeled transcripts - **Speaker Verification**: Extract utterance-level embeddings from intermediate or final layers for speaker identity comparison - **Emotion Recognition**: Use weighted combinations of all Transformer layers (learnable layer weights) to capture both acoustic and linguistic cues - **Language Identification**: Global average pooling over frame-level features followed by a classifier head identifies the spoken language - **Speech Translation**: Combine speech encoder with a text decoder to directly translate spoken audio to text in another language **Practical Deployment:** - **Computational Cost**: Whisper Large requires approximately 10x real-time factor on CPU but achieves real-time on modern GPUs; distilled variants (Distil-Whisper) run 6x faster with minimal quality loss - **Streaming Adaptation**: Most self-supervised models are non-causal; adapting them for streaming requires chunked attention, causal masking, or dedicated architectures like Emformer - **Noise Robustness**: Models pretrained on diverse audio (Whisper, WavLM) exhibit strong robustness to background noise, reverberation, and overlapping speakers Self-supervised speech models have **transformed speech technology by decoupling representation learning from task-specific supervision — enabling high-quality speech processing systems to be built for low-resource languages and novel tasks with orders of magnitude less labeled data than previously required**.