automatic speech recognition asr

**Automatic Speech Recognition (ASR)** is the **task of converting speech audio to text — employing neural networks with CTC loss, encoder-decoder architectures, and self-supervised pretraining to achieve high accuracy competitive with human performance on various domains**. **CTC Loss (Connectionist Temporal Classification):** - Alignment problem: speech frames ~30-100ms; target tokens variable duration; CTC solves alignment automatically - Blank token: CTC introduces special blank token for non-speech frames; enables flexible alignments - Forward-backward algorithm: efficiently computes probability of output sequence over all alignments - Training: minimize CTC loss (summed over all valid alignments); no manual frame-level alignment needed - Decoding: greedy selection or beam search; CTC removes consecutive duplicates and blanks - Advantages: enables end-to-end training; reduces pipeline complexity vs. traditional HMM-GMM systems **Encoder-Decoder Architecture (RNN-T/Transformer-Transducer):** - Encoder: BiLSTM or Transformer processes entire audio input; outputs context vector - Decoder: RNN predicts output tokens autoregressively; attends to encoder for context - Attention mechanism: soft attention over encoder outputs; learns to focus on relevant audio frames - Joint modeling: combines attention + autoregressive decoding; flexible architectures - Streaming capability: can process streaming audio (chunk-based processing) with appropriate modifications **Wav2Vec 2.0 Self-Supervised Pretraining:** - Masked prediction: mask input audio frames; predict masked frames from surrounding context - Contrastive learning: distinguish true target from negatives sampled from codebook - Learned quantization: continuous features quantized to discrete codebook; enables contrastive setup - Foundation model: pretrain on unlabeled audio (100x more than labeled); transfer to downstream ASR - Dramatic improvement: wav2vec 2.0 pretraining enables strong ASR with limited labeled data - Multilingual wav2vec: XLSR pretrains on 128 languages; enables zero-shot cross-lingual transfer **Conformer Architecture:** - Hybrid design: interleaves convolutional blocks (local feature extraction) with transformer blocks (long-range context) - Convolutional blocks: depthwise separable convolutions capture local patterns; positional information - Transformer blocks: multi-head self-attention captures long-range dependencies; parallel processing - Macaron-style FFN: position-wise feed-forward networks; improves gradient flow - Performance: Conformer achieves state-of-the-art on LibriSpeech, CommonVoice; outperforms pure CNN/RNN/Transformer **Language Model Integration:** - Shallow fusion: add language model logits to acoustic model logits during decoding; simple post-hoc method - Deep fusion: incorporate language model predictions into intermediate decoder layers; better integration - Shallow+deep fusion: combine both shallow and deep fusion; further improvements - External ARPA n-gram LMs: traditional language models integrated with neural acoustic models - Neural language models: LSTM or transformer LMs trained on text corpus; capture language structure **Beam Search Decoding:** - Heuristic search: maintain K best hypotheses (beam width); expand beam by predicting next token - Pruning: remove low-probability hypotheses; maintain tractable beam width (typically 8-128) - Language model rescoring: rerank beam hypotheses using language model probabilities - Length normalization: penalize overly long/short hypotheses; encourage appropriate sequence lengths - Inference speed: larger beam width improves accuracy but increases latency; accuracy-latency tradeoff **Word Error Rate (WER) Evaluation:** - WER metric: 100 * (S + D + I) / N; S = substitutions, D = deletions, I = insertions, N = reference words - Benchmark datasets: LibriSpeech (1000 hours clean/noisy English), CommonVoice (multilingual), VoxPopuli (European Parliament) - State-of-the-art: Conformer + wav2vec 2.0 + LM achieves ~2-3% WER on LibriSpeech test-clean - Robustness: test-other subset; noisy conditions with background noise, speakers, reverberation **Real-World ASR Challenges:** - Acoustic variation: speaker differences, background noise, reverberation, accents; robust acoustic modeling - Domain mismatch: training data distribution different from deployment; domain adaptation techniques - Streaming constraints: online streaming ASR requires low latency; incompatible with full lookahead - Computational constraints: edge deployment requires model compression; quantization, pruning, distillation - Multilingual/code-switching: handling multiple languages within single utterance; shared representations **ASR System Components:** - Feature extraction: Mel-frequency cepstral coefficients (MFCC) or log-Mel spectrogram; acoustic features - Normalization: mean-variance normalization per utterance; stabilizes training - Augmentation: SpecAugment (mask frequency/time bands); improves robustness without additional data - Contextualization: biased language models for domain-specific terms; personalization and named entities **Automatic speech recognition converts audio to text using neural networks with CTC alignment or encoder-decoder architectures — leveraging self-supervised pretraining (wav2vec 2.0) and language models to achieve near-human performance.**