speech recognition asr

**Automatic Speech Recognition (ASR)** is the **deep learning system that converts spoken audio into text — processing raw audio waveforms through neural encoder-decoder architectures that learn to map acoustic features to linguistic tokens, achieving human-level transcription accuracy across languages and accents through end-to-end training on hundreds of thousands of hours of paired audio-text data**. **Architecture Evolution** - **Traditional Pipeline (pre-2014)**: Acoustic model (GMM-HMM) → pronunciation dictionary → language model. Each component trained separately with hand-crafted features (MFCCs). Required linguistic expertise for each language. - **Hybrid DNN-HMM (2012-2018)**: Deep neural networks replaced GMMs as acoustic models while keeping the HMM framework. Dramatic accuracy improvement but still required forced alignment and separate language models. - **End-to-End (2018+)**: Single neural network maps audio directly to text. No separate components, no forced alignment. The model implicitly learns acoustics, pronunciation, and language modeling jointly. **End-to-End Architectures** - **CTC (Connectionist Temporal Classification)**: An alignment-free loss function that sums over all valid alignments between input audio frames and output tokens. The network outputs a probability distribution over tokens at each frame; CTC marginalizes over blank and repeated tokens. Used in DeepSpeech, early production systems. Limitation: assumes output tokens are conditionally independent. - **Attention-Based Encoder-Decoder (LAS)**: Encoder (Conformer or Transformer) processes audio into hidden representations. Decoder (autoregressive Transformer) generates text tokens one at a time, attending to encoder outputs. Captures dependencies between output tokens. Higher accuracy than CTC but cannot stream (must process complete utterance before decoding). - **Transducer (RNN-T)**: Combines CTC's streaming capability with attention's label dependency modeling. A joint network combines encoder (audio) and prediction network (previous tokens) outputs to produce the next token. The standard architecture for on-device streaming ASR (Google, Apple). **Whisper (OpenAI, 2022)** Trained on 680,000 hours of weakly-supervised web audio in 99 languages. Encoder-decoder Transformer with multitask training: transcription, translation, language identification, timestamp prediction — all controlled by text prompts. Achieves near-human accuracy on English without any fine-tuning. Demonstrated that scaling data (not architecture novelty) was the primary bottleneck for robust ASR. **Audio Feature Processing** - **Mel Spectrogram**: Audio signal → Short-Time Fourier Transform (STFT) → Mel-scale frequency binning → log amplitude. Produces a 2D time-frequency representation (80-128 mel bins × time frames at 10-20 ms intervals) that serves as input to the encoder. - **Conformer Encoder**: Combines convolution (local patterns — phonemes) with self-attention (global context — prosody, speaker characteristics). The dominant encoder architecture achieving state-of-the-art on all ASR benchmarks. Automatic Speech Recognition is **the interface between human speech and machine understanding** — a technology that has progressed from 50% word error rates to human-parity accuracy in a decade, enabling voice assistants, real-time captioning, and multilingual communication at planetary scale.