Automatic Speech Recognition (ASR) is the deep learning system that converts spoken audio into text — processing raw audio waveforms through neural encoder-decoder architectures that learn to map acoustic features to linguistic tokens, achieving human-level transcription accuracy across languages and accents through end-to-end training on hundreds of thousands of hours of paired audio-text data.

Architecture Evolution

• Traditional Pipeline (pre-2014): Acoustic model (GMM-HMM) → pronunciation dictionary → language model. Each component trained separately with hand-crafted features (MFCCs). Required linguistic expertise for each language.
• Hybrid DNN-HMM (2012-2018): Deep neural networks replaced GMMs as acoustic models while keeping the HMM framework. Dramatic accuracy improvement but still required forced alignment and separate language models.
• End-to-End (2018+): Single neural network maps audio directly to text. No separate components, no forced alignment. The model implicitly learns acoustics, pronunciation, and language modeling jointly.

End-to-End Architectures

• CTC (Connectionist Temporal Classification): An alignment-free loss function that sums over all valid alignments between input audio frames and output tokens. The network outputs a probability distribution over tokens at each frame; CTC marginalizes over blank and repeated tokens. Used in DeepSpeech, early production systems. Limitation: assumes output tokens are conditionally independent.
• Attention-Based Encoder-Decoder (LAS): Encoder (Conformer or Transformer) processes audio into hidden representations. Decoder (autoregressive Transformer) generates text tokens one at a time, attending to encoder outputs. Captures dependencies between output tokens. Higher accuracy than CTC but cannot stream (must process complete utterance before decoding).
• Transducer (RNN-T): Combines CTC's streaming capability with attention's label dependency modeling. A joint network combines encoder (audio) and prediction network (previous tokens) outputs to produce the next token. The standard architecture for on-device streaming ASR (Google, Apple).

Whisper (OpenAI, 2022)

Trained on 680,000 hours of weakly-supervised web audio in 99 languages. Encoder-decoder Transformer with multitask training: transcription, translation, language identification, timestamp prediction — all controlled by text prompts. Achieves near-human accuracy on English without any fine-tuning. Demonstrated that scaling data (not architecture novelty) was the primary bottleneck for robust ASR.

Audio Feature Processing

• Mel Spectrogram: Audio signal → Short-Time Fourier Transform (STFT) → Mel-scale frequency binning → log amplitude. Produces a 2D time-frequency representation (80-128 mel bins × time frames at 10-20 ms intervals) that serves as input to the encoder.
• Conformer Encoder: Combines convolution (local patterns — phonemes) with self-attention (global context — prosody, speaker characteristics). The dominant encoder architecture achieving state-of-the-art on all ASR benchmarks.

Automatic Speech Recognition is the interface between human speech and machine understanding — a technology that has progressed from 50% word error rates to human-parity accuracy in a decade, enabling voice assistants, real-time captioning, and multilingual communication at planetary scale.