Speech to Text

Speech-to-text, also called automatic speech recognition (ASR), is the technology that converts spoken audio into written text. The input is a waveform, a long stream of air-pressure samples (commonly 16,000 per second), and the output is a sequence of words; the difficulty is that the same word sounds different across speakers, accents, microphones and background noise, while different words can sound alike ("write" and "right"), so recognition requires combining acoustic evidence with knowledge of which word sequences are likely in the language. Its uses range from captions and meeting notes to voice assistants, dictation, call-center analytics and accessibility tools.

For several decades the standard design was a pipeline of separate hand-engineered parts built around hidden Markov models and Gaussian mixture models (HMM-GMM). Audio was cut into 10-millisecond frames and converted into features such as MFCCs (mel-frequency cepstral coefficients), an acoustic model scored how well each frame matched small sound units called phonemes, a pronunciation dictionary linked phonemes to words, and a statistical language model (typically n-grams) scored word sequences, with a decoder searching for the best overall path. This approach, developed through the 1970s to 1990s at places such as IBM, Carnegie Mellon and Bell Labs and commercialized in products like Dragon NaturallySpeaking in 1997, worked but was fragile and expensive to build per language. In 2012 Hinton and colleagues showed that deep neural networks could replace the GMMs for scoring frames with large error reductions, and for a few years hybrid DNN-HMM systems dominated.

The move to end-to-end neural models replaced most of that pipeline with a single network trained directly on audio and transcripts. Connectionist temporal classification (CTC, Graves, Fernández, Gomez and Schmidhuber, 2006) solved the alignment problem, that audio frames far outnumber characters and nobody labels which frame belongs to which letter, by summing over all possible alignments and adding a "blank" symbol. Baidu's Deep Speech (Hannun and colleagues, 2014) and Deep Speech 2 (2015) showed that CTC with recurrent networks and lots of data could match or beat engineered systems; Listen, Attend and Spell (Chan and colleagues, 2015) used an attention-based encoder-decoder; and the RNN transducer (Graves, 2012) became the favorite for streaming on-device recognition because it emits words as audio arrives. Self-supervised pretraining then reduced the need for transcripts: wav2vec 2.0 (Baevski and colleagues, 2020) learns speech representations from unlabeled audio and reaches strong accuracy with only minutes to hours of labeled data.

Whisper, released by OpenAI in September 2022 ("Robust Speech Recognition via Large-Scale Weak Supervision", Radford and colleagues), took the opposite bet on data: train a plain transformer encoder-decoder on 680,000 hours of imperfectly labeled audio collected from the web, covering many languages and tasks. One model transcribes in roughly a hundred languages, translates speech into English, identifies the spoken language and predicts timestamps, using special task tokens in the decoder's prompt, and its robustness to accents, noise and domain shift is its main selling point because it is not tuned to any single benchmark. The weights were open-sourced in several sizes, which made good transcription a local, low-cost tool, and derivatives such as faster inference ports and diarization add-ons are common in practice.

Accuracy is reported as word error rate (WER): count the words substituted, deleted and inserted relative to a human reference transcript, and divide by the number of reference words, so lower is better and the score can even exceed 100 percent. Headline claims need context. Microsoft reported in 2016 reaching a 5.9 percent WER on the Switchboard telephone-conversation benchmark, comparable to the human transcription rate it measured, but conversational speech in noisy rooms, with overlapping talkers, strong accents, jargon, names and code-switching remains much harder, and WER is an uneven yardstick since it counts a missed "not" the same as a missed "the". Pitfalls include models that hallucinate plausible text over silence or music (reported for Whisper-style decoders), poor performance on under-represented languages and dialects, unreliable punctuation, casing and numbers unless normalized, and privacy considerations, since transcripts of calls are sensitive data.

Read speech-to-text as probability over words given sound: it began as separate acoustic, pronunciation and language models stitched together by a decoder, collapsed into single end-to-end networks through CTC and attention, and now relies on large-scale transformers such as Whisper, with word error rate as the honest but imperfect scoreboard.