whisper

Whisper is OpenAI's robust multilingual speech recognition model trained on 680,000 hours of diverse audio data, achieving state-of-the-art transcription accuracy across languages, accents, and acoustic conditions with open-source weights. Architecture: encoder-decoder transformer—(1) audio encoder (log-mel spectrogram → embeddings), (2) decoder (autoregressive text generation). Model sizes: tiny (39M), base (74M), small (244M), medium (769M), large (1.5B)—trade-off between speed and accuracy. Training: weakly supervised on internet audio with transcripts—multilingual (99 languages), multitask (transcription, translation, language detection, voice activity detection). Key features: (1) multilingual (transcribe 99 languages), (2) robust (handles noise, accents, music), (3) punctuation and capitalization (proper formatting), (4) timestamps (word-level timing), (5) translation (translate non-English to English). Input: 30-second audio chunks (16kHz, mono)—longer audio split into chunks. Output: text transcript with optional timestamps. Special tokens: <|startoftranscript|>, <|language|>, <|task|>, <|notimestamps|>—control generation. Performance: achieves human-level accuracy on clean speech, degrades gracefully on noisy/accented speech. Outperforms commercial systems on diverse conditions. Applications: (1) transcription (meetings, podcasts, videos), (2) subtitles (automatic captioning), (3) voice assistants (speech-to-text), (4) accessibility (hearing impaired), (5) translation (speech translation). Deployment: (1) OpenAI API (cloud), (2) open-source (run locally—whisper Python package), (3) optimized (faster-whisper, whisper.cpp for CPU). Limitations: (1) hallucinations (may generate plausible but incorrect text), (2) latency (large models slow for real-time), (3) language mixing (struggles with code-switching). Whisper democratized high-quality speech recognition, providing open-source alternative to commercial systems with superior robustness and multilingual support.