speech synthesis tts

**Neural Text-to-Speech (TTS)** is the **deep learning pipeline that converts text into natural-sounding speech waveforms — typically through a two-stage architecture where an acoustic model (Tacotron, FastSpeech, VITS) converts text/phonemes into mel spectrograms, and a vocoder (WaveNet, HiFi-GAN, WaveRNN) converts mel spectrograms into audio waveforms, achieving human-level naturalness that is often indistinguishable from real speech in listening tests**. **Pipeline Architecture** **Stage 1 — Text to Mel Spectrogram (Acoustic Model)**: - Input: text string → grapheme-to-phoneme (G2P) conversion → phoneme sequence with prosody markers. - **Tacotron 2**: Encoder (character/phoneme embeddings → BiLSTM → encoded sequence) + attention-based decoder (autoregressive, predicts one mel frame at a time using the previous frame as input). Location-sensitive attention aligns input text to output mel frames. - **FastSpeech 2**: Non-autoregressive — predicts all mel frames in parallel. Duration predictor determines how many mel frames each phoneme occupies. Pitch and energy predictors provide prosody control. 10-100× faster than autoregressive Tacotron. **Stage 2 — Mel Spectrogram to Waveform (Vocoder)**: - **WaveNet**: Autoregressive — generates one audio sample at a time (16,000-24,000 samples/second). Dilated causal convolutions with exponentially increasing receptive field. Exceptional quality but extremely slow. - **WaveRNN**: Single-layer RNN generating one sample per step. Optimized for real-time on mobile CPUs through dual softmax and subscale prediction. - **HiFi-GAN**: GAN-based vocoder. Generator uses transposed convolutions to upsample mel spectrograms. Multi-period and multi-scale discriminators enforce both fine-grained and coarse waveform structure. Real-time on GPU, near-real-time on CPU. - **WaveGrad / DiffWave**: Diffusion-based vocoders. Start from Gaussian noise, iteratively refine to speech waveform conditioned on mel spectrogram. **End-to-End Models** - **VITS (Variational Inference TTS)**: Single model — text directly to waveform. VAE-based with normalizing flows and adversarial training. HiFi-GAN decoder built-in. Achieves state-of-the-art naturalness with a single forward pass. - **VALL-E (Microsoft)**: Language model approach — treats TTS as a language modeling problem over audio codec tokens. Given 3 seconds of a speaker's voice + text, generates speech in that speaker's voice (zero-shot voice cloning). Trained on 60,000 hours of speech. **Prosody and Control** - **Style Transfer**: GST (Global Style Tokens) — learn a bank of style embeddings. At inference, select or interpolate styles to control speaking style (happy, sad, whispered, shouted). - **Multi-Speaker**: Speaker embedding (d-vector or x-vector from speaker verification) conditions the acoustic model. One model serves thousands of speakers. - **Fine-Grained Control**: FastSpeech 2 allows explicit control of pitch contour, energy contour, and phoneme duration — enabling precise emotional expression and emphasis. Neural TTS is **the technology that made synthesized speech indistinguishable from human speech** — transforming text-to-speech from robotic concatenation to natural, expressive, controllable voice synthesis that powers virtual assistants, audiobooks, accessibility tools, and content creation.