text to speech synthesis tts

**Neural Text-to-Speech (TTS)** is the **deep learning system that converts written text into natural-sounding human speech — using neural network acoustic models to generate mel spectrograms from text, followed by neural vocoders that synthesize raw audio waveforms, achieving speech quality indistinguishable from human recordings and enabling voice cloning, multilingual synthesis, and emotional speech generation**. **TTS Pipeline** **Text Processing (Front-End)**: - Text normalization: expand abbreviations, numbers, dates ("$3.5M" → "three point five million dollars"). - Grapheme-to-phoneme (G2P): convert text to phoneme sequences using pronunciation dictionaries (CMUDict) or neural G2P models. - Prosody prediction: determine stress patterns, phrasing, and intonation from context. **Acoustic Model (Text → Mel Spectrogram)**: - **Tacotron 2**: Encoder-decoder with attention. Character/phoneme encoder → location-sensitive attention → autoregressive decoder producing mel spectrogram frames. Natural prosody but slow autoregressive generation. - **FastSpeech 2**: Non-autoregressive — predicts all mel frames in parallel using duration, pitch, and energy predictors. 100×+ faster than Tacotron 2. Duration predictor trained from forced alignment data. - **VITS (Variational Inference TTS)**: End-to-end model combining acoustic model and vocoder. Uses variational autoencoder + normalizing flows + adversarial training. Single-model text-to-waveform with near-human quality. - **VALL-E / Bark / XTTS**: Treat TTS as a language modeling problem — predict discrete audio tokens (from a neural codec like EnCodec) autoregressively, conditioned on text and a short audio prompt. Enables zero-shot voice cloning from 3-10 seconds of reference audio. **Neural Vocoder (Mel → Waveform)**: - **WaveNet**: Autoregressive sample-by-sample generation. Highest quality but extremely slow (minutes per second of audio). - **WaveGlow / HiFi-GAN**: Non-autoregressive. HiFi-GAN uses a GAN-based generator that upsamples mel spectrograms to 22/44 kHz waveforms in real-time. GPU inference: >100× real-time speed. - **BigVGAN**: Improved HiFi-GAN with anti-aliased activations, achieving state-of-the-art vocoder quality. **Voice Cloning** - **Speaker Conditioning**: Train a multi-speaker TTS model conditioned on speaker embeddings (d-vectors or x-vectors). At inference, provide a target speaker's embedding to generate speech in their voice. - **Few-Shot Cloning**: VALL-E, XTTS, and similar models clone a voice from 3-30 seconds of audio. The reference audio is encoded into discrete tokens that condition the generation of new speech. - **Fine-Tuning**: For highest quality, fine-tune a pre-trained TTS model on 5-30 minutes of target speaker data. Produces near-perfect voice reproduction. **Evaluation Metrics** - **MOS (Mean Opinion Score)**: Human listeners rate naturalness on a 1-5 scale. State-of-the-art neural TTS achieves MOS 4.2-4.6 (human speech: ~4.5). - **Character Error Rate (CER)**: Measure intelligibility by running ASR on generated speech. Good TTS achieves <2% CER. - **Speaker Similarity**: Cosine similarity between speaker embeddings of generated and reference speech. Neural TTS is **the technology that gave machines human-quality voices** — transforming text-to-speech from robotic concatenation of recorded syllables to fluid, expressive, and personalized speech synthesis that powers virtual assistants, audiobook narration, accessibility tools, and real-time translation.