Neural Text-to-Speech (TTS) is the deep learning pipeline that converts text into natural-sounding speech waveforms — typically through a two-stage architecture where an acoustic model (Tacotron, FastSpeech, VITS) converts text/phonemes into mel spectrograms, and a vocoder (WaveNet, HiFi-GAN, WaveRNN) converts mel spectrograms into audio waveforms, achieving human-level naturalness that is often indistinguishable from real speech in listening tests.

Pipeline Architecture

Stage 1 — Text to Mel Spectrogram (Acoustic Model):

• Input: text string → grapheme-to-phoneme (G2P) conversion → phoneme sequence with prosody markers.
• Tacotron 2: Encoder (character/phoneme embeddings → BiLSTM → encoded sequence) + attention-based decoder (autoregressive, predicts one mel frame at a time using the previous frame as input). Location-sensitive attention aligns input text to output mel frames.
• FastSpeech 2: Non-autoregressive — predicts all mel frames in parallel. Duration predictor determines how many mel frames each phoneme occupies. Pitch and energy predictors provide prosody control. 10-100× faster than autoregressive Tacotron.

Stage 2 — Mel Spectrogram to Waveform (Vocoder):

• WaveNet: Autoregressive — generates one audio sample at a time (16,000-24,000 samples/second). Dilated causal convolutions with exponentially increasing receptive field. Exceptional quality but extremely slow.
• WaveRNN: Single-layer RNN generating one sample per step. Optimized for real-time on mobile CPUs through dual softmax and subscale prediction.
• HiFi-GAN: GAN-based vocoder. Generator uses transposed convolutions to upsample mel spectrograms. Multi-period and multi-scale discriminators enforce both fine-grained and coarse waveform structure. Real-time on GPU, near-real-time on CPU.
• WaveGrad / DiffWave: Diffusion-based vocoders. Start from Gaussian noise, iteratively refine to speech waveform conditioned on mel spectrogram.

End-to-End Models

• VITS (Variational Inference TTS): Single model — text directly to waveform. VAE-based with normalizing flows and adversarial training. HiFi-GAN decoder built-in. Achieves state-of-the-art naturalness with a single forward pass.
• VALL-E (Microsoft): Language model approach — treats TTS as a language modeling problem over audio codec tokens. Given 3 seconds of a speaker's voice + text, generates speech in that speaker's voice (zero-shot voice cloning). Trained on 60,000 hours of speech.

Prosody and Control

• Style Transfer: GST (Global Style Tokens) — learn a bank of style embeddings. At inference, select or interpolate styles to control speaking style (happy, sad, whispered, shouted).
• Multi-Speaker: Speaker embedding (d-vector or x-vector from speaker verification) conditions the acoustic model. One model serves thousands of speakers.
• Fine-Grained Control: FastSpeech 2 allows explicit control of pitch contour, energy contour, and phoneme duration — enabling precise emotional expression and emphasis.

Neural TTS is the technology that made synthesized speech indistinguishable from human speech — transforming text-to-speech from robotic concatenation to natural, expressive, controllable voice synthesis that powers virtual assistants, audiobooks, accessibility tools, and content creation.