Neural Text-to-Speech (TTS) is the deep learning approach to speech synthesis that converts text into natural-sounding human speech using neural networks for both linguistic feature prediction and waveform generation — replacing the robotic, concatenative systems of the past with voices that are virtually indistinguishable from human recordings, while enabling capabilities like zero-shot voice cloning from seconds of reference audio.

Two-Stage Pipeline

Most neural TTS systems use a two-stage architecture: 1. Acoustic Model: Converts text (or phoneme sequences) into intermediate acoustic representations — typically mel-spectrograms (time-frequency energy maps). Models: Tacotron 2, FastSpeech 2, VITS. 2. Vocoder: Converts the mel-spectrogram into a raw audio waveform (16-44.1 kHz samples). Models: WaveNet, WaveGlow, HiFi-GAN, BigVGAN.

Acoustic Models

• Tacotron 2: Encoder-decoder with attention. The encoder processes input text through convolutions and a bidirectional LSTM. The decoder autoregressively predicts mel-spectrogram frames, attending to the encoded text. Produces high-quality but slow speech due to autoregressive decoding.
• FastSpeech 2: Non-autoregressive model that predicts all mel-spectrogram frames in parallel using a transformer encoder and duration/pitch/energy predictors. 10-100x faster than Tacotron 2 at comparable quality.
• VITS (Variational Inference TTS): End-to-end model that combines the acoustic model and vocoder into a single network using variational autoencoders and normalizing flows. Single-stage, real-time, and high quality.

Neural Vocoders

• WaveNet: Autoregressive dilated causal convolutions predicting one audio sample at a time. Groundbreaking quality but extremely slow (minutes per second of audio).
• HiFi-GAN: GAN-based vocoder with multi-period and multi-scale discriminators. Real-time synthesis on CPU with quality approaching WaveNet. The current industry standard.
• BigVGAN: Scaled-up HiFi-GAN with anti-aliased activations, achieving state-of-the-art universal vocoding (generalizes to unseen speakers and recording conditions).

Zero-Shot Voice Cloning

• VALL-E (Microsoft): Treats TTS as a language modeling problem — encodes speech as discrete audio tokens (from a neural audio codec like EnCodec) and trains a transformer to predict audio tokens from text+speaker prompt. 3 seconds of reference audio is sufficient for high-quality cloning.
• Tortoise TTS / XTTS: Open-source voice cloning systems using similar autoregressive audio token prediction with speaker conditioning.

Recent Advances

• Diffusion-based TTS: Models like Grad-TTS and NaturalSpeech 2/3 use diffusion processes for high-fidelity mel-spectrogram or waveform generation.
• Codec Language Models: SoundStorm, VoiceBox — generate speech tokens in parallel using masked prediction, achieving real-time zero-shot TTS.

Neural TTS is the technology that gave machines a human voice — transforming speech synthesis from an uncanny approximation into a medium where artificial and natural speech are perceptually indistinguishable.