Sequence-to-Sequence (Seq2Seq) Models are the neural network architecture pattern where an encoder processes a variable-length input sequence into a fixed or variable-length representation, and a decoder generates a variable-length output sequence from that representation — the foundational architecture for machine translation, summarization, speech recognition, and any task that maps one sequence to another of potentially different length.

Seq2Seq Evolution

Original RNN Seq2Seq

1. Encoder RNN: Processes input tokens x₁...xₙ → produces final hidden state hₙ (context vector). 2. Context vector: Fixed-size summary of entire input → bottleneck! 3. Decoder RNN: Initialized with context vector → generates output tokens y₁...yₘ autoregressively.

• Problem: Fixed-size context vector cannot capture all information from long sequences.

Attention Mechanism (Bahdanau, 2015)

• Instead of single context vector: Decoder attends to ALL encoder hidden states.
• At each decoder step t: Compute attention weights over encoder states → weighted sum = context.
• $c_t = \sum_i \alpha_{t,i} h_i$ where $\alpha_{t,i} = \text{softmax}(\text{score}(s_t, h_i))$
• Result: Decoder can focus on relevant parts of input → dramatically improved translation quality.

Transformer Encoder-Decoder

• Encoder: Stack of self-attention + FFN layers → contextual representations.
• Decoder: Masked self-attention + cross-attention to encoder + FFN.
• Cross-attention: Decoder queries attend to encoder outputs (keys and values from encoder).
• Fully parallelizable (no recurrence) → much faster training.

Pre-trained Seq2Seq Models

Seq2Seq vs. Decoder-Only

Sequence-to-sequence models are the architectural foundation that enabled neural approaches to surpass traditional methods in machine translation and structured generation — while decoder-only models now dominate general-purpose language modeling, the encoder-decoder pattern remains the superior choice for tasks with distinct input and output sequences.