encoder decoder
**Encoder-Decoder Models** are **transformer architectures that process input through a bidirectional encoder and generate output through an autoregressive decoder with cross-attention** — separating the "understanding" phase (encoder reads the full input with bidirectional attention) from the "generation" phase (decoder produces output tokens attending to both previous output tokens and the encoder's representations), as exemplified by T5, BART, and mBART for tasks like translation, summarization, and question answering.
**What Is an Encoder-Decoder Model?**
- **Definition**: A sequence-to-sequence architecture with two distinct components — an encoder that processes the input sequence with bidirectional self-attention (each token attends to all other tokens), and a decoder that generates the output sequence autoregressively with causal self-attention plus cross-attention to the encoder's output representations.
- **T5 (Text-to-Text Transfer Transformer)**: Google's encoder-decoder model that unifies all NLP tasks into a text-to-text format — classification becomes "sentiment: positive", summarization takes "summarize: [text]", and translation takes "translate English to French: [text]". Pre-trained with span corruption (mask and predict text spans).
- **Cross-Attention**: The decoder's cross-attention mechanism allows each generated token to attend to all positions in the encoder output — this is how the decoder "reads" the input while generating the output, providing full bidirectional access to the input context.
- **Bidirectional Encoding**: Unlike decoder-only models where each position can only see previous tokens, the encoder processes the full input with bidirectional attention — every token can attend to every other token, providing richer contextual representations.
**Why Encoder-Decoder Matters**
- **Bidirectional Understanding**: The encoder's bidirectional attention captures richer input representations than causal attention — particularly beneficial for tasks where understanding the full input context is critical (translation, summarization, question answering).
- **Structured Output**: Encoder-decoder naturally handles tasks where input and output are different sequences — translation (English → French), summarization (long text → short summary), and question answering (context + question → answer).
- **T5 Unification**: T5 demonstrated that framing all NLP tasks as text-to-text enables a single model architecture and training procedure for diverse tasks — simplifying the ML pipeline.
- **Efficiency for Short Outputs**: When the output is much shorter than the input (summarization), encoder-decoder can be more efficient — the encoder processes the long input once, and the decoder generates only the short output.
**Encoder-Decoder Models**
| Model | Parameters | Pre-Training | Key Innovation |
|-------|-----------|-------------|---------------|
| T5 | 60M-11B | Span corruption | Text-to-text unification |
| Flan-T5 | 80M-11B | Instruction tuning on T5 | Zero-shot task generalization |
| BART | 140M-400M | Denoising autoencoder | Flexible corruption strategies |
| mBART | 680M | Multilingual denoising | 25-language translation |
| mT5 | 300M-13B | Multilingual span corruption | 101-language coverage |
| UL2 | 20B | Mixture of denoisers | Unified pre-training |
**Encoder-decoder models are the natural architecture for sequence-to-sequence tasks** — leveraging bidirectional encoding for rich input understanding and autoregressive decoding with cross-attention for flexible output generation, with T5 and Flan-T5 demonstrating that the text-to-text framework enables a single model to handle translation, summarization, classification, and question answering through unified training.