fusion-in-decoder (fid)
**Fusion-in-Decoder (FiD)** is the **retrieval-augmented generation architecture that processes multiple retrieved documents independently through the encoder and fuses information from all documents in the decoder through cross-attention — enabling scalable multi-document reasoning without the context-length limitations of concatenation-based approaches** — the architectural pattern that became the standard backbone for retrieval-augmented question answering and knowledge-grounded generation systems.
**What Is Fusion-in-Decoder?**
- **Definition**: An encoder-decoder architecture (based on T5 or BART) where each retrieved passage is encoded independently with the query by the encoder, producing separate representations, and the decoder cross-attends to all encoder outputs simultaneously — performing information fusion across documents at the decoding stage.
- **Independent Encoding**: Each of k retrieved passages is concatenated with the query and encoded separately: hᵢ = Encoder(query ⊕ passageᵢ). This avoids the O(k²·n²) cost of concatenating all passages and running a single encoder.
- **Decoder Fusion**: The decoder cross-attends to the concatenated encoder outputs [h₁; h₂; ...; hₖ] — each decoder token can attend to any position in any retrieved passage, enabling information synthesis across documents.
- **Scalability**: Since encoding is independent and parallelizable, FiD scales to 50–100 retrieved passages without exceeding memory limits — far more context than concatenation allows.
**Why FiD Matters**
- **Scales to Many Documents**: Concatenating 50 passages of 200 tokens creates a 10,000-token input — exceeding most encoder limits. FiD encodes each passage independently (200 tokens each) and fuses in the decoder — handling any number of passages.
- **State-of-the-Art QA**: FiD achieved top results on Natural Questions, TriviaQA, and other open-domain QA benchmarks — demonstrating that multi-document fusion in the decoder is more effective than early fusion (concatenation) or late fusion (reranking).
- **Information Aggregation**: When the answer requires combining facts from multiple documents (multi-hop reasoning), FiD's decoder naturally learns to attend to different passages for different parts of the answer.
- **Foundation for ATLAS and RAG**: FiD became the generator component in ATLAS and influenced the design of many RAG systems — its encoder-decoder fusion pattern is the standard architectural choice for retrieval-augmented generation.
- **Efficient Encoding**: Independent passage encoding enables passage-level caching — when the corpus is fixed, encoder outputs can be pre-computed and reused across queries.
**FiD Architecture**
**Encoding Phase (Parallelized)**:
- For each retrieved passage pᵢ (i = 1, ..., k):
- Concatenate: inputᵢ = "question: [query] context: [passageᵢ]"
- Encode: hᵢ = T5Encoder(inputᵢ) → [seq_lenᵢ × d_model]
- All k passages encoded independently — embarrassingly parallel.
- Total encoder memory: O(k × max_passage_len × d_model).
**Fusion Phase (Decoder)**:
- Concatenate all encoder outputs: H = [h₁; h₂; ...; hₖ] → [k × seq_len × d_model].
- Decoder cross-attention attends to full H — each generated token can access any position in any passage.
- Decoder generates the answer auto-regressively.
**FiD Behavior Analysis**
| Number of Passages (k) | Natural Questions (EM) | Encoding Cost | Decoder Cost |
|------------------------|----------------------|---------------|-------------|
| **10** | 44.1% | Low | Low |
| **25** | 48.2% | Medium | Medium |
| **50** | 50.1% | Medium | Higher |
| **100** | 51.4% | High | Highest |
**Log-linear improvement**: Performance scales logarithmically with number of passages — strong early gains with diminishing returns beyond 50 passages.
**FiD vs. Alternative Fusion Strategies**
| Strategy | Approach | Max Passages | Quality |
|----------|----------|-------------|---------|
| **Concatenation** | All passages in one encoder input | ~5–10 | Limited by context length |
| **FiD** | Independent encoding, decoder fusion | 50–100+ | Best for many passages |
| **Reranking** | Select best single passage | 1 (final) | Loses multi-document info |
| **Iterative** | Sequential document reading | Variable | Complex, slower |
Fusion-in-Decoder is **the architectural workhorse of retrieval-augmented generation** — solving the fundamental scalability problem of multi-document reasoning by separating independent passage understanding (encoder) from cross-document information synthesis (decoder), enabling systems to effectively aggregate knowledge from dozens of retrieved documents into coherent, informed answers.