feedback transformers
**Feedback Transformers** are a variant of the transformer architecture that introduces a feedback connection from the output of the last layer back to the input of the first layer, creating a recurrent loop across the layer stack. At each time step, the top-layer representation from the previous step is fed back and concatenated with or added to the bottom-layer input, enabling the model to refine its representations iteratively and access global context from previous processing iterations.
**Why Feedback Transformers Matter in AI/ML:**
Feedback transformers address the **unidirectional, single-pass limitation** of standard transformers by enabling iterative refinement of representations, improving performance on tasks requiring multi-step reasoning or global context integration.
• **Top-down feedback** — The output of the final transformer layer at step t is fed back to the first layer at step t+1, creating a recurrent loop that allows higher-level abstract representations to influence lower-level processing in subsequent iterations
• **Memory via recurrence** — The feedback connection provides a form of working memory: information processed in earlier iterations persists through the feedback signal, enabling the model to maintain and update state across multiple passes over the input
• **Iterative refinement** — Complex representations benefit from multiple processing passes; feedback transformers naturally implement iterative refinement where each pass through the layer stack improves the representation using context from the previous pass
• **Attention to past representations** — Rather than simple feedback concatenation, some variants allow the first layer to attend over the history of top-layer outputs, creating an attention-based memory of all previous processing iterations
• **Training with truncated backpropagation** — The recurrent nature of feedback transformers requires either full backpropagation through time (expensive) or truncated BPTT for practical training, similar to training strategies for RNNs
| Property | Feedback Transformer | Standard Transformer |
|----------|---------------------|---------------------|
| Information Flow | Bidirectional (top↔bottom) | Unidirectional (bottom→top) |
| Processing Passes | Multiple (recurrent) | Single pass |
| Memory Mechanism | Feedback recurrence | Attention over context |
| Parameters | Same (+ feedback projection) | Standard |
| Training | BPTT or truncated BPTT | Standard backprop |
| Reasoning Depth | Deeper (iterative) | Fixed (layer count) |
| Latency | Higher (multiple passes) | Single pass |
**Feedback transformers extend the standard transformer architecture with top-down recurrent connections that enable iterative representation refinement and deeper reasoning, addressing the single-pass limitation that constrains standard transformers on tasks requiring multi-step inference and global context integration.**