Feedback Transformers are a variant of the transformer architecture that introduces a feedback connection from the output of the last layer back to the input of the first layer, creating a recurrent loop across the layer stack. At each time step, the top-layer representation from the previous step is fed back and concatenated with or added to the bottom-layer input, enabling the model to refine its representations iteratively and access global context from previous processing iterations.

Why Feedback Transformers Matter in AI/ML: Feedback transformers address the unidirectional, single-pass limitation of standard transformers by enabling iterative refinement of representations, improving performance on tasks requiring multi-step reasoning or global context integration.

• Top-down feedback — The output of the final transformer layer at step t is fed back to the first layer at step t+1, creating a recurrent loop that allows higher-level abstract representations to influence lower-level processing in subsequent iterations • Memory via recurrence — The feedback connection provides a form of working memory: information processed in earlier iterations persists through the feedback signal, enabling the model to maintain and update state across multiple passes over the input • Iterative refinement — Complex representations benefit from multiple processing passes; feedback transformers naturally implement iterative refinement where each pass through the layer stack improves the representation using context from the previous pass • Attention to past representations — Rather than simple feedback concatenation, some variants allow the first layer to attend over the history of top-layer outputs, creating an attention-based memory of all previous processing iterations • Training with truncated backpropagation — The recurrent nature of feedback transformers requires either full backpropagation through time (expensive) or truncated BPTT for practical training, similar to training strategies for RNNs

Feedback transformers extend the standard transformer architecture with top-down recurrent connections that enable iterative representation refinement and deeper reasoning, addressing the single-pass limitation that constrains standard transformers on tasks requiring multi-step inference and global context integration.