random synthesizer

**Random Synthesizer** is a **variant of the Synthesizer model where attention weights are drawn from a learned matrix that is entirely independent of the input** — each position in the attention map is a fixed, learned parameter, not computed from query-key interactions. **How Does Random Synthesizer Work?** - **Attention Matrix**: $A = ext{softmax}(R)$ where $R in mathbb{R}^{N imes N}$ is a learnable parameter matrix. - **No Input Dependence**: The attention pattern is the same regardless of the input sequence. - **Training**: $R$ is optimized via backpropagation alongside all other parameters. - **Fixed at Inference**: Once trained, the attention pattern is static. **Why It Matters** - **Surprising Result**: Achieves 90%+ of dot-product attention performance on many tasks despite being input-independent. - **Implications**: Suggests that transformers may partially rely on learned positional routing rather than semantic matching. - **Research Insight**: Challenges the assumption that dynamic, content-based attention is essential. **Random Synthesizer** is **the attention paradox** — showing that even fixed, input-independent attention patterns can capture surprisingly useful information.