synthesizer

**Synthesizer** is a **transformer variant that generates attention weights without computing query-key dot products** — "synthesizing" attention maps directly from the input or from learned parameters, questioning whether explicit pairwise comparisons are necessary. **How Does Synthesizer Work?** - **Dense Synthesizer**: $A = ext{softmax}(f(X))$ where $f$ is a feedforward network. Attention from content, no Q-K dot product. - **Random Synthesizer**: $A = ext{softmax}(R)$ where $R$ is a learnable random matrix. No input dependence at all. - **Mixture**: Combine dense, random, and standard dot-product attention. - **Paper**: Tay et al. (2021). **Why It Matters** - **Provocative**: Random attention (no Q-K interaction!) performs surprisingly well on many benchmarks. - **Insight**: Suggests that the specific pairwise token comparison in standard attention may not always be necessary. - **Efficiency**: Dense/random synthesizers can be faster than full dot-product attention. **Synthesizer** is **the experiment that questioned attention** — showing that attention weights can be generated without even comparing tokens to each other.