dense synthesizer

**Dense Synthesizer** is a **variant of the Synthesizer model where attention weights are generated by a feedforward network applied to each token independently** — replacing the pairwise query-key dot product with a per-token MLP that directly predicts attention over all positions. **How Does Dense Synthesizer Work?** - **Per-Token**: For each token $x_i$, compute $a_i = W_2 cdot ext{ReLU}(W_1 cdot x_i)$ producing a vector of length $N$. - **Attention**: $A = ext{softmax}([a_1; a_2; ...; a_N])$ (each row from one token's MLP output). - **No Key Interaction**: Token $i$'s attention weights are computed without looking at any other token. - **Value Aggregation**: Standard weighted sum of values using the synthesized attention. **Why It Matters** - **Content-Dependent but Not Pairwise**: Attention depends on the query token's content but not on explicit key comparison. - **Competitive**: Matches or approaches standard attention on sequence-to-sequence and classification tasks. - **Hybrid**: Can be combined with standard dot-product attention for best results. **Dense Synthesizer** is **attention from a single perspective** — each token decides its attention pattern based solely on its own content, without consulting keys.