set transformer

**Set Transformer** is a **transformer architecture designed for set-structured inputs (unordered collections)** — using attention-based mechanisms to process variable-size sets while maintaining permutation invariance, the key symmetry property of set functions. **How Does Set Transformer Work?** - **SAB** (Set Attention Block): Standard multi-head self-attention applied to set elements. - **ISAB** (Induced Set Attention Block): Uses $m$ inducing points to reduce $O(N^2)$ to $O(N cdot m)$ complexity. - **PMA** (Pooling by Multihead Attention): Aggregates set elements into $k$ output vectors using learned seed vectors. - **Paper**: Lee et al. (2019). **Why It Matters** - **Permutation Invariance**: The output is the same regardless of the order of input elements — essential for set functions. - **Efficient**: ISAB enables processing large sets (thousands of elements) efficiently. - **Applications**: Point cloud processing, amortized inference, few-shot learning, set prediction. **Set Transformer** is **attention for unordered collections** — processing variable-size sets with permutation invariance and efficient inducing-point attention.