induced set attention block

**ISAB** (Induced Set Attention Block) is a **memory-efficient attention block that uses a small set of learnable inducing points to compress the $O(N^2)$ self-attention** — tokens first attend to the inducing points (forming a bottleneck), then the inducing points attend back to the tokens. **How Does ISAB Work?** - **Inducing Points**: $I in mathbb{R}^{m imes d}$ — a set of $m$ learnable vectors ($m ll N$). - **Step 1**: $H = ext{MAB}(I, X)$ — inducing points attend to input tokens. $H in mathbb{R}^{m imes d}$. - **Step 2**: $ ext{ISAB}(X) = ext{MAB}(X, H)$ — input tokens attend to the compressed inducing points. - **Complexity**: $O(N cdot m)$ instead of $O(N^2)$. **Why It Matters** - **Bottleneck Attention**: The $m$ inducing points act as a compressed representation of the entire set. - **Scalable**: With $m = 32-128$, can process sets of thousands of elements efficiently. - **Perceivers**: The same principle was later adopted by Perceiver and Perceiver IO for general-purpose architectures. **ISAB** is **attention through a bottleneck** — using a small set of learned summary points to avoid the quadratic cost of full self-attention.