ISAB (Induced Set Attention Block) is a memory-efficient attention block that uses a small set of learnable inducing points to compress the $O(N^2)$ self-attention — tokens first attend to the inducing points (forming a bottleneck), then the inducing points attend back to the tokens.

How Does ISAB Work?

• Inducing Points: $I in mathbb{R}^{m imes d}$ — a set of $m$ learnable vectors ($m ll N$).
• Step 1: $H = ext{MAB}(I, X)$ — inducing points attend to input tokens. $H in mathbb{R}^{m imes d}$.
• Step 2: $ ext{ISAB}(X) = ext{MAB}(X, H)$ — input tokens attend to the compressed inducing points.
• Complexity: $O(N cdot m)$ instead of $O(N^2)$.

Why It Matters

• Bottleneck Attention: The $m$ inducing points act as a compressed representation of the entire set.
• Scalable: With $m = 32-128$, can process sets of thousands of elements efficiently.
• Perceivers: The same principle was later adopted by Perceiver and Perceiver IO for general-purpose architectures.

ISAB is attention through a bottleneck — using a small set of learned summary points to avoid the quadratic cost of full self-attention.