self-attention in capsules

**Self-Attention in Capsules** is the **architectural innovation that replaces the original slow iterative routing algorithm in Capsule Networks with the parallelizable self-attention mechanism** — merging the part-whole relationship philosophy of CapsNets with the computational efficiency of Transformers, enabling scalable capsule architectures capable of unsupervised object discovery in natural images. **What Is Self-Attention in Capsules?** - **Background**: Capsule Networks (Hinton et al., 2017) represent entities as vectors (capsules) whose orientation encodes properties and magnitude encodes existence probability — a compelling alternative to CNNs for modeling part-whole hierarchies. - **Routing Problem**: Original Dynamic Routing by Agreement uses iterative expectation-maximization (EM) to decide how lower-level capsules vote for higher-level capsules — sequential, slow, and hard to parallelize. - **Self-Attention Solution**: Replace iterative routing with scaled dot-product attention — lower capsules attend to upper capsules as queries attending to keys, with attention weights determining routing coefficients. - **Stacked Capsule Autoencoders (SCAE)**: The leading architecture combining self-attention and capsules — uses transformer-style attention for unsupervised object part discovery. **Why Self-Attention in Capsules Matters** - **Scalability**: Iterative routing requires sequential loops with 3-5 iterations; self-attention computes routing in one parallelizable matrix operation — 5-10x faster training. - **Gradient Flow**: Self-attention provides clean gradient paths through attention weights; iterative routing has gradient issues from the sequential EM procedure. - **Unsupervised Object Discovery**: Attention-based capsules can segment objects from scenes without supervision — each capsule "attends" to a different object part, learning part decompositions. - **Modularity**: Capsule self-attention is compatible with standard Transformer architectures — CapsNet layers can plug into existing Transformer pipelines. - **Interpretability**: Attention maps show which parts of the input each capsule focuses on — providing visual explanations of the routing decisions. **Routing Algorithms Compared** **Dynamic Routing by Agreement (Sabour 2017)**: - Iterative softmax over coupling coefficients. - 3-5 sequential iterations per forward pass. - Each iteration updates all coupling coefficients globally. - Time complexity: O(iterations × capsules²). **EM Routing (Hinton 2018)**: - Expectation-Maximization over Gaussian capsule poses. - More principled probabilistic interpretation. - Still sequential — 3 EM steps typical. **Self-Attention Routing**: - Compute attention weights in one forward pass: Attention(Q, K, V) = softmax(QK^T / sqrt(d)) V. - Lower capsules = queries; upper capsules = keys and values. - Parallelizable — same complexity as standard attention: O(capsules²) but one pass. - Compatible with multi-head attention for routing diversity. **Stacked Capsule Autoencoder (SCAE) Architecture** **Part Capsule Layer**: - Convolutional features grouped into part capsule templates. - Each template learns a prototype visual part (edges, curves, textures). - Self-attention determines which templates are active. **Object Capsule Layer**: - Part capsules vote for object capsule poses via learned viewpoint transformations. - Self-attention aggregates votes — each object capsule attends to relevant part capsules. - Trained unsupervised via capsule-level reconstruction loss. **Results on MNIST / SVHN**: - Discovers digit parts (strokes) without supervision. - Achieves competitive classification with 1-5 labeled examples per class (few-shot). **Applications** - **Medical Image Segmentation**: Organ capsules attend to anatomical part capsules — interpretable segmentation without pixel-level labels. - **3D Object Recognition**: Point cloud capsules with attention routing — handles occlusion and viewpoint variation. - **Visual Relationship Detection**: Object capsules attend to each other — relation capsules emerge from cross-object attention. **Tools and Implementations** - **SCAE Official**: TensorFlow implementation of Stacked Capsule Autoencoders. - **CapsNet-PyTorch**: Community implementations with attention routing variants. - **Einops**: Tensor manipulation library useful for implementing capsule reshaping operations. Self-Attention in Capsules is **the modernization of structural vision** — combining Hinton's vision of part-whole hierarchical representations with the computational efficiency of Transformers, unlocking scalable capsule networks capable of learning object structure without supervision.