Self-Attention in Capsules is the architectural innovation that replaces the original slow iterative routing algorithm in Capsule Networks with the parallelizable self-attention mechanism — merging the part-whole relationship philosophy of CapsNets with the computational efficiency of Transformers, enabling scalable capsule architectures capable of unsupervised object discovery in natural images.

What Is Self-Attention in Capsules?

• Background: Capsule Networks (Hinton et al., 2017) represent entities as vectors (capsules) whose orientation encodes properties and magnitude encodes existence probability — a compelling alternative to CNNs for modeling part-whole hierarchies.
• Routing Problem: Original Dynamic Routing by Agreement uses iterative expectation-maximization (EM) to decide how lower-level capsules vote for higher-level capsules — sequential, slow, and hard to parallelize.
• Self-Attention Solution: Replace iterative routing with scaled dot-product attention — lower capsules attend to upper capsules as queries attending to keys, with attention weights determining routing coefficients.
• Stacked Capsule Autoencoders (SCAE): The leading architecture combining self-attention and capsules — uses transformer-style attention for unsupervised object part discovery.

Why Self-Attention in Capsules Matters

• Scalability: Iterative routing requires sequential loops with 3-5 iterations; self-attention computes routing in one parallelizable matrix operation — 5-10x faster training.
• Gradient Flow: Self-attention provides clean gradient paths through attention weights; iterative routing has gradient issues from the sequential EM procedure.
• Unsupervised Object Discovery: Attention-based capsules can segment objects from scenes without supervision — each capsule "attends" to a different object part, learning part decompositions.
• Modularity: Capsule self-attention is compatible with standard Transformer architectures — CapsNet layers can plug into existing Transformer pipelines.
• Interpretability: Attention maps show which parts of the input each capsule focuses on — providing visual explanations of the routing decisions.

Routing Algorithms Compared

Dynamic Routing by Agreement (Sabour 2017):

• Iterative softmax over coupling coefficients.
• 3-5 sequential iterations per forward pass.
• Each iteration updates all coupling coefficients globally.
• Time complexity: O(iterations × capsules²).

EM Routing (Hinton 2018):

• Expectation-Maximization over Gaussian capsule poses.
• More principled probabilistic interpretation.
• Still sequential — 3 EM steps typical.

Self-Attention Routing:

• Compute attention weights in one forward pass: Attention(Q, K, V) = softmax(QK^T / sqrt(d)) V.
• Lower capsules = queries; upper capsules = keys and values.
• Parallelizable — same complexity as standard attention: O(capsules²) but one pass.
• Compatible with multi-head attention for routing diversity.

Stacked Capsule Autoencoder (SCAE) Architecture

Part Capsule Layer:

• Convolutional features grouped into part capsule templates.
• Each template learns a prototype visual part (edges, curves, textures).
• Self-attention determines which templates are active.

Object Capsule Layer:

• Part capsules vote for object capsule poses via learned viewpoint transformations.
• Self-attention aggregates votes — each object capsule attends to relevant part capsules.
• Trained unsupervised via capsule-level reconstruction loss.

Results on MNIST / SVHN:

• Discovers digit parts (strokes) without supervision.
• Achieves competitive classification with 1-5 labeled examples per class (few-shot).

Applications

• Medical Image Segmentation: Organ capsules attend to anatomical part capsules — interpretable segmentation without pixel-level labels.
• 3D Object Recognition: Point cloud capsules with attention routing — handles occlusion and viewpoint variation.
• Visual Relationship Detection: Object capsules attend to each other — relation capsules emerge from cross-object attention.

Tools and Implementations

• SCAE Official: TensorFlow implementation of Stacked Capsule Autoencoders.
• CapsNet-PyTorch: Community implementations with attention routing variants.
• Einops: Tensor manipulation library useful for implementing capsule reshaping operations.

Self-Attention in Capsules is the modernization of structural vision — combining Hinton's vision of part-whole hierarchical representations with the computational efficiency of Transformers, unlocking scalable capsule networks capable of learning object structure without supervision.