capsule network dynamic routing

**Capsule Networks** is the **alternative neural architecture where capsules (groups of neurons encoding pose and viewpoint parameters) are routed through dynamic agreement — capturing part-whole hierarchies and equivariance to transformations better than traditional convolutional networks**. **Capsule Entity Representation:** - Capsule abstraction: group of neurons (4-8 typically) represents entity at specific position/scale; vector encodes pose information - Pose vector: contains position, size, orientation, and other transformation parameters for detected feature - Equivariance property: when input transforms (rotation, translation), pose vectors transform correspondingly; not true for standard neurons - Routing responsibility: capsule outputs routed to higher-level capsules based on agreement; mechanism for part-whole relationships **Dynamic Routing by Agreement:** - Routing algorithm: iterative procedure routes lower-level capsule outputs to higher-level capsules based on prediction agreement - Coupling coefficients: learned soft weights determining capsule routing; updated each iteration based on agreement metrics - Routing iterations: typically 2-3 iterations; each iteration refines coupling coefficients to route to agreeing capsules - Squashing activation: output capsule activations squeezed to unit norm via non-linear squashing function - Prediction agreement: if lower-capsule predicts upper-capsule's activity, coupling strength increases (routing to agreeing capsules) **EM Routing (Hinton 2018):** - Expectation-Maximization routing: alternative to dynamic routing; more principled probabilistic approach - Gaussian modeling: model capsule outputs as mixture of Gaussians; EM algorithm learns mixture weights and parameters - Linear transformation: pose predictions from lower to higher capsules via learned transformation matrices - Iterative EM: alternating expectation (assign capsules to clusters) and maximization (update cluster parameters) - Improved performance: EM routing slightly improves accuracy; computational cost vs marginal gain tradeoff **Part-Whole Relationships:** - Hierarchical structure: capsules explicitly encode part-whole relationships; lower-level features → higher-level entities - Compositional learning: model learns that wheels, doors, windows compose cars; explicit semantic hierarchy - Robustness to viewpoint: capsule vectors contain viewpoint information; networks generalize across viewpoints - Inverse graphics: capsules hypothesized to learn inverse graphics model (generate images from poses) **Equivariance to Transformations:** - Equivariance advantage: standard CNNs have limited equivariance (only translation for convolution); capsules equivariant to more transforms - Pose generalization: viewpoint transformation in input reflected in pose vector; enables better generalization - Affine transformations: capsule networks hypothesized to be equivariant to affine transforms; supported empirically - Robustness benefits: equivariance hypothesized to improve adversarial robustness; empirical validation ongoing **Capsule Network Architecture:** - CapsNet for MNIST: two convolutional capsule layers + fully-connected capsule layer; margin loss for multiclass classification - Instance parameters: each capsule type shares weights across spatial positions; reduces parameters vs fully-connected networks - Reconstruction regularizer: add reconstruction loss (decoder reconstructs image from class capsule); additional supervision signal **Limitations and Challenges:** - Scalability: routing complexity increases with network depth; computational overhead substantial - Training difficulty: capsule networks harder to train than CNNs; require careful initialization and hyperparameter tuning - Performance gains: improvements over CNNs modest on standard benchmarks; larger benefits hypothesized for novel viewpoints - Interpretability: capsule poses should be interpretable (rotations, positions, etc.); empirical pose interpretability mixed **Capsule networks introduce geometric structure — routing by agreement and pose vectors encoding transformations — proposing a more biologically inspired alternative to standard convolutions with advantages in capturing part-whole hierarchies.**