SimAM (Simple Parameter-Free Attention Module) is a 3D attention mechanism that generates weights for each neuron without any learnable parameters — using energy-based neuroscience principles to estimate each neuron's importance based on its distinctiveness from surrounding neurons.

How Does SimAM Work?

• Energy Function: $e_t = frac{1}{M-1}sum_{i=1}^{M-1}(-1-(x_t - x_i)^2)^2 + (1-hat{x}_t)^2$ per neuron.
• Importance: Neurons with lower energy (more distinct from neighbors) get higher attention weights.
• 3D Attention: Produces per-neuron weights across all three dimensions (C, H, W) simultaneously.
• No Parameters: Entirely computed from the feature values — zero learnable parameters.
• Paper: Yang et al. (2021).

Why It Matters

• Parameter-Free: No additional parameters to train — attention is purely computed from input statistics.
• Neuroscience-Inspired: Based on the visual neuroscience concept of neuronal spatial suppression.
• Unified: Simultaneously provides channel and spatial attention in a single mechanism.

SimAM is parameter-free 3D attention — using neuroscience-inspired energy functions to assess each neuron's importance without learning a single extra weight.