groupnorm

**GroupNorm** is a **normalization technique that divides channels into groups and normalizes within each group** — independent of batch size, making it the preferred normalization for tasks with small batch sizes (detection, segmentation, video). **How Does GroupNorm Work?** - **Groups**: Divide $C$ channels into $G$ groups of $C/G$ channels each (typically $G = 32$). - **Normalize**: Compute mean and variance within each group (across spatial + channels-in-group dimensions). - **Affine**: Apply learnable scale and shift per channel. - **Paper**: Wu & He (2018). **Why It Matters** - **Batch-Independent**: Unlike BatchNorm, GroupNorm's statistics don't depend on batch size. Works with batch size 1. - **Detection/Segmentation**: Standard in Mask R-CNN, DETR, and other detection frameworks where batch sizes are tiny (1-4). - **Special Cases**: GroupNorm with $G = C$ is InstanceNorm. GroupNorm with $G = 1$ is LayerNorm. **GroupNorm** is **normalization for small batches** — computing statistics within channel groups instead of across the batch for batch-size-independent training.