filter response normalization

**FRN** (Filter Response Normalization) is a **normalization technique designed to work without batch or group dependencies** — normalizing each filter response individually and using a learnable thresholded linear unit (TLU) as the activation function. **How Does FRN Work?** - **Normalize**: $hat{x}_c = x_c / sqrt{frac{1}{HW}sum_{h,w} x_{c,h,w}^2 + epsilon}$ (divide by RMS of spatial dimensions for each channel). - **TLU Activation**: $y = max(x, au)$ where $ au$ is a learnable threshold (replaces ReLU). - **No Mean Subtraction**: Like RMSNorm, FRN skips mean centering. - **Paper**: Singh & Krishnan (2020). **Why It Matters** - **Batch-Free**: Works with batch size 1, unlike BatchNorm. - **SOTA**: Achieved competitive results with BatchNorm across various CNN architectures. - **TLU**: The learnable threshold activation is key — standard ReLU doesn't work well with FRN. **FRN** is **self-sufficient normalization** — each filter channel normalizes itself independently, with a learnable activation threshold for optimal performance.