FRN (Filter Response Normalization) is a normalization technique designed to work without batch or group dependencies — normalizing each filter response individually and using a learnable thresholded linear unit (TLU) as the activation function.

How Does FRN Work?

• Normalize: $hat{x}_c = x_c / sqrt{frac{1}{HW}sum_{h,w} x_{c,h,w}^2 + epsilon}$ (divide by RMS of spatial dimensions for each channel).
• TLU Activation: $y = max(x, au)$ where $ au$ is a learnable threshold (replaces ReLU).
• No Mean Subtraction: Like RMSNorm, FRN skips mean centering.
• Paper: Singh & Krishnan (2020).

Why It Matters

• Batch-Free: Works with batch size 1, unlike BatchNorm.
• SOTA: Achieved competitive results with BatchNorm across various CNN architectures.
• TLU: The learnable threshold activation is key — standard ReLU doesn't work well with FRN.

FRN is self-sufficient normalization — each filter channel normalizes itself independently, with a learnable activation threshold for optimal performance.