per-channel quantization

**Per-channel quantization** applies **different quantization parameters** (scale and zero-point) to each output channel (filter) in a convolutional or linear layer, rather than using a single set of parameters for the entire tensor. **How It Works** - **Per-Tensor**: One scale $s$ and zero-point $z$ for the entire weight tensor. All channels share the same quantization range. - **Per-Channel**: Each output channel $c$ has its own scale $s_c$ and zero-point $z_c$. Channels with larger weight magnitudes get larger scales. **Formula** For a weight tensor $W$ with shape [out_channels, in_channels, height, width]: $$q_{c,i,h,w} = ext{round}(W_{c,i,h,w} / s_c + z_c)$$ Where $c$ is the output channel index. **Why Per-Channel Matters** - **Channel Variance**: Different filters in a layer often have very different weight magnitude distributions. Some channels may have weights in [-0.1, 0.1], others in [-2.0, 2.0]. - **Better Utilization**: Per-channel quantization allows each channel to use the full quantization range optimally, reducing quantization error. - **Accuracy Improvement**: Typically provides 1-3% accuracy improvement over per-tensor quantization with minimal overhead. **Trade-offs** - **Storage**: Requires storing one scale (and optionally zero-point) per output channel. For a layer with 256 channels, this adds 256 floats (~1KB) — negligible compared to the weight tensor itself. - **Computation**: Slightly more complex dequantization (each channel uses its own scale), but modern hardware handles this efficiently. - **Compatibility**: Widely supported in quantization frameworks (TensorFlow Lite, PyTorch, ONNX Runtime). **Example** Consider a Conv2D layer with 64 output channels: - **Per-Tensor**: All 64 channels share one scale. If channel 0 has weights in [-0.05, 0.05] and channel 63 has weights in [-1.5, 1.5], the shared scale must accommodate [-1.5, 1.5], wasting precision for channel 0. - **Per-Channel**: Channel 0 gets scale $s_0 = 0.05/127$, channel 63 gets scale $s_{63} = 1.5/127$. Both channels use their quantization range optimally. **Standard Practice** - **Weights**: Almost always use per-channel quantization (standard in TensorFlow Lite, PyTorch). - **Activations**: Typically use per-tensor quantization (per-channel activations are less common due to runtime overhead). Per-channel quantization is a **best practice** for weight quantization, providing significant accuracy benefits with minimal cost.