Per-channel quantization applies different quantization parameters (scale and zero-point) to each output channel (filter) in a convolutional or linear layer, rather than using a single set of parameters for the entire tensor.

How It Works

• Per-Tensor: One scale $s$ and zero-point $z$ for the entire weight tensor. All channels share the same quantization range.
• Per-Channel: Each output channel $c$ has its own scale $s_c$ and zero-point $z_c$. Channels with larger weight magnitudes get larger scales.

Formula

For a weight tensor $W$ with shape [out_channels, in_channels, height, width]:

Where $c$ is the output channel index.

Why Per-Channel Matters

• Channel Variance: Different filters in a layer often have very different weight magnitude distributions. Some channels may have weights in [-0.1, 0.1], others in [-2.0, 2.0].
• Better Utilization: Per-channel quantization allows each channel to use the full quantization range optimally, reducing quantization error.
• Accuracy Improvement: Typically provides 1-3% accuracy improvement over per-tensor quantization with minimal overhead.

Trade-offs

• Storage: Requires storing one scale (and optionally zero-point) per output channel. For a layer with 256 channels, this adds 256 floats (~1KB) — negligible compared to the weight tensor itself.
• Computation: Slightly more complex dequantization (each channel uses its own scale), but modern hardware handles this efficiently.
• Compatibility: Widely supported in quantization frameworks (TensorFlow Lite, PyTorch, ONNX Runtime).

Example

Consider a Conv2D layer with 64 output channels:

• Per-Tensor: All 64 channels share one scale. If channel 0 has weights in [-0.05, 0.05] and channel 63 has weights in [-1.5, 1.5], the shared scale must accommodate [-1.5, 1.5], wasting precision for channel 0.
• Per-Channel: Channel 0 gets scale $s_0 = 0.05/127$, channel 63 gets scale $s_{63} = 1.5/127$. Both channels use their quantization range optimally.

Standard Practice

• Weights: Almost always use per-channel quantization (standard in TensorFlow Lite, PyTorch).
• Activations: Typically use per-tensor quantization (per-channel activations are less common due to runtime overhead).

Per-channel quantization is a best practice for weight quantization, providing significant accuracy benefits with minimal cost.