symmetric vs asymmetric quantization

**Symmetric vs. Asymmetric Quantization** refers to how the quantization range is mapped to the original floating-point value range, specifically whether the zero point is fixed or learned. **Symmetric Quantization** - **Zero-Point Fixed**: The quantized zero is mapped to the floating-point zero. The quantization range is **symmetric** around zero. - **Formula**: $q = ext{round}(x / s)$ where $s$ is the scale factor. - **Range**: For 8-bit signed integers, the range is [-127, 127], with 0 mapping to 0. - **Advantages**: Simpler implementation, faster inference (no zero-point offset calculation), better for hardware acceleration. - **Disadvantages**: Wastes one quantization level if the data distribution is asymmetric (e.g., ReLU activations are always non-negative). **Asymmetric Quantization** - **Zero-Point Learned**: The quantized zero can map to any floating-point value. The quantization range is **asymmetric**. - **Formula**: $q = ext{round}(x / s + z)$ where $s$ is scale and $z$ is the zero-point offset. - **Range**: For 8-bit unsigned integers, the range is [0, 255], with the zero-point $z$ learned to minimize quantization error. - **Advantages**: Better utilizes the quantization range for asymmetric distributions (e.g., post-ReLU activations), lower quantization error. - **Disadvantages**: Slightly more complex, requires storing and applying the zero-point offset. **When to Use Each** - **Symmetric**: Weights (typically centered around zero), when hardware acceleration is critical, when simplicity matters. - **Asymmetric**: Activations (especially after ReLU, which are non-negative), when minimizing quantization error is the priority. **Example** Consider values in range [0.5, 3.5]: - **Symmetric**: Maps [-3.5, 3.5] to [-127, 127], wasting half the range on negative values that don't exist. - **Asymmetric**: Maps [0.5, 3.5] to [0, 255], using the full quantization range efficiently. **Practical Impact** Most modern quantization frameworks (TensorFlow Lite, PyTorch) use: - **Symmetric quantization for weights** (simpler, hardware-friendly). - **Asymmetric quantization for activations** (better accuracy for ReLU outputs). The choice between symmetric and asymmetric quantization is a fundamental design decision that impacts both model accuracy and inference efficiency.