Per-tensor quantization uses a single set of quantization parameters (scale and zero-point) for an entire tensor, regardless of its shape or the variance across its dimensions. This is the simplest and most common quantization granularity.

How It Works

For a tensor $T$ with arbitrary shape:

Where:

• $s$ is the scale factor (computed from the tensor's min/max values).
• $z$ is the zero-point offset (for asymmetric quantization).

Scale Calculation

For 8-bit quantization:

(For symmetric quantization, use $max(|T|)$ instead.)

Advantages

• Simplicity: One scale and zero-point for the entire tensor — minimal storage overhead.
• Fast Inference: Dequantization is straightforward with no per-channel or per-element overhead.
• Hardware Friendly: Most quantization-aware hardware accelerators (TPUs, NPUs) are optimized for per-tensor quantization.

Disadvantages

• Suboptimal for Heterogeneous Data: If different regions of the tensor have very different value ranges, per-tensor quantization wastes precision. For example, if one channel has values in [-0.1, 0.1] and another in [-10, 10], the shared scale must accommodate [-10, 10], losing precision for the first channel.
• Outliers: A single outlier value can dominate the scale calculation, reducing precision for the majority of values.

When to Use Per-Tensor

• Activations: Standard choice for activation quantization because per-channel activations would require runtime overhead.
• Small Tensors: For tensors with relatively uniform value distributions.
• Hardware Constraints: When deploying to hardware that only supports per-tensor quantization.

Comparison to Per-Channel

Example

For a weight tensor with shape [64, 128, 3, 3] (64 output channels):

• Per-Tensor: Compute $min$ and $max$ across all 73,728 values, derive one scale.
• Per-Channel: Compute $min$ and $max$ for each of the 64 output channels separately, derive 64 scales.

Per-tensor quantization is the default choice for activations and a reasonable baseline for weights, though per-channel quantization typically provides better accuracy for weights.