qkv bias
**QKV Bias** refers to the **learnable bias vectors ($b_q$, $b_k$, $b_v$) optionally added to the linear projection matrices within the Query, Key, and Value computation layers of a Transformer's Multi-Head Self-Attention mechanism — providing the critical mathematical degree of freedom that allows each attention subspace to shift its origin away from zero.**
**The Core Mathematics**
- **Without Bias**: The standard linear projection is $Q = XW_q$, $K = XW_k$, $V = XW_v$. This means if the input embedding $X$ is exactly zero (or near zero), the Query, Key, and Value vectors are also forced to be exactly zero. The mathematical origin $(0, 0, ..., 0)$ is permanently locked in place.
- **With Bias**: The projection becomes $Q = XW_q + b_q$, $K = XW_k + b_k$, $V = XW_v + b_v$. The learnable bias vector $b_q$ allows the model to shift the entire Query hyperplane to any arbitrary position in the high-dimensional attention subspace.
**Why Bias Matters for Vision Transformers**
- **The ViT Sensitivity**: Empirical studies (particularly DeiT and BEiT) demonstrated that removing QKV bias from Vision Transformers causes a measurable and consistent accuracy degradation (typically $0.3\%$ to $0.5\%$ top-1 on ImageNet).
- **The Hypothesis**: Unlike large language models (which sometimes drop bias without penalty due to their massive token diversity), ViTs process a relatively homogeneous set of image patch embeddings. The bias vectors provide essential flexibility for attention heads to specialize — one head can focus on texture by shifting its Query subspace toward high-frequency features, while another head shifts toward low-frequency color gradients, even when the raw patch embeddings are numerically similar.
- **The Exception (Modern LLMs)**: Interestingly, several modern large language models (LLaMA, PaLM) deliberately remove QKV bias to reduce parameter count and simplify quantization, relying on the sheer diversity and scale of their text token distributions to compensate for the lost flexibility.
**QKV Bias** is **the intercept of attention** — a simple but critical learnable offset that grants each attention head the mathematical freedom to position its sensory receptive field anywhere in the abstract feature space, rather than being permanently anchored to the origin.