attention bias addition

**Attention bias addition** is the **injection of structured bias terms into attention logits to encode positional or task priors before softmax** - it influences which token relationships are favored without changing core attention mechanics. **What Is Attention bias addition?** - **Definition**: Adding learned or fixed bias values to QK score matrices prior to normalization. - **Common Forms**: Relative position bias, ALiBi slopes, segment bias, and task-specific masking bias. - **Placement**: Applied after raw score computation and before softmax scaling or normalization. - **Kernel Concern**: Efficient implementations fuse bias injection with score computation. **Why Attention bias addition Matters** - **Model Expressiveness**: Encodes inductive structure that helps learning sequence relationships. - **Long-Range Behavior**: Relative biases improve extrapolation for longer contexts in many settings. - **Task Adaptation**: Domain-specific bias terms can improve performance for structured inputs. - **Runtime Cost**: Naive bias handling can create extra memory movement and kernel launches. - **Optimization Opportunity**: In-kernel bias addition preserves speed while retaining modeling benefits. **How It Is Used in Practice** - **Bias Strategy**: Choose fixed versus learned bias based on architecture and generalization goals. - **Fused Execution**: Integrate bias math into fused attention kernels to minimize overhead. - **Ablation Testing**: Measure quality gain and latency impact across sequence lengths. Attention bias addition is **a powerful control point in attention design** - when implemented efficiently, it adds structural priors with minimal performance penalty.