attention mask

Attention masks indicate which tokens the model should attend to versus ignore during self-attention computation. **Purpose**: Prevent attention to padding tokens, mask future tokens in causal models, handle variable-length sequences in batches. **Format**: Binary tensor same shape as input, 1 = attend, 0 = ignore. Applied as additive mask (large negative value) to attention scores before softmax. **Padding mask**: Mask out PAD tokens so they dont influence representations. Essential for batched inference with different sequence lengths. **For training**: Prevents padding from affecting gradients, ensures loss computed only on real tokens. **Creation**: Usually automatic from tokenizer when padding. Can be manually constructed for custom masking. **Multi-head attention**: Same mask typically applied across all attention heads. **Cross-attention**: May have different masks for encoder and decoder sequences. **Debugging**: Incorrect attention masks cause subtle bugs, degraded performance, or training instability. Always verify mask shapes and values.