causal mask

**Causal masks** prevent attention to future tokens in autoregressive transformer models, enabling left-to-right generation. **Purpose** - During training on sequences, ensure each position can only see previous positions. - Prevents information leakage from future tokens. **Implementation** - Lower triangular matrix of 1s, upper triangle masked with large negative values. - Position i can attend to positions 0 to i, not i+1 onwards. **Why autoregressive** - Language generation is sequential, each token depends only on previous tokens. - Model must learn to predict without seeing answer. **Training Efficiency** - Train on full sequence in parallel (teacher forcing) while maintaining causal constraint through masking. **Inference** - Not strictly needed (only past tokens exist), but often kept for consistency. **Combined with Padding** - Combine causal mask with padding mask for batched training. **KV Cache** - At inference, causal property enables KV caching since past representations don't change. **Decoder-only Models** - GPT, LLaMA, and most LLMs use causal masking throughout.