Causal masks prevent attention to future tokens in autoregressive transformer models, enabling left-to-right generation.

Purpose

• During training on sequences, ensure each position can only see previous positions.
• Prevents information leakage from future tokens.

Implementation

• Lower triangular matrix of 1s, upper triangle masked with large negative values.
• Position i can attend to positions 0 to i, not i+1 onwards.

Why autoregressive

• Language generation is sequential, each token depends only on previous tokens.
• Model must learn to predict without seeing answer.

Training Efficiency

• Train on full sequence in parallel (teacher forcing) while maintaining causal constraint through masking.

Inference

• Not strictly needed (only past tokens exist), but often kept for consistency.

Combined with Padding

• Combine causal mask with padding mask for batched training.

KV Cache

• At inference, causal property enables KV caching since past representations don't change.

Decoder-only Models

• GPT, LLaMA, and most LLMs use causal masking throughout.