padding token

Padding tokens fill sequences to uniform length for efficient batched processing. **Why needed**: Batched computation requires same sequence length, real sequences vary in length, padding fills the gap. **Padding strategy**: **Right padding**: Add PAD tokens at end (common for causal/decoder models). **Left padding**: Add PAD tokens at start (sometimes used for generation so outputs align). **Attention mask**: Critical companion to padding, tells model to ignore PAD tokens. Without mask, model would attend to meaningless PAD tokens. **Token ID**: Often 0, but varies by tokenizer. Should never contribute to loss or attention. **Loss masking**: Training loss excludes PAD positions, only compute loss on real tokens. **Efficiency concern**: Long padding wastes computation. Solutions include dynamic batching (group similar lengths), sequence packing. **Memory**: Padding inflates batch memory usage. Maximum sequence length should match data needs. **Implementation**: Tokenizer handles padding with padding=True, pad_to_max_length parameters. Always pair with attention_mask.