special tokens

Special tokens are tokens with specific purposes in model architecture, like sequence boundaries and masking. **Common special tokens**: **BOS/SOS**: Beginning of sequence, signals start. **EOS**: End of sequence, signals completion. **PAD**: Padding for batch uniformity. **MASK**: Masked token for MLM training (BERT). **SEP**: Separator between segments. **CLS**: Classification token (BERT). **UNK**: Unknown token for OOV (legacy). **Model-specific examples**: BERT uses CLS, SEP, MASK, PAD. GPT uses end-of-text token. LLaMA uses bos and eos tokens. **Chat tokens**: System, user, assistant role markers for instruction-tuned models. **Why they matter**: Enable model to understand structure, separate inputs in multi-turn chat, know when to stop generating. **Token IDs**: Usually assigned first IDs in vocabulary (0, 1, 2...). **Training**: Model learns behavior for each special token through training data patterns. **Prompt engineering**: Understanding special tokens helps craft effective prompts, especially for chat models.