tokenization

Tokenization splits text into tokens (subwords, characters, or words) for model input processing. **Why subword tokenization?**: Balance vocabulary size vs sequence length. Rare words split into known pieces. Handles OOV (out-of-vocabulary) naturally. **Major algorithms**: **BPE (Byte-Pair Encoding)**: Iteratively merge most frequent character pairs, used by GPT models. **WordPiece**: Similar to BPE, used by BERT. **SentencePiece**: Language-agnostic, includes BPE and Unigram variants. **Unigram**: Starts with large vocabulary, prunes based on likelihood. **Key trade-offs**: Smaller vocabulary means more tokens per text means longer sequences. Larger vocabulary means shorter sequences but more embeddings to learn. **Token characteristics**: Average 4 characters/token in English for GPT. Spaces often attached to following token. Non-English may use more tokens. **Impact on cost/limits**: API pricing often per-token, context limits in tokens, tokenization efficiency affects multilingual performance. **Special tokens**: CLS, SEP, PAD, BOS, EOS for model-specific purposes. **Tools**: tiktoken (OpenAI), tokenizers (HuggingFace). Understanding tokenization essential for cost estimation and prompt engineering.