bpe (byte-pair encoding)

BPE (Byte-Pair Encoding) is a tokenization algorithm that builds vocabulary by iteratively merging the most frequent character pairs. **Algorithm**: Start with character vocabulary, count all adjacent pair frequencies, merge most frequent pair into new token, repeat until vocabulary size reached. **Example**: The word lowest might tokenize as low + est if those subwords are in vocabulary. **Training**: Run on corpus, learn merge operations, store merge rules for encoding new text. **Inference**: Apply learned merges greedily to tokenize new text. **Advantages**: Handles rare words (split into subwords), no OOV, compact vocabulary, language-agnostic. **Used by**: GPT-2, GPT-3, GPT-4 (with byte-level variant), RoBERTa. **Variants**: Byte-level BPE (operates on bytes, handles any Unicode), BPE with dropout (regularization). **Comparison**: WordPiece uses likelihood-based selection, Unigram uses probabilistic model. **Trade-offs**: Vocabulary size affects sequence length and model size. **Implementation**: tiktoken (OpenAI), tokenizers library (HuggingFace). Foundational algorithm for modern LLM tokenization.