byte pair encoding bpe

**Byte Pair Encoding (BPE) and Subword Tokenization** is the **text segmentation technique that breaks input text into a vocabulary of variable-length subword units — learned by iteratively merging the most frequent character pairs in a training corpus — balancing between character-level granularity (handles any text) and word-level efficiency (common words are single tokens), forming the critical preprocessing layer that determines how every LLM perceives and generates language**. **Why Subword Tokenization** Word-level tokenization creates enormous vocabularies (100K+ entries) and cannot handle unseen words (out-of-vocabulary problem). Character-level tokenization handles everything but creates very long sequences (a word like "understanding" becomes 13 tokens), overwhelming the model's context window and attention mechanism. Subword tokenization splits text into meaningful pieces: "understanding" might become ["under", "stand", "ing"] — handling novel compounds while keeping common words as single tokens. **BPE Algorithm** 1. **Initialize**: Start with a vocabulary of all individual bytes (256 entries) or characters. 2. **Count Pairs**: Find the most frequent adjacent pair of tokens in the training corpus. 3. **Merge**: Create a new token by merging this pair. Add it to the vocabulary. 4. **Repeat**: Continue merging until the desired vocabulary size is reached (typically 32K-128K tokens). For example: starting from characters, "th" and "e" merge into "the", "in" and "g" merge into "ing", gradually building up to common words and morphemes. **Tokenizer Variants** - **WordPiece** (BERT): Similar to BPE but selects merges based on likelihood increase of a language model rather than raw frequency. Uses "##" prefix for continuation tokens. - **SentencePiece** (T5, LLaMA): Treats the input as raw bytes/Unicode, handles whitespace as a regular character (using the ▁ prefix), and doesn't require pre-tokenization. Language-agnostic. - **Unigram** (SentencePiece variant): Starts with a large vocabulary and iteratively removes tokens that least decrease the corpus likelihood, instead of building up from characters. - **Tiktoken** (OpenAI/GPT-4): BPE trained on bytes with regex-based pre-tokenization that prevents merges across certain boundaries (numbers, punctuation patterns). **Impact on Model Behavior** - **Fertility**: The number of tokens per word varies by language. English averages ~1.3 tokens/word; morphologically complex languages (Turkish, Finnish) or non-Latin scripts may average 3-5x more, effectively shrinking the usable context window. - **Arithmetic**: Numbers are often split unpredictably ("12345" → ["123", "45"] or ["1", "234", "5"]), contributing to LLMs' difficulty with arithmetic. - **Compression Ratio**: A well-trained tokenizer compresses English text to ~3.5-4 bytes/token. Better compression means more text fits in the context window. Byte Pair Encoding is **the invisible translation layer between human text and neural computation** — the first and last step in every LLM interaction, whose vocabulary choices silently shape what the model can efficiently learn, understand, and express.