tokenizer bpe
**Byte-Pair Encoding (BPE)** is a **subword tokenization algorithm that iteratively merges the most frequent character pairs** — producing a vocabulary of subword units that balances vocabulary size with sequence length and handles unknown words gracefully.
**Why Tokenization Matters**
- LLMs process tokens, not characters or words.
- Word-level vocabulary: 500K+ words, fails on unseen words.
- Character-level: Very long sequences, slow training.
- Subword (BPE): Best of both — compact vocabulary, handles rare words.
**BPE Algorithm**
1. Initialize vocabulary with individual characters.
2. Count frequency of all adjacent byte/character pairs.
3. Merge the most frequent pair → new token.
4. Repeat until vocabulary size V is reached (typically 32K–100K).
**Example**:
- "l o w", "l o w e r", "n e w" → merge most frequent "ow" → "low", "lower", "new"
- Result: common words become single tokens; rare words split into subwords.
**Tokenizer Variants**
- **BPE (GPT-2, GPT-3, LLaMA)**: Operates on bytes, handles any Unicode.
- **WordPiece (BERT)**: Like BPE but maximizes likelihood of training data instead of frequency.
- **SentencePiece (LLaMA, T5)**: Language-independent, treats whitespace as a token.
- **Unigram (ALBERT)**: Probabilistic subword model — prunes tokens that minimize overall likelihood.
**Tokenization Impact on Models**
- Number of tokens per word varies by language — English ~1.3 tokens/word, Chinese ~2-3 tokens/word.
- Code tokenizers often use code-specific BPE (dedented whitespace, common identifiers).
- Tokenization artifacts can cause reasoning errors (e.g., counting letters in words).
**Vocabulary Sizes**
| Model | Vocabulary | Tokenizer |
|-------|-----------|----------|
| GPT-2 | 50,257 | BPE |
| GPT-4 | 100,277 | tiktoken BPE |
| LLaMA | 32,000 | SentencePiece |
| BERT | 30,522 | WordPiece |
Tokenization is **a foundational but often overlooked design decision** — vocabulary size, granularity, and algorithm directly affect training efficiency, multilingual performance, and arithmetic reasoning.