byte pair encoding
**Byte Pair Encoding (BPE)** is **a tokenization algorithm that iteratively merges the most frequent adjacent character/token pairs to create a compact vocabulary of subword units — reducing vocabulary size from 130K+ raw characters to 50K tokens while maintaining 99.8% coverage of natural language**.
**Algorithm and Mechanism:**
- **Iterative Merging**: starting with character-level tokens, algorithm identifies most frequent pair and merges all occurrences (e.g., "t" + "h" → "th") — repeats 10,000-50,000 iterations building 50K vocabulary
- **Frequency Counting**: corpus-level frequency analysis using hash tables with O(n) complexity per iteration on modern GPUs — GPT-3 training analyzed 300B tokens to derive final BPE table
- **Encoding Process**: greedy left-to-right matching using learned merge rules applied in order — converts "butterfly" to ["but", "ter", "fly"] rather than 9 characters
- **Decode Compatibility**: reversible process where adding special markers () preserves word boundaries without ambiguity
**Technical Advantages:**
- **Vocabulary Efficiency**: reduces embedding matrix size from 130K×768 (100M params) to 50K×768 (38M params) — 62% reduction saves memory in transformer models
- **Rare Word Handling**: unknown words decomposed to subwords with embeddings (e.g., "polymorphism" split as ["poly", "morph", "ism"]) — handles 99.97% of English correctly
- **Compression Ratio**: average 1.3 tokens per word in English vs 1.8 with WordPiece and 2.1 with character-level — saves 30-40% in sequence length
- **Cross-Lingual**: single BPE vocabulary handles 100+ languages by pre-training on multilingual corpus — achieves uniform compression across scripts
**Implementation Details:**
- **FastBPE**: C++ implementation processes 1B tokens in <1 minute on single CPU core — open-source used by Meta's XLM model
- **Sentencepiece**: Google framework supporting BPE, Unigram, and Char tokenization with lossless reversibility — standard for BERT, mT5, and multilingual models
- **Hugging Face Tokenizers**: Rust-based library with 50,000 tokens/sec throughput — powers all models on Hugging Face Hub
- **Training Stability**: deterministic algorithm with fixed random seed enables reproducible vocabulary across runs
**Byte Pair Encoding is the dominant tokenization standard for transformer models — enabling efficient representation of natural language while maintaining semantic meaning and cross-lingual generalization.**