Byte Pair Encoding (BPE) is a tokenization algorithm that iteratively merges the most frequent adjacent character/token pairs to create a compact vocabulary of subword units — reducing vocabulary size from 130K+ raw characters to 50K tokens while maintaining 99.8% coverage of natural language.

Algorithm and Mechanism:

• Iterative Merging: starting with character-level tokens, algorithm identifies most frequent pair and merges all occurrences (e.g., "t" + "h" → "th") — repeats 10,000-50,000 iterations building 50K vocabulary
• Frequency Counting: corpus-level frequency analysis using hash tables with O(n) complexity per iteration on modern GPUs — GPT-3 training analyzed 300B tokens to derive final BPE table
• Encoding Process: greedy left-to-right matching using learned merge rules applied in order — converts "butterfly" to ["but", "ter", "fly"] rather than 9 characters
• Decode Compatibility: reversible process where adding special markers () preserves word boundaries without ambiguity

Technical Advantages:

• Vocabulary Efficiency: reduces embedding matrix size from 130K×768 (100M params) to 50K×768 (38M params) — 62% reduction saves memory in transformer models
• Rare Word Handling: unknown words decomposed to subwords with embeddings (e.g., "polymorphism" split as ["poly", "morph", "ism"]) — handles 99.97% of English correctly
• Compression Ratio: average 1.3 tokens per word in English vs 1.8 with WordPiece and 2.1 with character-level — saves 30-40% in sequence length
• Cross-Lingual: single BPE vocabulary handles 100+ languages by pre-training on multilingual corpus — achieves uniform compression across scripts

Implementation Details:

• FastBPE: C++ implementation processes 1B tokens in <1 minute on single CPU core — open-source used by Meta's XLM model
• Sentencepiece: Google framework supporting BPE, Unigram, and Char tokenization with lossless reversibility — standard for BERT, mT5, and multilingual models
• Hugging Face Tokenizers: Rust-based library with 50,000 tokens/sec throughput — powers all models on Hugging Face Hub
• Training Stability: deterministic algorithm with fixed random seed enables reproducible vocabulary across runs

Byte Pair Encoding is the dominant tokenization standard for transformer models — enabling efficient representation of natural language while maintaining semantic meaning and cross-lingual generalization.