tokenization algorithms

**Tokenization Algorithms and Vocabulary Design** — Tokenization transforms raw text into discrete units that neural networks can process, fundamentally shaping model capacity and linguistic understanding. **Core Tokenization Approaches** — Character-level tokenization splits text into individual characters, yielding small vocabularies but long sequences. Word-level tokenization uses whitespace and punctuation boundaries, creating large vocabularies with out-of-vocabulary problems. Subword tokenization balances these extremes by breaking words into meaningful fragments that capture morphological patterns while maintaining manageable vocabulary sizes. **Byte Pair Encoding (BPE)** — BPE iteratively merges the most frequent adjacent token pairs in a training corpus. Starting from individual characters, the algorithm builds a merge table that defines the vocabulary. GPT-2 and GPT-3 use byte-level BPE, operating on UTF-8 bytes rather than Unicode characters, ensuring complete coverage of any input text. The merge operations create tokens that often correspond to common syllables, prefixes, and suffixes, enabling efficient representation of diverse languages. **WordPiece and Unigram Models** — WordPiece, used by BERT, selects merges that maximize likelihood of the training data rather than simple frequency. The Unigram model from SentencePiece takes the opposite approach — starting with a large vocabulary and iteratively removing tokens whose loss has minimal impact on corpus likelihood. SentencePiece treats the input as a raw byte stream, eliminating the need for language-specific pre-tokenization rules and enabling truly multilingual tokenization. **Vocabulary Design Considerations** — Vocabulary size directly impacts embedding table memory and softmax computation costs. Typical sizes range from 32,000 to 256,000 tokens. Larger vocabularies reduce sequence lengths but increase parameter counts. Domain-specific tokenizers trained on specialized corpora — such as code, scientific text, or multilingual data — significantly improve downstream performance. Fertility rate, measuring average tokens per word, indicates tokenization efficiency across languages. **Tokenization directly determines a model's ability to represent and generate text, making vocabulary design one of the most consequential yet often overlooked architectural decisions in modern NLP systems.**