tokenization
**Tokenization** is the **process of converting raw text into a sequence of discrete tokens (subword units) that serve as the input vocabulary for language models** — determining how text is segmented into meaningful units, where the tokenizer's vocabulary size and algorithm directly impact model performance, multilingual capability, and inference efficiency.
**Tokenization Approaches**
| Method | Granularity | Vocabulary Size | Example: "unhappiness" |
|--------|-----------|----------------|------------------------|
| Word-level | Full words | 50K-500K | ["unhappiness"] |
| Character-level | Single chars | 26-256 | ["u","n","h","a","p","p","i","n","e","s","s"] |
| BPE (Subword) | Subword units | 32K-100K | ["un", "happiness"] |
| Byte-level BPE | Byte sequences | 50K-100K | ["un", "happ", "iness"] |
**Byte Pair Encoding (BPE)**
1. Start with character vocabulary + special end-of-word token.
2. Count all adjacent character pairs in training corpus.
3. Merge the most frequent pair into a new token.
4. Repeat steps 2-3 until desired vocabulary size reached.
- Example: "l o w" appears 5 times → merge to "lo w" → "low" appears 5 times → merge to single token "low".
- Rare words split into subwords; common words become single tokens.
- GPT-2/3/4 use byte-level BPE (operates on bytes, not Unicode characters → handles any text).
**WordPiece (BERT)**
- Similar to BPE but merges based on likelihood improvement, not frequency.
- Merge pair that maximizes: $\log P(AB) - \log P(A) - \log P(B)$.
- Uses ## prefix for continuation tokens: "playing" → ["play", "##ing"].
- Vocabulary: 30,522 tokens for BERT.
**SentencePiece**
- **Language-agnostic**: Treats input as raw Unicode bytes — no pre-tokenization (no word splitting rules).
- Supports BPE and Unigram methods.
- Unigram: Start with large vocab → iteratively remove tokens that least affect likelihood.
- Used by: T5, LLaMA, mBART, XLM-R.
- Advantage: Handles any language (CJK, Arabic, etc.) without language-specific rules.
**Vocabulary Size Impact**
| Vocab Size | Tokens/Word | Sequence Length | Compute |
|-----------|------------|----------------|--------|
| 4K | ~2.5 | Long sequences | High |
| 32K | ~1.3 | Medium | Medium |
| 100K | ~1.1 | Short | Lower |
| 256K | ~1.0 | Shortest | Lowest |
- Larger vocab → shorter sequences → faster inference, but larger embedding table.
- GPT-4: ~100K tokens. LLaMA: 32K. LLaMA-3: 128K.
**Tokenization Challenges**
- **Number handling**: "123456" might tokenize as ["123", "456"] → model doesn't understand mathematical relationship.
- **Multilingual fairness**: English words are often single tokens; other languages get split into many subwords → higher cost per concept.
- **Whitespace sensitivity**: Leading spaces, tabs, newlines affect tokenization in surprising ways.
Tokenization is **the often-overlooked foundation that constrains everything a language model can do** — a poorly designed tokenizer wastes model capacity on suboptimal text segmentation, while a well-designed one enables efficient multilingual processing and better numerical reasoning.