Tokenization is the process of converting raw text into a sequence of discrete tokens (subword units) that serve as the input vocabulary for language models — determining how text is segmented into meaningful units, where the tokenizer's vocabulary size and algorithm directly impact model performance, multilingual capability, and inference efficiency.

Tokenization Approaches

Byte Pair Encoding (BPE)

1. Start with character vocabulary + special end-of-word token. 2. Count all adjacent character pairs in training corpus. 3. Merge the most frequent pair into a new token. 4. Repeat steps 2-3 until desired vocabulary size reached.

• Example: "l o w" appears 5 times → merge to "lo w" → "low" appears 5 times → merge to single token "low".
• Rare words split into subwords; common words become single tokens.
• GPT-2/3/4 use byte-level BPE (operates on bytes, not Unicode characters → handles any text).

WordPiece (BERT)

• Similar to BPE but merges based on likelihood improvement, not frequency.
• Merge pair that maximizes: $\log P(AB) - \log P(A) - \log P(B)$.
• Uses ## prefix for continuation tokens: "playing" → ["play", "##ing"].
• Vocabulary: 30,522 tokens for BERT.

SentencePiece

• Language-agnostic: Treats input as raw Unicode bytes — no pre-tokenization (no word splitting rules).
• Supports BPE and Unigram methods.
• Unigram: Start with large vocab → iteratively remove tokens that least affect likelihood.
• Used by: T5, LLaMA, mBART, XLM-R.
• Advantage: Handles any language (CJK, Arabic, etc.) without language-specific rules.

Vocabulary Size Impact

• Larger vocab → shorter sequences → faster inference, but larger embedding table.
• GPT-4: ~100K tokens. LLaMA: 32K. LLaMA-3: 128K.

Tokenization Challenges

• Number handling: "123456" might tokenize as ["123", "456"] → model doesn't understand mathematical relationship.
• Multilingual fairness: English words are often single tokens; other languages get split into many subwords → higher cost per concept.
• Whitespace sensitivity: Leading spaces, tabs, newlines affect tokenization in surprising ways.

Tokenization is the often-overlooked foundation that constrains everything a language model can do — a poorly designed tokenizer wastes model capacity on suboptimal text segmentation, while a well-designed one enables efficient multilingual processing and better numerical reasoning.