Tokenizer Design for Language Models covers the algorithms and engineering decisions for converting raw text into the integer token sequences that language models process — including BPE (Byte-Pair Encoding), WordPiece, Unigram (SentencePiece), and byte-level approaches that must balance vocabulary size, compression efficiency, multilingual coverage, and downstream model performance.

Why Tokenization Matters

Input: "unhappiness"

Character-level: [u,n,h,a,p,p,i,n,e,s,s] → 11 tokens (too long)
Word-level:      [unhappiness]             → 1 token (vocabulary too large)
Subword:         [un, happiness]           → 2 tokens (balanced!)
BPE:             [un, happ, iness]         → 3 tokens (data-driven)

Tokenization directly affects: context window utilization (fewer tokens = more text per context), training efficiency, handling of rare/novel words, multilingual fairness, and compute cost (cost ∝ number of tokens).

Major Tokenization Algorithms

BPE (Byte-Pair Encoding) Algorithm

# Training:
vocab = set of all bytes (256 base tokens)
for merge_step in range(num_merges):  # e.g., 32K merges
    # Count frequency of all adjacent token pairs in corpus
    pair_counts = count_pairs(corpus_tokens)
    # Merge the most frequent pair into a new token
    best_pair = argmax(pair_counts)
    new_token = best_pair[0] + best_pair[1]
    vocab.add(new_token)
    # Replace all occurrences in corpus
    corpus_tokens = replace_pair(corpus_tokens, best_pair, new_token)

# Encoding (inference):
# Greedily apply learned merges in priority order

Vocabulary Size Tradeoffs

Smaller vocab (e.g., 4K-8K):
  + Smaller embedding table
  + Each token well-trained (high frequency)
  - More tokens per text (longer sequences)
  - Higher compute cost for same text

Larger vocab (e.g., 100K-250K):
  + Fewer tokens per text (more efficient)
  + Better coverage of words/subwords
  - Larger embedding table (memory)
  - Rare tokens poorly trained
  - Larger LM head (classification over vocab)

Typical choices: 32K (Llama/Mistral), 50K (GPT-2), 100K (GPT-4/Llama3), 250K (Gemini)

Byte-Level BPE

GPT-2 introduced byte-level BPE: base vocabulary is 256 byte values, so any text (any language, any encoding) can be represented without UNK tokens. Combined with pre-tokenization rules (regex to split on whitespace, punctuation, numbers) to prevent merges across word boundaries.

Multilingual Tokenization Challenges

English-centric tokenizers compress English well (~1.3 tokens/word) but fragment non-Latin scripts:

• Chinese: 2-3 tokens per character (vs. 1 for English words)
• Arabic/Hindi: 3-5× more tokens per equivalent text
• This means non-English users get less 'value' per context window and per API dollar

Solutions: train BPE on balanced multilingual corpora, increase vocabulary size (100K+ for multilingual), or use separate tokenizers per language family.

Special Tokens

[BOS] / <s>        Beginning of sequence
[EOS] / </s>        End of sequence
[PAD]               Padding for batching
[UNK]               Unknown (avoided in byte-level BPE)
<|im_start|>        Chat formatting (OpenAI)
[INST] [/INST]      Instruction markers (Llama)
<tool_call>         Function calling markers

Tokenizer design is a foundational and often underappreciated decision in LLM development — the choice of algorithm, vocabulary size, training corpus, and special tokens has cascading effects on model efficiency, multilingual fairness, capability, and serving cost, making it one of the earliest and most consequential design decisions in the LLM development pipeline.