Subword Tokenization is the text preprocessing technique that segments input text into a vocabulary of subword units — smaller than whole words but larger than individual characters — enabling language models to handle any text (including rare words, misspellings, and novel compounds) by decomposing unknown words into known subword pieces while keeping common words as single tokens for efficiency.

Why Not Words or Characters?

• Word-level tokenization: Creates a fixed vocabulary of whole words. Any word not in the vocabulary is mapped to a generic [UNK] token, losing all information. Vocabulary must be enormous (500K+) to cover rare words, inflections, and compound words across languages.
• Character-level tokenization: Every possible text is representable, but sequences become very long (a 500-word paragraph becomes ~2500 characters), increasing compute cost quadratically for attention-based models. Characters also carry less semantic information per token.
• Subword tokenization: The sweet spot — vocabulary of 32K-100K subword units captures common words as single tokens ("the", "running") and decomposes rare words into meaningful pieces ("un" + "predict" + "ability").

Major Algorithms

• BPE (Byte Pair Encoding): Start with individual characters. Repeatedly merge the most frequent adjacent pair into a new token. After K merges, the vocabulary contains K+base_chars tokens. GPT-2, GPT-3/4, and Llama use BPE variants. "tokenization" → ["token", "ization"]. Training is greedy frequency-based.

• WordPiece: Similar to BPE but selects merges that maximize the language model likelihood of the training corpus (not just frequency). The merge that most increases the probability of the training data is chosen. Used by BERT and its variants. Uses ## prefix for continuation pieces: "tokenization" → ["token", "##ization"].

• Unigram (SentencePiece): Starts with a large candidate vocabulary and iteratively removes tokens whose removal least decreases the training corpus likelihood. The final vocabulary is the smallest set that represents the training corpus well. Used by T5, ALBERT, and XLNet. SentencePiece implements both BPE and Unigram with raw text input (no pre-tokenization by spaces).

Vocabulary Size Tradeoffs

Larger vocabularies produce shorter token sequences (better for long contexts) but require a larger embedding matrix and may underfit rare tokens. Most modern LLMs use 32K-128K tokens.

Multilingual Considerations

For multilingual models, the tokenizer must allocate vocabulary across languages. If 90% of training data is English, 90% of the vocabulary will be English-optimized, causing non-Latin scripts (Chinese, Arabic, Devanagari) to be over-segmented into many small pieces per word — increasing sequence length and degrading efficiency for those languages.

Subword Tokenization is the linguistic compression layer that makes language models tractable — resolving the fundamental tension between vocabulary completeness and vocabulary efficiency by learning a data-driven decomposition that balances the two.