Subword Tokenization
Subword tokenization splits text into units smaller than words but larger than single characters, so that a fixed vocabulary of tens of thousands of pieces can represent any string, including rare words, names, typos and words never seen in training. Common words stay whole, while rarer ones are assembled from reusable fragments such as a stem and a suffix. It resolves the old dilemma of word-level vocabularies, which cannot represent unseen words and become enormous, versus character-level ones, which are open-ended but make sequences very long.
Byte pair encoding is the most widely used algorithm, and it was borrowed from data compression. Philip Gage described it in 1994 as a way to replace the most frequent pair of bytes with an unused symbol. Sennrich, Haddow and Birch adapted it for translation in "Neural Machine Translation of Rare Words with Subword Units" (ACL 2016): start with individual characters, count adjacent pairs across the corpus, merge the most frequent pair into a new symbol, and repeat for a chosen number of merges. Frequent pairs such as e and s become es, then es and t become est, and the ordered merge list is the tokenizer. At inference the same merges are replayed greedily on new text. GPT-2 (2019) popularized a byte-level variant that starts from the 256 byte values, so any input can be encoded without an unknown token, giving a vocabulary of 50,257.
WordPiece and the unigram model differ in how the vocabulary is chosen, not in the goal. WordPiece, developed by Schuster and Nakajima in 2012 for Japanese and Korean voice search and used in Google's neural translation system and in BERT (Devlin and colleagues, 2018), also builds a vocabulary bottom-up, but picks merges that most increase the likelihood of the training data rather than by raw frequency, and marks word-internal pieces with a prefix such as ##. The unigram language model approach of Taku Kudo ("Subword Regularization", ACL 2018) works top-down: begin with a large candidate vocabulary and repeatedly prune the pieces whose removal hurts a unigram likelihood least. It can output several plausible segmentations of the same word with probabilities, which enables sampling different splits during training as regularization.
SentencePiece is a toolkit rather than a fourth algorithm. Kudo and Richardson's 2018 EMNLP system paper described a library that implements both BPE and unigram training directly on raw text, treating whitespace as an ordinary symbol, usually rendered as an underscore-like mark. That makes tokenization reversible and language-independent, with no need for a separate word-splitting step, which suits languages such as Chinese and Japanese that do not separate words with spaces. Many open models, including the Llama and T5 families, use it, though newer ones sometimes switch to byte-level BPE with larger vocabularies.
Tokens are not words, and several pitfalls follow from that. Models never see letters directly, so tasks like counting characters in a word, reversing strings or exact spelling are hard in ways that look like stupidity but reflect the representation. Multi-digit numbers may be split unevenly, which has been linked to arithmetic errors. Leading spaces are usually part of the token, so a word at the start of a line and after a space map to different ids. Rarely seen strings can map to undertrained tokens that trigger bizarre behavior, as in the glitch tokens reported in GPT models in 2023. And tokenizers trained mostly on English text are far less efficient for other languages, a cost disparity covered under low-resource languages.
The tokenizer is frozen before model training and tied to it forever. Changing the vocabulary means retraining or adapting embeddings, so tokenizer choice is an early architectural decision that affects context length, cost per request and how well rare scripts are handled. Vocabulary size is the central trade: larger vocabularies shorten sequences but enlarge the embedding and output layers and leave rare tokens undertrained.
Read subword tokenization as a learned compromise between letters and words: it spends vocabulary on the fragments that recur most, so ordinary text stays compact while anything unfamiliar can still be spelled out from smaller reusable parts.