Tokenization splits text into tokens that models process. Byte Pair Encoding BPE learns subword units by iteratively merging frequent character pairs. SentencePiece treats text as raw bytes enabling language-agnostic tokenization. Tiktoken is OpenAI fast BPE implementation. Vocabulary size trades off between granularity and efficiency: smaller vocabularies mean longer sequences larger vocabularies mean more parameters. Typical sizes range from 32K for efficiency to 100K for coverage. Tokenization affects model performance: poor tokenization increases sequence length and degrades understanding of rare words. Subword tokenization handles out-of-vocabulary words by breaking them into known pieces. Special tokens mark boundaries like beginning-of-sequence end-of-sequence and padding. Tokenizer training requires large diverse corpora and careful handling of whitespace punctuation and special characters. Multilingual tokenizers balance coverage across languages. Tokenization is often overlooked but critical: the same model with different tokenizers performs differently. Modern tokenizers are reversible allowing perfect reconstruction of original text.