sentencepiece

SentencePiece tokenizes text directly from raw Unicode without language-specific pre-tokenization. **Key innovation**: Treats input as stream of characters, no pre-tokenization (like splitting on spaces) needed. Language-agnostic. **Algorithms**: Supports BPE and Unigram algorithms within the SentencePiece framework. **Unigram mode**: Starts with large vocabulary, prunes tokens that least reduce corpus likelihood. Maintains probability per token. **Whitespace handling**: Can represent spaces as special character (underscore), enabling lossless tokenization-detokenization. **Multilingual benefit**: Works uniformly across languages, critical for multilingual models. No language-specific preprocessing. **Used by**: T5, mT5, LLaMA, XLNet, ALBERT, and many multilingual models. **Training**: Learn vocabulary from corpus, produces model file for encoding. **Inference**: Probabilistic sampling possible (multiple tokenizations), deterministic mode for consistency. **Command line tools**: spm_train for training, spm_encode/spm_decode for usage. **Comparison**: More flexible than WordPiece, handles diverse scripts better. Standard choice for multilingual and open-source LLMs.