unigram tokenization

**Unigram tokenization** is the **subword modeling approach that selects token segmentations by maximizing probability under a unigram language model over candidate pieces** - it uses probabilistic vocabulary pruning rather than deterministic merges. **What Is Unigram tokenization?** - **Definition**: Tokenizer training method where a candidate vocabulary is iteratively reduced by likelihood impact. - **Segmentation Logic**: Multiple possible segmentations are scored and best-probability path is chosen. - **Model Fit**: Implemented in frameworks like SentencePiece for flexible subword learning. - **Behavioral Trait**: Can produce compact vocabularies with good coverage of rare forms. **Why Unigram tokenization Matters** - **Probability Grounding**: Objective-driven segmentation can align better with language distribution. - **Rare Token Handling**: Maintains compositional paths for uncommon words and morphologies. - **Compression Tradeoff**: Balances sequence length and vocabulary size effectively. - **Multilingual Support**: Performs well on diverse scripts with shared subword units. - **Model Performance**: Tokenizer quality impacts downstream learning efficiency and output fluency. **How It Is Used in Practice** - **Candidate Initialization**: Start with broad piece set before iterative pruning. - **Likelihood Monitoring**: Track objective changes to choose stable pruning endpoints. - **Task Validation**: Evaluate segmentation impact on both training loss and downstream metrics. Unigram tokenization is **a probabilistic alternative to merge-based subword tokenization** - unigram methods can offer strong robustness when tuned on representative corpora.