vocabulary size selection

**Vocabulary size selection** is the **decision process for choosing tokenizer vocabulary cardinality to balance compression, coverage, and model efficiency** - size choice strongly influences sequence length and memory behavior. **What Is Vocabulary size selection?** - **Definition**: Setting the number of learned token entries in the tokenizer vocabulary. - **Tradeoff Axis**: Larger vocabularies shorten sequences but increase embedding parameters. - **Coverage Effect**: Smaller vocabularies increase subword fragmentation for rare terms. - **Context Impact**: Token granularity affects effective use of fixed context windows. **Why Vocabulary size selection Matters** - **Compute Planning**: Vocabulary size changes embedding memory and inference throughput. - **Domain Performance**: Technical corpora may need enough tokens for frequent specialized terms. - **Training Efficiency**: Overly fragmented tokenization can slow convergence. - **Serving Cost**: Sequence length directly affects attention compute in transformer models. - **Quality Stability**: Balanced sizes reduce both OOV fragmentation and parameter bloat. **How It Is Used in Practice** - **Curve Analysis**: Plot sequence-length reduction versus vocabulary growth on target corpora. - **Task Benchmarks**: Compare downstream quality across candidate vocabulary sizes. - **Lifecycle Review**: Reassess size decisions when language mix or product domain shifts. Vocabulary size selection is **a central tokenizer design decision with system-wide impact** - correct sizing improves model efficiency and domain coverage simultaneously.