vocabulary size

Vocabulary size defines the number of unique tokens a model can represent, balancing coverage against efficiency. **Trade-offs**: **Smaller vocabulary** (8K-32K): Longer sequences (more tokens per text), smaller embedding matrix, harder time with rare words. **Larger vocabulary** (50K-150K): Shorter sequences, larger embedding matrix, better coverage of subwords. **Typical sizes**: GPT-2: 50,257. LLaMA: 32,000. BERT: 30,522. GPT-4: around 100K. **Impact on training**: Vocabulary size affects embedding parameter count (vocab_size x hidden_dim), softmax computation cost, memory usage. **Impact on inference**: Affects token usage (API costs), context window utilization, per-token information density. **Language considerations**: English-centric vocabularies inefficient for other languages. Multilingual models need larger vocabularies. **Specialized vocabularies**: Code models may add programming tokens, domain models may add technical terms. **Hyperparameter choice**: Balance between vocabulary size, sequence length, and model capacity. No universally optimal size.