vocabulary

Vocabulary in NLP models is the set of discrete tokens (sub-words, characters, or words) that the model can represent and generate, directly impacting model size, inference speed, and multilingual capability. Tokenization: text is broken into tokens; BPE (Byte Pair Encoding) or WordPiece are standard algorithms. Size trade-off: Small vocab (~30k) = fewer parameters in embedding matrix, but longer sequences (more tokens per word); Large vocab (~100k-250k) = larger embedding matrix, but shorter sequences (more compression). Embeddings: Input/Output embedding matrices are often largest part of small models (e.g., 50% of 1B param model). OOV (Out Of Vocabulary): sub-word tokenizers eliminate OOV by falling back to bytes/characters. Multilingual: requires much larger vocab to cover different scripts efficiently (e.g., GPT-4 vs Llama 2). Efficiency: larger vocab increases softmax compute cost at output. Special tokens: [CLS], [SEP], [PAD], . Vocabulary selection is a static decision made before pre-training and cannot be changed easily.