A vocabulary is not a list of words; it is a compression codebook that determines how many tokens a model needs to represent any given text, and that token count directly controls training cost, inference latency, context consumption, and cross-lingual fairness. Every language model — BERT, GPT-2, LLaMA, GPT-4o — begins by converting raw text into a sequence of integer indices drawn from a fixed table. The table is the vocabulary. Its size $V$ is a single number, but it couples to four things simultaneously: the embedding matrix ($V \times d$ parameters, duplicated in the output projection unless weight-tying is used), the softmax bottleneck ($2Vd$ FLOPs per token at the output layer), the sequence length (more tokens per word means fewer words fit in a fixed context window), and the per-API-call cost (tokens are the billing unit). A user sending the same paragraph in Chinese through GPT-2's 50,257-token vocabulary pays 2.7 times as many tokens as an English user; through LLaMA-3's 128,256-token vocabulary the ratio falls to 1.3 times. The entire gap is a design choice made at the tokenizer training stage, months before the first gradient is computed.

The construction algorithm — BPE, WordPiece, or Unigram — determines the merge table, and the merge table IS the model's prior over sub-word structure. Byte Pair Encoding starts with 256 byte-level tokens and iteratively merges the most frequent adjacent pair. After $k$ merges the vocabulary has $256 + k$ entries and the corpus has shrunk; the fertility (tokens per word) drops from 4.50 at the byte level toward an asymptote that depends on the language and the corpus. On English, GPT-2's 50,000 merges reach fertility 1.30, LLaMA-3's 128,000 merges reach 1.12, and GPT-4o's 200,000 merges reach 1.05. The marginal value of each additional merge follows Zipf's law: the first merge replaces roughly 50,000 occurrences in a million-token corpus; the thousandth merge replaces only 281 — a 177.8-fold decline. This is why doubling the vocabulary from 50K to 100K buys only a few percent improvement in fertility while doubling the embedding matrix. WordPiece (used by BERT) works top-down — it starts with individual characters and scores candidate merges by mutual information rather than raw frequency, producing a slightly different segmentation but comparable fertility at the same $V$. Unigram (SentencePiece) works in reverse: it starts with a large candidate set and prunes tokens whose removal least increases the corpus likelihood, yielding a vocabulary optimised under a probabilistic model rather than a greedy frequency count.

The cross-lingual fertility gap is the single largest hidden tax in multilingual deployment, and it is entirely a function of vocabulary design. At $V = 50{,}257$ (GPT-2), Chinese text produces 3.47 tokens per word against English's 1.30 — a 2.7$\times$ ratio. This means a Chinese user's prompt consumes 2.7$\times$ the context window, costs 2.7$\times$ in API billing, and runs 2.7$\times$ slower at inference. The cause is straightforward: BPE trained on an English-dominated corpus allocates most of its merge budget to English bigrams, leaving CJK characters largely unmerged. LLaMA-3's decision to quadruple the vocabulary to 128,256 and train on a balanced multilingual corpus cut the ratio to 1.26$\times$; GPT-4o at 200,019 cut it further to 1.14$\times$. The cost of this fix is a 4$\times$ larger embedding matrix — from 262 M parameters at $V = 32{,}000$ to 3,277 M at $V = 200{,}019$ with $d = 8{,}192$ — but at 200B total parameters that embedding is only 1.64% of the model, so the trade is overwhelmingly favorable at scale. At small scale it is not: BERT-base's 30,522-token vocabulary already consumes 42.62% of its 110M parameters in the embedding and output layers.

The embedding matrix is the one component whose parameter count scales with vocabulary size rather than depth, and this creates a crossover in model design. The input embedding is a lookup table of shape $V \times d$; the output projection (which computes logits for the softmax) is another matrix of shape $d \times V$. When weights are tied (BERT, GPT-2, T5) the two share parameters, halving the embedding cost; when untied (LLaMA, Mistral) they are independent, and the total embedding cost is $2Vd$. For LLaMA-7B ($V = 32{,}000$, $d = 4{,}096$): $2 \times 32{,}000 \times 4{,}096 = 262.1$ M parameters, or 3.89% of 6.738B. For LLaMA-70B ($V = 32{,}000$, $d = 8{,}192$): $2 \times 32{,}000 \times 8{,}192 = 524.3$ M, or 0.76% of 69.0B. The fraction drops because depth-dependent parameters (attention and MLP, scaling as $12d^2$ per layer) grow quadratically with $d$ while embedding grows linearly. This is why small models keep small vocabularies (BERT-base: 30,522) and large models can afford large ones (GPT-4o: 200,019). LLaMA-3-8B ($V = 128{,}256$, $d = 4{,}096$) sits at the crossover: its embedding is 13.08% of total parameters — large enough to matter for fine-tuning memory, small enough to justify the multilingual fertility gains.

The softmax bottleneck at the output layer is the computational cost that vocabulary size imposes on every forward pass. Computing logits requires a matrix multiply of shape $[B \times d] \times [d \times V]$, costing $2Vd$ FLOPs per token. At $V = 32{,}000$ and $d = 4{,}096$: $2 \times 32{,}000 \times 4{,}096 = 262$ M FLOPs, equivalent to 2.9 MLP-layer forward passes. At $V = 200{,}019$ and $d = 8{,}192$: $2 \times 200{,}019 \times 8{,}192 = 3{,}277$ M FLOPs, equivalent to 18.2 MLP-layer forward passes. This is why adaptive softmax and mixture-of-softmax were invented for large-vocabulary language modeling: they replace the full $V$-way dot product with a hierarchical structure that evaluates rare tokens only when a coarse gate selects their cluster. In modern transformer practice, however, the standard approach is to accept the full softmax cost and amortize it through tensor parallelism — the output projection is trivially shardable across GPUs because it is a single matmul with no sequential dependency.

Pre-tokenisation determines what the algorithm can see, and it is more consequential than the merge count. GPT-2 splits text with a regex that isolates contractions, punctuation, and whitespace before BPE runs on each fragment; this prevents merges that cross word boundaries (so the and theory never merge into a single token) but also prevents cross-word patterns like _of_the from becoming single tokens that would reduce fertility. SentencePiece treats whitespace as a regular character (encoded as ▁) and runs BPE or Unigram over the entire sentence, allowing cross-word merges and producing tokens like ▁of▁the. This is why SentencePiece tokenizers often have slightly lower fertility at the same $V$ for agglutinative languages (Finnish, Turkish) where morpheme boundaries do not align with whitespace. The regex pattern in tiktoken (used by GPT-3.5, GPT-4, GPT-4o) is more permissive than GPT-2's original, allowing multi-word merges for common phrases while still preventing merges across most punctuation boundaries.

Normalization is the irreversible step: once applied during tokenizer training, the model cannot distinguish the input variants that were collapsed. NFKC normalization maps Unicode compatibility variants to their canonical forms: fi → fi, ² → 2, ½ → 1/2, Ω → Ω. This helps coverage (fewer unique characters means the byte-level fallback activates less often) but destroys information. The expression 2² normalizes to 22, which is mathematically wrong. GPT-2 and tiktoken apply no normalization and preserve the raw byte stream; BERT lowercases and strips accents (so Résumé becomes resume, losing two distinctions); SentencePiece defaults to NFKC. The choice propagates irreversibly: a model trained on NFKC-normalized text cannot learn to distinguish fi from the ligature fi, and a model trained on lowercased text cannot generate case-sensitive output. This is why GPT-4o's tokenizer uses no normalization despite supporting 100+ languages — the model is large enough to learn the variants from data rather than collapsing them at the tokenizer.

Special tokens are the vocabulary entries that carry structural rather than linguistic meaning, and their count ranges from 1 to over 100. GPT-2 has a single special token: <|endoftext|> (ID 50256). BERT has 5: [CLS], [SEP], [PAD], [MASK], [UNK]. LLaMA has 3: , , . ChatGPT-family models add dozens of control tokens for roles (<|im_start|>, <|im_end|>), tool calls, and system prompts. These tokens have learned embeddings like any other, but they never appear in natural text — their embeddings are trained entirely from the structured formatting of fine-tuning data. The token is the fallback for byte sequences outside the vocabulary; BPE with byte-level fallback sets its probability to zero because every byte is in the base vocabulary, making true coverage 100%. WordPiece without byte fallback has a nonzero rate that depends on $V$: at $V = 30{,}522$ (BERT), roughly 0.5% of tokens in web-crawled text are [UNK], mostly emoji, rare scripts, and control characters.

Byte-level BPE is the reason modern tokenizers never produce unknowns: the base vocabulary is the 256 possible byte values, so any input byte sequence is representable even if no merge applies. This is a fundamental shift from character-level tokenizers (where the base is Unicode codepoints, roughly 150,000 entries) and word-level tokenizers (where the base is the observed word set, typically 50,000–500,000 entries with a long tail of unknowns). The cost of byte-level fallback is fertility: a 4-byte UTF-8 character that has not been merged consumes 4 tokens instead of 1. For English (mostly ASCII, 1 byte per character) this cost is near zero. For Chinese (3 bytes per character in UTF-8) it is severe at small vocabularies: byte-level fertility for Chinese is 3.00 tokens per word at $V = 256$, worse than English's 4.50 tokens per word (because Chinese words are shorter in character count but each character is 3 bytes). As $V$ grows and CJK merges accumulate, the gap inverts: at $V = 128{,}256$ the Chinese fertility (1.41) is close to English (1.12). This inversion happens at roughly $V = 100{,}000$, which is why no tokenizer with $V < 100{,}000$ is truly multilingual-competitive.

The optimal vocabulary size scales with the square root of the model's parameter count, and deviations from this scaling law waste either compute or capacity. The heuristic $V_{\text{opt}} \propto \sqrt{N}$ emerges from a simple argument: the embedding cost is $2Vd$ and the transformer cost is $12Ld^2$ (where $L$ is the number of layers), so the embedding is a constant fraction of the total when $V \propto d \propto \sqrt{N/L}$. The ratio $V / \sqrt{N}$ is a rough diagnostic: BERT-base sits at 2.910 (overscaled — embedding dominates), LLaMA-7B at 0.390 (underscaled for multilingual, acceptable for English-only), GPT-4o at 0.447. LLaMA-3-8B's jump to $V = 128{,}256$ pushed its ratio to 1.431, which explains the 13.08% embedding fraction — aggressive, but the fertility gains (1.12 English, 1.41 Chinese, down from 1.45/3.82) justify it because the context-window and API-cost savings compound across every inference. The scaling law also implies that the 32,000-token vocabularies used by LLaMA-70B and Mistral-7B are relics of the 7B design: at 70B parameters the optimal $V$ would be roughly $0.4 \times \sqrt{70 \times 10^9} \approx 106{,}000$, and LLaMA-3-70B's actual 128,256 sits close to that mark.

Weight tying between the input embedding and the output projection halves the parameter cost but constrains the model's representational freedom. When tied, a single $V \times d$ matrix serves both as the token-to-vector lookup and the logit-computing projection. The intuition is that the geometry of token embeddings should match the geometry of output logits — a token's embedding should be "close" to the logit direction that predicts it. This works well when $d$ is large relative to $V$ (every token gets a distinct direction) but breaks down when $V \gg d$: with $V = 200{,}019$ and $d = 8{,}192$, the output layer must map to a 200K-dimensional simplex using only 8,192 directions, and constraining the input embedding to the same geometry limits the model's ability to learn distinct input representations for near-synonyms. This is why LLaMA and Mistral untie weights despite the 2$\times$ parameter cost, and why GPT-2 (which ties weights) has $d = 1{,}600$ for $V = 50{,}257$ — a ratio of 31:1 that keeps the constraint loose.

Through the lens of system design, a vocabulary is the first and least revisable decision in a language model pipeline: it is frozen before pretraining begins, it determines the coordinate system in which every embedding, attention weight, and output logit operates, and changing it after training requires either retraining from scratch or a lossy adapter that maps old token IDs to new ones. The table above shows that the industry has converged toward larger vocabularies (128K–250K) as models have grown large enough to absorb the embedding cost, driven primarily by the multilingual fertility argument: the gap between English and Chinese shrinks from 2.67$\times$ at $V = 50{,}257$ to 1.12$\times$ at $V = 250{,}002$, and that compression ratio difference propagates into every downstream metric — context length, latency, cost, and the quality of cross-lingual transfer.