tokenization
**Tokenization** and **embeddings** are the first two steps of every language model — the bridge that turns human text into something a neural network can actually compute on. A model has no notion of letters or words; it only does arithmetic on vectors. Tokenization chops text into discrete units and assigns each an integer ID, and the embedding layer converts each ID into a learned vector that places it in a high-dimensional "meaning-space." Everything the model does afterward operates on those vectors. The diagram walks a short phrase through all four stages.\n\n```svg
```\n\n**Tokenization splits text into subword units.** Modern LLMs do not use whole words (the vocabulary would be enormous and would break on anything unseen) nor individual characters (sequences would be far too long). Instead they use subword schemes — Byte-Pair Encoding (BPE), WordPiece, or SentencePiece/Unigram — that learn a fixed vocabulary of common fragments. Frequent words become a single token; rare or novel words fracture into pieces. This is why "tokenizing" might split into `token` + `izing`, and why token counts, not word counts, drive context limits and API pricing.\n\n**Token IDs are just indices.** Once the text is segmented, each token is looked up in the vocabulary and replaced by its row number — a plain integer. A sentence becomes a list of IDs like `[8256, 4680, 1044]`. At this point there is still no meaning attached; the ID is only an address.\n\n**Embeddings turn IDs into learned vectors.** The embedding layer is a large table with one row per vocabulary entry, each row a vector of hundreds or thousands of numbers. Looking up a token ID returns its vector. Crucially these vectors are *learned* during training, so tokens that behave similarly drift close together, and directions in the space come to encode relationships — the classic illustration being that the vector arithmetic of `king − man + woman` lands near `queen`.\n\n**Position has to be added separately.** A raw embedding says *what* a token is but not *where* it sits in the sequence, and attention alone is order-blind. So positional information — learned position embeddings, or rotary encodings (RoPE) — is combined with the token embedding before the first Transformer layer, giving the model both identity and order.\n\n**The tokenizer is a fixed, upstream choice.** It is trained once, before the model, and then frozen — the model and tokenizer are a matched pair. A poor vocabulary hurts everywhere downstream: it inflates sequence length, wastes context, and handles some languages or code far less efficiently than others, which is why tokenizer design is a quietly consequential part of building a model.\n\n| Stage | Input | Output | Note |\n|---|---|---|---|\n| Tokenization | raw text | subword tokens | BPE / WordPiece / SentencePiece |\n| ID lookup | tokens | integer IDs | index into the vocabulary |\n| Embedding | IDs | dense vectors | learned table, meaning-space |\n| Positional encoding | vectors | vectors + order | learned or rotary (RoPE) |\n\nRead tokenization and embeddings through a *representation* lens rather than a *preprocessing* lens: they are not throwaway plumbing but the model's entire interface to language, and the choices there ripple through everything. The tokenizer fixes the granularity the model can ever perceive and sets how many tokens a given text costs, while the embedding table is where discrete symbols first become continuous geometry — the moment words become math, and the reason a neural network can reason about meaning at all.\n