word embedding

**Embeddings** — learned vector representations that map discrete tokens (words, items, categories) to dense, continuous vectors where semantically similar items are close together. **Why Embeddings?** - One-hot encoding: "cat" = [0,0,1,0,...,0] — 50,000-dimensional, sparse, no similarity info - Embedding: "cat" = [0.23, -0.45, 0.87, ...] — 256-dimensional, dense, "cat" close to "dog" **Classic Word Embeddings** - **Word2Vec** (2013): "King - Man + Woman ≈ Queen." Skip-gram or CBOW training on word co-occurrence - **GloVe** (2014): Matrix factorization of word co-occurrence statistics - **FastText** (2016): Sub-word embeddings — handles misspellings and rare words **Modern Contextual Embeddings** - **BERT/GPT embeddings**: Same word gets different vectors depending on context ("bank" near "river" vs "bank" near "money") - Subword tokenization (BPE, SentencePiece) replaced whole-word embedding **Embedding Layer in Practice** - Just a lookup table: Matrix of shape (vocab_size × embedding_dim) - Token ID → row of the matrix → embedding vector - Trained end-to-end with the rest of the network **Beyond NLP** - Recommendation systems: User/item embeddings - Graph neural networks: Node embeddings - Code: Code token embeddings (Codex, StarCoder) **Embeddings** are one of deep learning's most fundamental concepts — they convert the discrete world into the continuous space where neural networks operate.