Word Embeddings Word2vec Semantic Representation Learning
# Word Embeddings: Word2Vec & Semantic Representation Learning
## Introduction & Motivation
Word2Vec learns dense word representations via prediction tasks (Skip-gram: predict context from word; CBOW: predict word from context). 300-dimensional vectors capture semantic meaning—king - man + woman ≈ queen. Foundation for NLP; enables transfer learning; interpretable similarity.
Motivation: One-hot encoding wastes dimensionality; no semantic information. Dense embeddings compact; semantic relationships encoded in vector geometry. Pre-trained embeddings reduce training data requirements.
Applications: Text classification, machine translation, sentiment analysis, question answering, knowledge graphs.
---
## Core Concepts & Theory
### Skip-gram Model
Predict context words given center word. For each word, maximize:
$$\sum_{c \in ext{context}} \log P(w_c | w_i)$$
### CBOW Model
Predict center word from context. Simpler; faster training.
### Softmax Approximation
Full softmax intractable (vocabulary size ~100K). Use negative sampling: discriminate true context from noise.
---
## Mathematical Formulation
Skip-gram objective:
$$\mathcal{L} = -\sum_{(w_i, w_c) \in D} \log \sigma(\mathbf{w}_c^T \mathbf{w}_i) - \sum_{k=1}^{K} \log(1 - \sigma(\mathbf{w}_k^T \mathbf{w}_i))$$
where D is training pairs, K is negative samples.
---
## Advanced Theory & Extensions
### Negative Sampling
Sample K noise words; avoid computing full softmax. K = 5-20 typical.
### Subword Information (FastText)
Learn character n-gram embeddings; handle OOV words via compositionality.
### Contextual Embeddings (BERT, GPT)
Context-dependent representations; surpass static word embeddings.
---
## Computational Considerations
Training: O(|D| imes d imes K) via SGD (|D| is corpus size, d is embedding dim).
Inference: O(d) for similarity computation.
---
## Practical Implementation Strategies
### Embedding Dimension
d = 50-300 typical; larger for complex vocabularies.
### Window Size
context_window = 2-10; smaller → syntactic, larger → semantic.
### Negative Samples
K = 5 for large corpora, K = 15-20 for small.
---
## Benchmark Datasets & Evaluation
Word Analogy: Semantic (king-man+woman) and syntactic (city-cities) relations.
Similarity: Correlation with human similarity judgments (SimLex999, RW, WordSim353).
---
## Key Challenges & Limitations
### Out-of-Vocabulary
Rare words poorly represented. Solutions: character n-grams (FastText), subword tokenization (BPE).
### Static Embeddings
Same embedding regardless of context; BERT/GPT address via contextual embeddings.
---
## Hyperparameter Tuning
embedding_dim \in {50, 100, 300\}, window_size \in {5, 10\}, negative_samples \in {5, 15\}, min_count \in {1, 5\}.
---
## Real-World Applications & Case Studies
Search Engines: Query-document similarity via embeddings.
Recommendation: User/item embeddings for collaborative filtering.
---
## Integration with Other Methods
Word2Vec + RNN → Sequence models with semantic initialization.
Word2Vec + Knowledge Graphs → Alignment of text and structured data.
---
## Future Research Directions
Multilingual embeddings; dynamic embeddings; fairness in embeddings.
---
## Summary & Key Takeaways
Word2Vec learns dense semantic word representations via prediction tasks, enabling efficient transfer learning and interpretable similarity relationships.
Principles:
1. Skip-gram/CBOW learn from prediction tasks.
2. Negative sampling enables scalability.
3. Embeddings capture semantic relationships.
4. Dimensionality scales with vocabulary.
5. Pre-trained embeddings transfer to downstream tasks.
---
---
## Appendix: Practical Labs
### Lab 1: Word2Vec Basics
from gensim.models import Word2Vec
from gensim.corpora import PlaintextCorpus
import numpy as np
# Create simple corpus
sentences = [
['the', 'cat', 'sat', 'on', 'the', 'mat'],
['the', 'dog', 'played', 'in', 'the', 'yard'],
['cats', 'and', 'dogs', 'are', 'friends'],
['I', 'like', 'cats', 'very', 'much']
]
model = Word2Vec(sentences, vector_size=50, window=2, min_count=1, workers=1, seed=42)
# Get embedding for 'cat'
cat_vec = model.wv['cat']
print(f"Embedding shape: {cat_vec.shape}")
assert cat_vec.shape == (50,), "Should have 50-D embedding"
assert len(model.wv) > 0, "Should have learned embeddings"
print("✓ Word2Vec working")
if __name__ == "__main__":
print("Lab 1: Word2Vec - PASSED")### Lab 2: Semantic Similarity
from gensim.models import Word2Vec
import numpy as np
sentences = [
['king', 'queen', 'man', 'woman'],
['paris', 'london', 'france', 'england'],
['red', 'blue', 'color', 'shade'],
['good', 'bad', 'happy', 'sad'],
['cat', 'dog', 'animal', 'pet'],
['car', 'truck', 'vehicle', 'automobile'],
]
model = Word2Vec(sentences, vector_size=32, window=2, min_count=1, workers=1, seed=42)
# Compute similarity
sim_king_queen = model.wv.similarity('king', 'queen')
sim_king_dog = model.wv.similarity('king', 'dog')
print(f"Similarity(king, queen): {sim_king_queen:.4f}")
print(f"Similarity(king, dog): {sim_king_dog:.4f}")
assert 0 <= sim_king_queen <= 1, "Similarity should be in [0,1]"
print("✓ Semantic similarity working")
if __name__ == "__main__":
print("Lab 2: Semantic Similarity - PASSED")### Lab 3: Analogy
from gensim.models import Word2Vec
sentences = [
['king', 'queen', 'prince', 'princess', 'man', 'woman', 'boy', 'girl'],
['france', 'paris', 'england', 'london', 'germany', 'berlin'],
['big', 'bigger', 'small', 'smaller', 'fast', 'faster'],
]
model = Word2Vec(sentences, vector_size=32, window=2, min_count=1, workers=1, seed=42)
# Compute analogy: king - man + woman ≈ queen
king_vec = model.wv['king']
man_vec = model.wv['man']
woman_vec = model.wv['woman']
analogy_vec = king_vec - man_vec + woman_vec
# Find closest word
closest = model.wv.most_similar(positive=[analogy_vec], topn=1)
print(f"king - man + woman ≈ {closest[0][0]}")
assert len(closest) > 0, "Should find similar words"
print("✓ Analogy working")
if __name__ == "__main__":
print("Lab 3: Analogy - PASSED")### Lab 4: Vector Dimension Effect
from gensim.models import Word2Vec
import numpy as np
sentences = [
['the', 'quick', 'brown', 'fox', 'jumps', 'over', 'the', 'lazy', 'dog'],
['a', 'fast', 'brown', 'fox', 'leaps', 'across', 'a', 'sleepy', 'dog'],
] * 20 # Repeat for more data
dims = [16, 32, 64, 128]
for dim in dims:
model = Word2Vec(sentences, vector_size=dim, window=2, min_count=1, workers=1, seed=42)
vocab_size = len(model.wv)
print(f"Dimension {dim}: Vocab size {vocab_size}")
assert vocab_size > 0, "Should have vocabulary"
print("✓ Vector dimension effect working")
if __name__ == "__main__":
print("Lab 4: Vector Dimension - PASSED")