Word Embeddings Word2vec Fasttext Glove
# Word Embeddings: Word2Vec, FastText & GloVe
## Introduction & Motivation
Word embeddings: dense vector representations of words. Word2Vec: skip-gram, CBOW models; context prediction. FastText: subword information; robust to rare words. GloVe: global co-occurrence; matrix factorization. Applications: NLP tasks, semantic similarity, downstream tasks.
Motivation: One-hot encoding sparse; embeddings dense and semantic. Transfer embeddings across tasks.
Applications: NLP, semantic similarity, text classification.
---
## Core Concepts & Theory
### Skip-Gram Model
Predict context from target word.
### CBOW (Continuous Bag of Words)
Predict target from context words.
### FastText
Subword n-grams; handle OOV words.
---
## Mathematical Formulation
Skip-gram objective:
$$L = -\sum_{t} \sum_{-m \leq j \leq m} \log p(w_{t+j} | w_t)$$
Softmax probability:
$$p(w_o | w_i) = \frac{\exp(v_o^T v_i)}{\sum_w \exp(v_w^T v_i)}$$
GloVe objective:
$$J = \sum_{i,j} f(X_{ij}) (v_i^T v_j + b_i + b_j - \log X_{ij})^2$$
where f(X_ij) = weighting function, X_ij = co-occurrence count.
---
## Advanced Theory & Extensions
### Negative Sampling
Approximate softmax; computational efficiency.
### Hierarchical Softmax
Tree structure; faster inference.
### Contextualized Embeddings
ELMo, BERT; context-dependent.
---
## Computational Considerations
Word2Vec: O(V · D · |C|) where V = vocab, D = dimension, |C| = context.
FastText: O(V · D · |C| · n-gram size).
GloVe: O(|X|) where |X| = non-zero co-occurrence entries.
---
## Practical Implementation Strategies
### Vocabulary Size
10K-100K typical; frequency cutoff.
### Embedding Dimension
100-300 standard; 768+ modern.
### Context Window
2-10 words typical; symmetric or asymmetric.
---
## Benchmark Datasets & Evaluation
Word Similarity: WordSim353, SimLex999.
Analogy: GoogleNews word analogy dataset.
Downstream Tasks: GLUE, SQuAD.
---
## Key Challenges & Limitations
### Static Embeddings
Same vector regardless of context; addressed by BERT.
### Training Data
Quality and size matter; corpus dependent.
### Polysemy
Homonyms map to single embedding.
---
## Hyperparameter Tuning
Embedding dimension: 100-300; downstream task dependent.
Context window: 2-10; larger → more general.
Learning rate: 0.01-0.1; SGD standard.
---
## Real-World Applications & Case Studies
Text Classification: Embedding + attention.
Semantic Similarity: Cosine distance in embedding space.
Machine Translation: Initialize model embeddings.
---
## Integration with Other Methods
Embeddings + RNN → sequence modeling.
Embeddings + CNN → text classification.
---
## Summary & Key Takeaways
Word embeddings via Skip-Gram, CBOW, FastText, and GloVe learn dense semantic representations for NLP tasks.
Principles:
1. Skip-Gram: context prediction.
2. CBOW: target prediction.
3. Negative sampling: efficiency.
4. FastText: subword robustness.
5. Transfer: reuse embeddings.
---
---
## Appendix: Practical Labs
### Lab 1: Skip-Gram Training
import numpy as np
def skip_gram_step(target, context, embeddings, learning_rate=0.01):
"""Single skip-gram training step"""
# Forward pass
target_embed = embeddings[target] # (D,)
context_embed = embeddings[context] # (D,)
# Softmax
scores = np.dot(embeddings, target_embed) # (V,)
exp_scores = np.exp(scores - np.max(scores))
probs = exp_scores / exp_scores.sum() # Softmax
# Loss: cross-entropy
loss = -np.log(probs[context] + 1e-8)
# Gradient
probs[context] -= 1 # Hard assignment for this context
grad = np.dot(embeddings.T, probs[:, np.newaxis]) # (D, 1)
# Update
embeddings[target] -= learning_rate * grad.flatten()
return loss
# Test
np.random.seed(42)
embeddings = np.random.randn(100, 50) # 100 vocab, 50-D
loss = skip_gram_step(target=5, context=10, embeddings=embeddings)
assert np.isfinite(loss), "Loss finite"
print("✓ Skip-gram working")
if __name__ == "__main__":
print("Lab 1: SkipGram - PASSED")### Lab 2: CBOW Training
import numpy as np
def cbow_step(context_words, target_word, embeddings, learning_rate=0.01):
"""CBOW training step"""
# Average context embeddings
context_embed = embeddings[context_words].mean(axis=0) # (D,)
# Softmax
scores = np.dot(embeddings, context_embed)
exp_scores = np.exp(scores - np.max(scores))
probs = exp_scores / exp_scores.sum()
# Loss
loss = -np.log(probs[target_word] + 1e-8)
# Gradient
probs[target_word] -= 1
# Update (simplified)
for cw in context_words:
embeddings[cw] -= learning_rate * np.dot(embeddings.T, probs)[:, np.newaxis].flatten() / len(context_words)
return loss
# Test
np.random.seed(42)
embeddings = np.random.randn(100, 50)
loss = cbow_step(context_words=[5, 6, 7], target_word=10, embeddings=embeddings)
assert np.isfinite(loss), "Loss finite"
print("✓ CBOW working")
if __name__ == "__main__":
print("Lab 2: CBOW - PASSED")### Lab 3: Negative Sampling
import numpy as np
def negative_sampling_loss(target, context, embeddings, neg_samples, learning_rate=0.01):
"""Negative sampling for efficiency"""
target_embed = embeddings[target]
context_embed = embeddings[context]
# Positive pair
pos_score = np.dot(target_embed, context_embed)
pos_loss = -np.log(1 / (1 + np.exp(-pos_score)) + 1e-8)
# Negative pairs
neg_loss = 0
for neg in neg_samples:
neg_embed = embeddings[neg]
neg_score = np.dot(target_embed, neg_embed)
neg_loss += -np.log(1 - 1 / (1 + np.exp(-neg_score)) + 1e-8)
total_loss = pos_loss + neg_loss / len(neg_samples)
return total_loss
# Test
np.random.seed(42)
embeddings = np.random.randn(100, 50)
loss = negative_sampling_loss(target=5, context=10, embeddings=embeddings, neg_samples=[20, 30, 40])
assert np.isfinite(loss), "Loss finite"
print("✓ Negative sampling working")
if __name__ == "__main__":
print("Lab 3: NegativeSampling - PASSED")### Lab 4: Word Similarity
import numpy as np
def compute_word_similarities(embeddings, word_idx, top_k=5):
"""Find similar words"""
word_embed = embeddings[word_idx]
# Cosine similarity
similarities = np.dot(embeddings, word_embed) / (
np.linalg.norm(embeddings, axis=1) * np.linalg.norm(word_embed) + 1e-8
)
# Top-k
top_indices = np.argsort(-similarities)[1:top_k+1] # Exclude self
return top_indices, similarities[top_indices]
# Test
np.random.seed(42)
embeddings = np.random.randn(100, 50)
embeddings = embeddings / np.linalg.norm(embeddings, axis=1, keepdims=True)
similar_words, sims = compute_word_similarities(embeddings, word_idx=5, top_k=5)
assert len(similar_words) == 5, "Top-5 returned"
assert all(0 <= s <= 1 for s in sims), "Similarities in [0,1]"
print("✓ Word similarity working")
if __name__ == "__main__":
print("Lab 4: WordSimilarity - PASSED")