Embedding Layers Word Representations

# Embedding Layers & Word Representations

## Introduction & Motivation

Embeddings: map discrete to continuous. Word2Vec, GloVe, FastText. Applications: NLP, language understanding.

Motivation: Learn distributed representations.

Applications: Language modeling, downstream tasks.

---

## Core Concepts & Theory

### Word Embeddings

Learned vector representations.

### Contextual Embeddings

Context-dependent vectors.

### Semantic Similarity

Distance-based meaning.

### Subword Representations

Handling rare words.

---

## Mathematical Formulation

Skip-gram Objective:
$$L = -\log P(w_{ ext{context}} | w_{ ext{target}}) = -\log \frac{e^{v_c \cdot v_t}}{\sum_w e^{v_w \cdot v_t}}$$

GloVe Objective:
$$L = \sum_{i,j} f(X_{ij})(v_i \cdot v_j - \log X_{ij})^2$$

Cosine Similarity:
$$ ext{sim}(u, v) = \frac{u \cdot v}{\|u\| \|v\|}$$

---

## Advanced Theory & Extensions

### Contextualized Embeddings

ELMo, BERT representations.

### Subword Segmentation

BPE, SentencePiece.

### Static vs. Dynamic

Pre-trained vs. fine-tuned.

---

## Computational Considerations

Lookup: O(1).

Training: O(batch·seq_len·emb_dim).

Storage: O(vocab·emb_dim).

---

## Practical Implementation Strategies

### Vocabulary Selection

Frequency-based filtering.

### Embedding Initialization

Random vs. pre-trained.

### Subword Handling

OOV word strategies.

---

## Benchmark Datasets & Evaluation

Word Analogies: Semantic/syntactic.

SimLex-999: Similarity ranking.

RareWord: OOV handling.

---

## Key Challenges & Limitations

### Vocabulary Size

Memory constraints.

### Polysemy

Multiple meanings.

### Rare Words

Limited examples.

---

## Hyperparameter Tuning

Embedding dimension: 100-600.

Window size: 2-10.

Negative samples: 5-25.

---

## Real-World Applications & Case Studies

Language Models: Foundation embeddings.

Recommendation: Collaborative filtering.

Information Retrieval: Semantic search.

---

## Integration with Other Methods

Embeddings + encoders for sequences; + attention for focus.

---

## Summary & Key Takeaways

Embeddings represent discrete tokens as continuous vectors.

Principles:
1. Word vectors: Distributed representations.
2. Contextual: Position-aware embeddings.
3. Similarity: Semantic relatedness.
4. Subwords: Rare word handling.
5. Pre-training: Transfer learning.

---

## Appendix: Practical Labs

### Lab 1: Embedding Lookup

import numpy as np

def embedding_lookup(token_ids, embedding_matrix):
 """Look up embeddings for tokens"""
 return embedding_matrix[token_ids]

np.random.seed(42)
vocab_size = 10000
emb_dim = 300
embedding_matrix = np.random.randn(vocab_size, emb_dim)
token_ids = np.array([1, 5, 10, 100])
embeddings = embedding_lookup(token_ids, embedding_matrix)
assert embeddings.shape == (4, emb_dim), "Correct embedding shape"
print("✓ Embedding lookup working")

### Lab 2: Cosine Similarity

import numpy as np

def cosine_similarity(u, v):
 """Compute cosine similarity"""
 dot_product = np.dot(u, v)
 norm_u = np.linalg.norm(u)
 norm_v = np.linalg.norm(v)
 return dot_product / (norm_u * norm_v + 1e-8)

np.random.seed(42)
u = np.random.randn(300)
v = np.random.randn(300)
sim = cosine_similarity(u, v)
assert -1 <= sim <= 1, "Similarity in valid range"
print("✓ Cosine similarity working")

### Lab 3: Negative Sampling

import numpy as np

def negative_sampling(context_word, num_negatives, vocab_size, freqs):
 """Sample negative words weighted by frequency"""
 probs = freqs / np.sum(freqs)
 negatives = np.random.choice(vocab_size, size=num_negatives, 
 p=probs, replace=False)
 
 while context_word in negatives:
 negatives = np.random.choice(vocab_size, size=num_negatives, 
 p=probs, replace=False)
 
 return negatives

np.random.seed(42)
freqs = np.power(np.arange(1, 1001), 0.75)
negatives = negative_sampling(context_word=5, num_negatives=10, 
 vocab_size=1000, freqs=freqs)
assert len(negatives) == 10, "Correct negative count"
print("✓ Negative sampling working")

### Lab 4: Subword Tokenization

import numpy as np

def character_ngrams(word, n=3):
 """Generate character n-grams for subword"""
 ngrams = []
 for i in range(len(word) - n + 1):
 ngrams.append(word[i:i+n])
 return ngrams

word = "running"
trigrams = character_ngrams(word, n=3)
assert len(trigrams) == len(word) - 2, "Correct n-gram count"
print(f"✓ Character n-grams: {trigrams}")

---