Elmo - Embeddings from Language Models

# ELMo - Embeddings from Language Models

## Introduction & Motivation

ELMo: contextual word representations from bidirectional LSTMs. Context-dependent embeddings. Applications: pre-trained representations, transfer learning.

Motivation: Learn context-dependent word representations.

Applications: Improved downstream task performance via pre-trained embeddings.

---

## Core Concepts & Theory

### Bidirectional LSTM

Language model from both directions.

### Contextual Embeddings

Context-dependent word vectors.

### Layer-wise Embeddings

Use intermediate layers, not just output.

### Transfer Learning

Pre-trained representations for tasks.

---

## Mathematical Formulation

Bidirectional Language Model:
$$P(t_1, ..., t_n) = \prod_k P(t_k | t_1, ..., t_{k-1}) \cdot P(t_k | t_{k+1}, ..., t_n)$$

ELMo Representation:
$$ ext{ELMo}_k^{task} = \gamma^{task} \sum_{j=0}^L s_j^{task} h_{k,j}^{LM}$$

Task-Specific Weighting:
$$s_j^{task} = ext{softmax}(w_j^{task})$$

---

## Advanced Theory & Extensions

### Deep Contextualized Representations

Multiple LSTM layers.

### Task-Specific Weighting

Learnable layer combination.

### Character CNNs

Subword feature extraction.

---

## Computational Considerations

BiLSTM forward: O(T·H²).

Backward pass: O(T·H²).

Weighting: O(L·H).

---

## Practical Implementation Strategies

### Pre-trained Weights

Reuse language model weights.

### Layer Combination

Mix layers for downstream tasks.

### Fine-tuning Strategy

Task adaptation with ELMo.

---

## Benchmark Datasets & Evaluation

GLUE: General understanding.

SQuAD: Question answering.

NER: Named entity recognition.

---

## Key Challenges & Limitations

### Computational Cost

BiLSTM processing expensive.

### Model Size

Large pre-trained models.

### Limited Context

Fixed-window language model.

---

## Hyperparameter Tuning

Hidden dimension: 512-2048.

Number of layers: 1-3.

Learning rate: 1e-4 to 1e-3.

---

## Real-World Applications & Case Studies

NER: Named entity recognition.

Sentiment Analysis: Text classification.

Machine Translation: Sequence-to-sequence tasks.

---

## Integration with Other Methods

ELMo + task-specific models; + attention mechanisms.

---

## Summary & Key Takeaways

ELMo provides contextual word embeddings via bidirectional language models.

Principles:
1. Bidirectional: Full context utilization.
2. Contextual: Position-dependent representations.
3. Multi-layer: Hierarchical features.
4. Transfer learning: Pre-trained reuse.
5. Task-specific: Learnable layer weighting.

---

## Appendix: Practical Labs

### Lab 1: BiLSTM Forward Pass

import numpy as np

def bilstm_forward(inputs, weights_forward, weights_backward):
 """Simplified BiLSTM forward pass"""
 seq_len = inputs.shape[0]
 
 # Forward LSTM
 forward_states = []
 h_f = np.zeros(weights_forward[0].shape[1])
 for t in range(seq_len):
 h_f = inputs[t] @ weights_forward[0] + h_f @ weights_forward[1]
 h_f = np.tanh(h_f)
 forward_states.append(h_f)
 
 # Backward LSTM
 backward_states = []
 h_b = np.zeros(weights_backward[0].shape[1])
 for t in range(seq_len-1, -1, -1):
 h_b = inputs[t] @ weights_backward[0] + h_b @ weights_backward[1]
 h_b = np.tanh(h_b)
 backward_states.insert(0, h_b)
 
 bilstm_out = np.array([np.concatenate([f, b]) for f, b in zip(forward_states, backward_states)])
 return bilstm_out

np.random.seed(42)
inputs = np.random.randn(10, 100)
w_f = [np.random.randn(100, 128) * 0.01, np.random.randn(128, 128) * 0.01]
w_b = [np.random.randn(100, 128) * 0.01, np.random.randn(128, 128) * 0.01]
output = bilstm_forward(inputs, w_f, w_b)
assert output.shape == (10, 256), "Correct BiLSTM output"
print("✓ BiLSTM forward working")

### Lab 2: Layer Weighting

import numpy as np

def task_specific_elmo_weight(layer_states, weights):
 """Weight layers for task-specific ELMo"""
 # Softmax over weights
 normalized = np.exp(weights) / np.sum(np.exp(weights))
 
 # Weighted combination
 elmo = np.sum([s * w for s, w in zip(layer_states, normalized)], axis=0)
 return elmo

np.random.seed(42)
layers = [np.random.randn(10, 256) for _ in range(3)]
w = np.array([0.3, 0.5, 0.2])
elmo = task_specific_elmo_weight(layers, w)
assert elmo.shape == (10, 256), "Correct ELMo shape"
print("✓ Layer weighting working")

### Lab 3: Character CNN

import numpy as np

def character_cnn_embeddings(word, embedding_dim=50):
 """Extract character-level features via CNN"""
 # Simplified: average character embeddings
 char_embeds = np.random.randn(len(word), 32)
 
 # Max pooling over characters
 pooled = np.max(char_embeds, axis=0)
 
 # Project to embedding dimension
 W = np.random.randn(32, embedding_dim) * 0.01
 embedding = pooled @ W
 
 return embedding

np.random.seed(42)
word = "hello"
embedding = character_cnn_embeddings(word, embedding_dim=256)
assert embedding.shape == (256,), "Correct embedding shape"
print("✓ Character CNN working")

### Lab 4: ELMo Contextualization

import numpy as np

def contextualize_embeddings(static_embedding, context_vectors, context_weight=0.5):
 """Combine static and contextual embeddings"""
 # Context-dependent adjustment
 context_contribution = context_weight * context_vectors.mean(axis=0)
 contextualized = static_embedding + context_contribution
 
 return contextualized

np.random.seed(42)
static = np.random.randn(256)
context = np.random.randn(10, 256)
contextualized = contextualize_embeddings(static, context)
assert contextualized.shape == (256,), "Correct contextualized shape"
print("✓ ELMo contextualization working")

---