BERT - Bidirectional Encoder Representations from Transformers

# BERT - Bidirectional Encoder Representations from Transformers

## Introduction & Motivation

BERT: bidirectional pre-training via masked language modeling. Contextual word representations. Applications: NLP tasks, text understanding.

Motivation: Learn bidirectional context for language representation.

Applications: Text classification, named entity recognition, question answering.

---

## Core Concepts & Theory

### Masked Language Modeling

Predict masked tokens from context.

### Next Sentence Prediction

Sentence-level relationships.

### Bidirectional Context

Full left-right context usage.

### Transfer Learning

Fine-tune on downstream tasks.

---

## Mathematical Formulation

Masked LM Loss:
$$\mathcal{L} = -\log p(w_m | ext{context})$$

NSP Loss:
$$\mathcal{L}_{ ext{NSP}} = -\log p( ext{IsNext} | [CLS])$$

Joint Loss:
$$\mathcal{L}_{ ext{total}} = \mathcal{L}_{ ext{MLM}} + \mathcal{L}_{ ext{NSP}}$$

---

## Advanced Theory & Extensions

### RoBERTa

Improved pre-training methodology.

### ALBERT

Parameter-efficient architecture.

### Domain-Specific BERT

Task-adapted pre-training.

---

## Computational Considerations

Pre-training: O(T·D·H·L).

Fine-tuning: O(E·B·T·D).

Inference: O(T·D·H).

---

## Practical Implementation Strategies

### Token Masking

15% random masking strategy.

### Learning Rate Scheduling

Warmup and decay schedules.

### Gradient Accumulation

Effective large batch training.

---

## Benchmark Datasets & Evaluation

GLUE: General Language Understanding.

SuperGLUE: Challenging NLU tasks.

SQuAD: Question answering benchmark.

---

## Key Challenges & Limitations

### Pre-training Cost

Massive computational requirements.

### Task Specificity

Limited cross-domain transfer.

### Fine-tuning Data

Performance with limited labels.

---

## Hyperparameter Tuning

Learning rate: 1e-5 to 5e-5.

Batch size: 16-128.

Epochs: 2-10.

---

## Real-World Applications & Case Studies

Text Classification: Sentiment analysis, topic modeling.

Named Entity Recognition: Information extraction.

Question Answering: Document-based QA.

---

## Integration with Other Methods

BERT + task-specific heads; + multi-task learning for efficiency.

---

## Summary & Key Takeaways

BERT provides bidirectional contextual representations via masked language modeling.

Principles:
1. Masked LM: Cloze task pre-training.
2. Bidirectional: Full context utilization.
3. NSP: Sentence-level understanding.
4. Transfer learning: Task-specific fine-tuning.
5. Scalability: Effective pre-training.

---

## Appendix: Practical Labs

### Lab 1: Token Masking Strategy

import numpy as np

def mask_tokens(token_ids, mask_ratio=0.15):
 """Apply masking strategy for MLM"""
 masked_ids = token_ids.copy()
 mask_positions = np.random.choice(len(token_ids), int(len(token_ids) * mask_ratio), replace=False)
 
 for pos in mask_positions:
 rand = np.random.rand()
 if rand < 0.8:
 masked_ids[pos] = 103 # [MASK]
 elif rand < 0.9:
 masked_ids[pos] = np.random.randint(0, 30522)
 # else: keep original (10%)
 
 return masked_ids, mask_positions

np.random.seed(42)
tokens = np.array([101, 2054, 2003, 1045, 102])
masked, positions = mask_tokens(tokens)
assert len(positions) > 0, "Tokens masked"
print("✓ Token masking working")

### Lab 2: MLM Loss Computation

import numpy as np

def mlm_loss(logits, labels, mask_positions):
 """Compute masked language model loss"""
 batch_size = logits.shape[0]
 loss = 0.0
 
 for i in mask_positions:
 probs = np.exp(logits[i]) / np.sum(np.exp(logits[i]))
 loss -= np.log(probs[labels[i]] + 1e-8)
 
 return loss / len(mask_positions)

np.random.seed(42)
logits = np.random.randn(5, 30522)
labels = np.array([103, 2054, 2003, 1045, 102])
loss = mlm_loss(logits, labels, [0, 1, 2])
assert loss > 0, "Positive loss"
print(f"✓ MLM loss working: {loss:.4f}")

### Lab 3: NSP Loss

import numpy as np

def nsp_loss(cls_output, is_next_label):
 """Compute next sentence prediction loss"""
 probs = 1 / (1 + np.exp(-cls_output))
 loss = -np.mean(is_next_label * np.log(probs + 1e-8) + 
 (1 - is_next_label) * np.log(1 - probs + 1e-8))
 return loss

np.random.seed(42)
cls_output = np.random.randn(32)
labels = np.random.randint(0, 2, 32)
loss = nsp_loss(cls_output, labels)
assert loss > 0, "Positive loss"
print(f"✓ NSP loss working: {loss:.4f}")

### Lab 4: CLS Token Extraction

import numpy as np

def extract_cls_representation(bert_output, sequence_length=512):
 """Extract [CLS] token representation"""
 cls_token = bert_output[:, 0, :]
 return cls_token

np.random.seed(42)
bert_output = np.random.randn(32, 512, 768)
cls_repr = extract_cls_representation(bert_output)
assert cls_repr.shape == (32, 768), "Correct CLS shape"
print("✓ CLS extraction working")

---