BERT Transformer Language Models Pretraining
# BERT: Transformer Language Models & Pretraining
## Introduction & Motivation
BERT: Bidirectional Encoder Representations from Transformers. Pretraining on masked language modeling. Fine-tuning for downstream tasks. Applications: text classification, NER, question answering, semantic similarity.
Motivation: Transfer learning for NLP; bidirectional context.
Applications: Classification, NER, semantic tasks.
---
## Core Concepts & Theory
### Masked Language Model
Predict masked tokens; bidirectional.
### Next Sentence Prediction
Sentence relationship prediction task.
### Fine-tuning
Task-specific adaptation.
---
## Mathematical Formulation
MLM objective:
$$L_{ ext{MLM}} = -\mathbb{E}_{t \in ext{masked}} [\log P(x_t | x_{\backslash t})]$$
NSP objective (deprecated in RoBERTa):
$$L_{ ext{NSP}} = -\log P( ext{IsNext} | [CLS] s_1 [SEP] s_2)$$
Fine-tuning loss:
$$L = L_{ ext{task}} + \lambda L_{ ext{reg}}$$
---
## Advanced Theory & Extensions
### RoBERTa
Improved pretraining; better results.
### ALBERT
Parameter-efficient; factorization.
### Domain-Specific BERT
SciBERT, BioBERT, FinBERT.
---
## Computational Considerations
Pretraining: O(vocabulary·sequence_length·layers) per iteration.
Fine-tuning: O(batch_size·sequence_length·layers).
Inference: O(sequence_length·layers).
---
## Practical Implementation Strategies
### Tokenization
WordPiece tokenizer; subword units.
### Special Tokens
[CLS], [SEP], [PAD], [UNK].
### Learning Rates
Lower for fine-tuning; e.g., 2e-5.
---
## Benchmark Datasets & Evaluation
GLUE: General language understanding.
SuperGLUE: Challenging benchmark.
SQuAD: Question answering.
---
## Key Challenges & Limitations
### Computational Cost
Large models; expensive pretraining.
### Tokenization Mismatch
Subword splits; information loss.
### Domain Adaptation
Limited by pretraining domain.
---
## Hyperparameter Tuning
Learning rate: 2e-5 to 5e-5 for fine-tuning.
Batch size: 16-32; memory constraints.
Epochs: 2-4 for fine-tuning.
---
## Real-World Applications & Case Studies
Text Classification: Sentiment, topic.
Named Entity Recognition: BERT-BiLSTM-CRF.
Question Answering: SQuAD-style tasks.
---
## Integration with Other Methods
BERT + CRF → sequence tagging.
BERT + Attention → information extraction.
---
## Summary & Key Takeaways
BERT via bidirectional pretraining enables powerful transfer learning for NLP through masked language modeling and fine-tuning.
Principles:
1. Bidirectional: masked LM context.
2. Pretraining: self-supervised learning.
3. Fine-tuning: task adaptation.
4. Tokenization: subword units.
5. Transfer: downstream effectiveness.
---
---
## Appendix: Practical Labs
### Lab 1: Tokenization
import numpy as np
def simple_wordpiece_tokenize(text, vocab_size=1000):
"""Simple WordPiece tokenization simulation"""
# Basic split
tokens = text.lower().split()
# WordPiece: simulate with dummy IDs
token_ids = []
for token in tokens:
# In reality, use WordPiece vocab
token_id = hash(token) % vocab_size
token_ids.append(token_id)
return token_ids
# Test
text = "the quick brown fox jumps"
token_ids = simple_wordpiece_tokenize(text)
assert len(token_ids) == 5, "Token count"
assert all(0 <= id < 1000 for id in token_ids), "Valid IDs"
print("✓ Tokenization working")
if __name__ == "__main__":
print("Lab 1: Tokenization - PASSED")### Lab 2: MLM Loss
import numpy as np
def masked_language_model_loss(logits, targets, mask):
"""Compute MLM loss"""
# Softmax probabilities
exp_logits = np.exp(logits - np.max(logits, axis=1, keepdims=True))
probs = exp_logits / exp_logits.sum(axis=1, keepdims=True)
# Loss: only masked positions
batch_size = len(logits)
loss = 0
count = 0
for i in range(batch_size):
if mask[i]:
loss -= np.log(probs[i, targets[i]] + 1e-8)
count += 1
return loss / (count + 1e-8) if count > 0 else 0
# Test
np.random.seed(42)
logits = np.random.randn(32, 1000) # 32 batch, 1000 vocab
targets = np.random.randint(0, 1000, 32)
mask = np.random.rand(32) > 0.5 # 50% masked
loss = masked_language_model_loss(logits, targets, mask)
assert np.isfinite(loss), "Loss finite"
print("✓ MLM loss working")
if __name__ == "__main__":
print("Lab 2: MLMLoss - PASSED")### Lab 3: Attention Visualization
import numpy as np
def compute_attention_weights(Q, K, V):
"""Compute scaled dot-product attention"""
# Scaled dot-product
scores = Q @ K.T / np.sqrt(K.shape[1])
# Softmax
exp_scores = np.exp(scores - np.max(scores, axis=1, keepdims=True))
attention_weights = exp_scores / exp_scores.sum(axis=1, keepdims=True)
# Apply to values
output = attention_weights @ V
return output, attention_weights
# Test
np.random.seed(42)
seq_len, d_model = 10, 64
Q = np.random.randn(seq_len, d_model)
K = np.random.randn(seq_len, d_model)
V = np.random.randn(seq_len, d_model)
output, weights = compute_attention_weights(Q, K, V)
assert output.shape == V.shape, "Output shape"
assert weights.shape == (seq_len, seq_len), "Attention shape"
assert np.allclose(weights.sum(axis=1), 1.0), "Attention sums to 1"
print("✓ Attention working")
if __name__ == "__main__":
print("Lab 3: Attention - PASSED")### Lab 4: Fine-tuning Loss
import numpy as np
def fine_tuning_loss(logits, targets, l2_weight=0.01):
"""Compute fine-tuning loss with L2 regularization"""
# Task loss (cross-entropy)
exp_logits = np.exp(logits - np.max(logits, axis=1, keepdims=True))
probs = exp_logits / exp_logits.sum(axis=1, keepdims=True)
batch_size = len(logits)
task_loss = -np.log(probs[np.arange(batch_size), targets] + 1e-8).mean()
# L2 regularization (simulated weights)
weights = np.random.randn(logits.shape[1])
reg_loss = l2_weight * (weights ** 2).mean()
total_loss = task_loss + reg_loss
return total_loss
# Test
np.random.seed(42)
logits = np.random.randn(32, 10) # 32 batch, 10 classes
targets = np.random.randint(0, 10, 32)
loss = fine_tuning_loss(logits, targets)
assert np.isfinite(loss), "Loss finite"
assert loss > 0, "Loss positive"
print("✓ Fine-tuning loss working")
if __name__ == "__main__":
print("Lab 4: FineTuningLoss - PASSED")