Attention Mechanisms Transformers Self-Attention Parallelization
# Attention Mechanisms & Transformers: Self-Attention & Parallelization
## Introduction & Motivation
Attention weights relevant input elements for each output; Transformers stack multi-head self-attention and feed-forward layers. Parallelizable unlike RNNs; captures long-range dependencies efficiently. Foundation for BERT, GPT, T5.
Motivation: RNN sequential—slow. Attention allows direct information flow between distant steps. Transformers scale to billions of parameters; enable pre-training on massive corpora.
Applications: Machine translation, language modeling, question answering, vision transformers, multi-modal models.
---
## Core Concepts & Theory
### Self-Attention
Query (Q), Key (K), Value (V) projections. Attention weights = softmax(QK^T/√d_k).
Output = attention(Q,K,V) = softmax(QK^T/√d_k)V.
### Multi-Head Attention
Parallel attention heads capture different representations; concatenate and project.
---
## Mathematical Formulation
Scaled dot-product attention:
$$ ext{Attention}(Q,K,V) = ext{softmax}\left(\frac{QK^T}{\sqrt{d_k}}
ight)V$$
Multi-head:
$$ ext{MultiHead}(Q,K,V) = ext{Concat}( ext{head}_1,..., ext{head}_h)W^O$$
---
## Advanced Theory & Extensions
### Positional Encoding
Sine/cosine positional embeddings encode position; necessary for sequence order.
### Cross-Attention
Query from one sequence, Keys/Values from another; enables encoder-decoder models.
---
## Computational Considerations
Complexity: O(L^2 imes d) for sequence length L, embedding dim d. Quadratic in L; problematic for long sequences.
---
## Practical Implementation Strategies
### Attention Dropout
Prevent attention overfitting; typical 0.1-0.2.
### Layer Normalization
Applied before/after attention and feed-forward.
### Position Embeddings
Learnable or fixed sinusoidal; both effective.
---
## Benchmark Datasets & Evaluation
GLUE: Language understanding benchmark.
SQuAD: Reading comprehension.
---
## Key Challenges & Limitations
### Quadratic Complexity
Long documents expensive. Solutions: sparse attention, linear approximations.
### Position Generalization
Extrapolate beyond training sequence length.
---
## Hyperparameter Tuning
num_heads \in {8, 12, 16\}, d_model \in {512, 768, 1024\}, dropout \in {0.1, 0.2\}.
---
## Real-World Applications & Case Studies
GPT-3: Generative model via Transformer.
BERT: Bidirectional encoder via Masked Language Modeling.
---
## Integration with Other Methods
Transformer + Vision → Vision Transformer (ViT).
---
## Summary & Key Takeaways
Transformers enable parallelizable, long-range-dependent sequence modeling via multi-head self-attention, fundamentally changing NLP and enabling scale.
Principles:
1. Self-attention weights sequence elements.
2. Multi-head captures diverse relationships.
3. Feed-forward layers add non-linearity.
4. Positional encoding preserves order.
5. Parallelizable unlike RNNs.
---
---
## Appendix: Practical Labs
### Lab 1: Self-Attention
import torch
import torch.nn as nn
batch_size, seq_len, d_model = 2, 4, 64
Q = torch.randn(batch_size, seq_len, d_model)
K = torch.randn(batch_size, seq_len, d_model)
V = torch.randn(batch_size, seq_len, d_model)
# Scaled dot-product attention
scores = torch.matmul(Q, K.transpose(-2, -1)) / (d_model ** 0.5)
attn_weights = torch.softmax(scores, dim=-1)
output = torch.matmul(attn_weights, V)
print(f"Attention output shape: {output.shape}")
assert output.shape == (batch_size, seq_len, d_model), "Correct shape"
print("✓ Self-attention working")
if __name__ == "__main__":
print("Lab 1: Self-Attention - PASSED")### Lab 2: Multi-Head Attention
import torch
import torch.nn as nn
batch_size, seq_len, d_model, num_heads = 2, 4, 64, 8
X = torch.randn(batch_size, seq_len, d_model)
mha = nn.MultiheadAttention(embed_dim=d_model, num_heads=num_heads, batch_first=True)
output, weights = mha(X, X, X)
print(f"Multi-head output shape: {output.shape}")
assert output.shape == (batch_size, seq_len, d_model), "Correct shape"
print("✓ Multi-head attention working")
if __name__ == "__main__":
print("Lab 2: Multi-Head Attention - PASSED")### Lab 3: Positional Encoding
import torch
import numpy as np
seq_len, d_model = 10, 64
# Sinusoidal positional encoding
position = np.arange(seq_len)[:, np.newaxis]
div_term = np.exp(np.arange(0, d_model, 2) * -(np.log(10000.0) / d_model))
pe = np.zeros((seq_len, d_model))
pe[:, 0::2] = np.sin(position * div_term)
pe[:, 1::2] = np.cos(position * div_term)
pe_tensor = torch.FloatTensor(pe)
print(f"Positional encoding shape: {pe_tensor.shape}")
assert pe_tensor.shape == (seq_len, d_model), "Correct shape"
print("✓ Positional encoding working")
if __name__ == "__main__":
print("Lab 3: Positional Encoding - PASSED")### Lab 4: Transformer Block
import torch
import torch.nn as nn
class TransformerBlock(nn.Module):
def __init__(self, d_model=64, num_heads=8, d_ff=256):
super().__init__()
self.attn = nn.MultiheadAttention(d_model, num_heads, batch_first=True)
self.norm1 = nn.LayerNorm(d_model)
self.ff = nn.Sequential(nn.Linear(d_model, d_ff), nn.ReLU(), nn.Linear(d_ff, d_model))
self.norm2 = nn.LayerNorm(d_model)
def forward(self, x):
attn_out, _ = self.attn(x, x, x)
x = self.norm1(x + attn_out)
ff_out = self.ff(x)
x = self.norm2(x + ff_out)
return x
model = TransformerBlock()
X = torch.randn(2, 4, 64)
out = model(X)
print(f"Transformer block output shape: {out.shape}")
assert out.shape == (2, 4, 64), "Correct shape"
print("✓ Transformer block working")
if __name__ == "__main__":
print("Lab 4: Transformer Block - PASSED")