transformer architecture attention is all you need
# Transformer Architecture: Attention is All You Need
## Introduction & Motivation
Transformers: encoder-decoder architecture based entirely on attention. No recurrence; fully parallelizable. Scales to large datasets/models. Applications: NLP (BERT, GPT), vision (ViT), multimodal.
Motivation: RNNs sequential; bottleneck for parallelization. Transformers parallel; efficient training/inference.
Applications: Machine translation, language modeling, summarization, question answering.
---
## Core Concepts & Theory
### Encoder Stack
Multi-head self-attention + feed-forward; layer normalization.
### Decoder Stack
Masked self-attention + encoder-decoder attention + feed-forward.
### Positional Encoding
Sinusoidal position embeddings; captures sequence order.
---
## Mathematical Formulation
Positional encoding:
$$PE_{(pos, 2i)} = \sin(pos / 10000^{2i/d_{ ext{model}}})$$
$$PE_{(pos, 2i+1)} = \cos(pos / 10000^{2i/d_{ ext{model}}})$$
Transformer layer:
$$ ext{Output} = ext{FFN}( ext{LayerNorm}( ext{MultiHeadAttn}( ext{LayerNorm}(X))))$$
---
## Advanced Theory & Extensions
### Relative Position Bias
Bias attention by relative distances; improves generalization.
### Sandwich Norm (Pre-Norm)
Normalize before layer (vs after); improved stability.
### Sparse Attention
Sparse patterns reduce O(n²) complexity; Longformer, Linformer.
---
## Computational Considerations
Attention: O(n² d).
FFN: O(n d²).
Memory: O(n²) for attention weights.
---
## Practical Implementation Strategies
### Warmup Learning Rate
Gradual increase early; prevents instability.
### Gradient Clipping
Clip norms; prevent gradient explosion.
### Label Smoothing
Smooth targets (0.9 / 0.1 instead of 1.0 / 0.0); regularization.
---
## Benchmark Datasets & Evaluation
WMT14 (Translation): 28.4 BLEU (Transformer base).
COCO Captioning: CIDEr 113 (Transformer decoder).
SQuAD (QA): EM 88.5%.
---
## Key Challenges & Limitations
### Training Instability
Large gradients early; requires careful initialization.
### Memory Constraints
Attention O(n²); limits sequence length.
### Positional Encoding
Extrapolation beyond training length; challenging.
---
## Hyperparameter Tuning
d_model: 512 (base), 768-1024 (larger).
num_layers: 6-12.
num_heads: 8-16.
d_ff: 2048 (4× d_model).
---
## Real-World Applications & Case Studies
Machine Translation: WMT14 state-of-the-art.
Language Modeling: Basis for BERT, GPT.
Summarization: Abstractive via seq2seq transformer.
---
## Integration with Other Methods
Transformer + Copy Mechanism → pointer-generator networks.
Transformer + Retrieval → retrieval-augmented generation.
---
## Summary & Key Takeaways
Transformers via stacked attention enable fully parallel, efficient sequence-to-sequence modeling without recurrence.
Principles:
1. Encoder-decoder: independent stacks with cross-attention.
2. Multi-head self-attention: diverse representation learning.
3. Positional encoding: capture sequence order.
4. Residual connections + layer norm: stable deep networks.
5. Feed-forward: non-linear transformations per position.
---
---
## Appendix: Practical Labs
### Lab 1: Positional Encoding
import torch
import numpy as np
def positional_encoding(seq_len, d_model):
"""Generate sinusoidal positional encodings"""
pos = np.arange(seq_len).reshape(-1, 1)
i = np.arange(d_model).reshape(1, -1)
angle_rates = 1 / np.power(10000, (2 * (i // 2)) / np.float32(d_model))
pe = np.zeros((seq_len, d_model))
pe[:, 0::2] = np.sin(pos * angle_rates[:, 0::2])
pe[:, 1::2] = np.cos(pos * angle_rates[:, 1::2])
return torch.from_numpy(pe).unsqueeze(0).float()
# Test
pe = positional_encoding(seq_len=100, d_model=512)
print(f"PE shape: {pe.shape}")
assert pe.shape == (1, 100, 512), "Should match seq_len and d_model"
assert (-1 <= pe).all() and (pe <= 1).all(), "Values should be normalized"
print("✓ Positional encoding working")
if __name__ == "__main__":
print("Lab 1: PE - PASSED")### Lab 2: Transformer Encoder Layer
import torch
import torch.nn as nn
class TransformerEncoderLayer(nn.Module):
def __init__(self, d_model=512, num_heads=8, d_ff=2048):
super().__init__()
self.mha = nn.MultiheadAttention(d_model, num_heads, batch_first=True)
self.ffn = nn.Sequential(
nn.Linear(d_model, d_ff),
nn.ReLU(),
nn.Linear(d_ff, d_model)
)
self.ln1 = nn.LayerNorm(d_model)
self.ln2 = nn.LayerNorm(d_model)
def forward(self, x):
# Self-attention
attn_out, _ = self.mha(x, x, x)
x = x + attn_out # Residual
x = self.ln1(x)
# Feed-forward
ffn_out = self.ffn(x)
x = x + ffn_out # Residual
x = self.ln2(x)
return x
# Test
layer = TransformerEncoderLayer(d_model=512, num_heads=8, d_ff=2048)
x = torch.randn(2, 10, 512) # [batch, seq_len, d_model]
output = layer(x)
print(f"Encoder layer output shape: {output.shape}")
assert output.shape == x.shape, "Should preserve shape"
print("✓ Encoder layer working")
if __name__ == "__main__":
print("Lab 2: Encoder Layer - PASSED")### Lab 3: Full Transformer
import torch
import torch.nn as nn
class Transformer(nn.Module):
def __init__(self, vocab_size=10000, d_model=512, num_layers=6, num_heads=8, d_ff=2048):
super().__init__()
self.embedding = nn.Embedding(vocab_size, d_model)
self.encoder_layers = nn.ModuleList([
TransformerEncoderLayer(d_model, num_heads, d_ff) for _ in range(num_layers)
])
self.fc_out = nn.Linear(d_model, vocab_size)
def forward(self, x):
# Embedding
x = self.embedding(x)
# Encoder stack
for layer in self.encoder_layers:
x = layer(x)
# Output projection
out = self.fc_out(x)
return out
class TransformerEncoderLayer(nn.Module):
def __init__(self, d_model, num_heads, d_ff):
super().__init__()
self.mha = nn.MultiheadAttention(d_model, num_heads, batch_first=True)
self.ffn = nn.Sequential(nn.Linear(d_model, d_ff), nn.ReLU(), nn.Linear(d_ff, d_model))
self.ln1 = nn.LayerNorm(d_model)
self.ln2 = nn.LayerNorm(d_model)
def forward(self, x):
attn_out, _ = self.mha(x, x, x)
x = x + attn_out
x = self.ln1(x)
ffn_out = self.ffn(x)
x = x + ffn_out
x = self.ln2(x)
return x
# Test
model = Transformer(vocab_size=1000, d_model=256, num_layers=3, num_heads=8)
x = torch.randint(0, 1000, (4, 20)) # [batch, seq_len]
output = model(x)
print(f"Transformer output shape: {output.shape}")
assert output.shape == (4, 20, 1000), "Should output logits for each token"
print("✓ Transformer working")
if __name__ == "__main__":
print("Lab 3: Transformer - PASSED")### Lab 4: Layer Normalization Effect
import torch
import torch.nn as nn
import numpy as np
def compare_with_without_ln(x, layer):
"""Compare outputs with and without layer norm"""
# With layer norm
ln = nn.LayerNorm(x.size(-1))
x_norm = ln(x)
out_norm = layer(x_norm)
# Without layer norm
out_plain = layer(x)
return out_norm, out_plain
# Test
x = torch.randn(4, 10, 512)
layer = nn.Linear(512, 512)
out_norm, out_plain = compare_with_without_ln(x, layer)
# Check stability (variance)
var_norm = out_norm.var(dim=-1).mean()
var_plain = out_plain.var(dim=-1).mean()
print(f"Variance with LN: {var_norm:.4f}, Without LN: {var_plain:.4f}")
assert var_norm < var_plain or np.isclose(var_norm.item(), var_plain.item()), "LN should stabilize"
print("✓ Layer normalization comparison working")
if __name__ == "__main__":
print("Lab 4: LN Comparison - PASSED")