Transformer Encoder-Decoder Architecture
# Transformer: Encoder-Decoder Architecture
## Introduction & Motivation
Transformer: self-attention based sequence transduction. Encoder: process input sequence. Decoder: autoregressive output generation. Encoder-decoder: source-target alignment. Applications: machine translation, image captioning, summarization, speech recognition.
Motivation: RNNs sequential; slow training. Transformers parallel; enable large-scale pretraining.
Applications: NLP, vision-language, speech.
---
## Core Concepts & Theory
### Encoder Stack
Self-attention + feed-forward; multiple layers.
### Decoder Stack
Self-attention + cross-attention + feed-forward.
### Cross-Attention
Decoder attends to encoder; information flow.
---
## Mathematical Formulation
Encoder layer:
$$ ext{FFN}( ext{Attention}(X)) = ext{MaxOut}(W_2 ext{ReLU}(W_1 X + b_1) + b_2) + X$$
Decoder layer:
$$ ext{FFN}( ext{CrossAttn}( ext{SelfAttn}(Y), X)) = \ldots$$
where X = encoder output, Y = decoder input.
---
## Advanced Theory & Extensions
### Layer Normalization
Normalize per token; better training stability.
### Residual Connections
Skip connections; gradient flow.
### Feed-Forward Network
Two dense layers; non-linearity per position.
---
## Computational Considerations
Encoder: O(L · n · d²) where L = layers, n = seq length, d = dimension.
Decoder inference: Autoregressive; O(n) tokens generated sequentially.
Training: Parallel; O(1) per time step.
---
## Practical Implementation Strategies
### Initialization
Layer-dependent; careful Xavier/He.
### Warmup Schedule
Prevent instability; linear warmup crucial.
### Gradient Checkpointing
Trade compute for memory; enable larger models.
---
## Benchmark Datasets & Evaluation
Machine Translation: WMT; BLEU/METEOR.
Image Captioning: COCO; CIDEr, SPICE.
Summarization: CNN/DailyMail; ROUGE.
---
## Key Challenges & Limitations
### Inference Speed
Autoregressive decoding slow; beam search overhead.
### Hallucination
Generate plausible but false content.
### Context Window
Limited by O(n²) attention; long documents truncated.
---
## Hyperparameter Tuning
Encoder layers: 6-12; model capacity.
Decoder layers: 6-12; output complexity.
FFN hidden: 2048-4096; expansion factor 4.
---
## Real-World Applications & Case Studies
Neural Machine Translation: Google Translate; en-de standard.
Text Summarization: Document → concise summary.
Visual Captioning: Image → natural description.
---
## Integration with Other Methods
Transformer + CNN → hybrid; visual feature extraction.
Transformer + GNN → structured inputs; graph sequences.
---
## Summary & Key Takeaways
Transformer encoder-decoder enables parallel sequence-to-sequence modeling with self-attention and cross-attention for information flow.
Principles:
1. Encoder: self-attention over input.
2. Decoder: masked self-attention over output.
3. Cross-attention: encoder-decoder interaction.
4. Residuals: gradient flow.
5. Parallelism: training efficiency.
---
---
## Appendix: Practical Labs
### Lab 1: Encoder Layer
import numpy as np
class EncoderLayer:
def __init__(self, d_model=512, num_heads=8, d_ff=2048):
self.d_model = d_model
self.num_heads = num_heads
self.d_ff = d_ff
def forward(self, x):
"""Encoder: self-attention + FFN"""
# Self-attention (simplified)
d_k = self.d_model // self.num_heads
scores = np.matmul(x, x.T) / np.sqrt(d_k)
weights = np.exp(scores - np.max(scores, axis=-1, keepdims=True))
weights = weights / weights.sum(axis=-1, keepdims=True)
attended = np.matmul(weights, x)
# Residual
x = x + attended
# FFN
hidden = np.maximum(0, np.dot(x, np.random.randn(self.d_model, self.d_ff)))
output = np.dot(hidden, np.random.randn(self.d_ff, self.d_model))
# Residual
x = x + output
return x
# Test
np.random.seed(42)
layer = EncoderLayer(d_model=8, num_heads=2, d_ff=32)
x = np.random.randn(4, 8) # 4 tokens, dim 8
output = layer.forward(x)
assert output.shape == (4, 8), "Shape preserved"
print("✓ Encoder layer working")
if __name__ == "__main__":
print("Lab 1: EncoderLayer - PASSED")### Lab 2: Decoder Layer
import numpy as np
class DecoderLayer:
def __init__(self, d_model=512, num_heads=8):
self.d_model = d_model
self.num_heads = num_heads
def forward(self, tgt, src):
"""Decoder: masked self-attn + cross-attn + FFN"""
d_k = self.d_model // self.num_heads
# Masked self-attention
tgt_scores = np.matmul(tgt, tgt.T) / np.sqrt(d_k)
mask = np.tril(np.ones_like(tgt_scores))
tgt_scores = np.where(mask, tgt_scores, -1e9)
tgt_weights = np.exp(tgt_scores - np.max(tgt_scores, axis=-1, keepdims=True))
tgt_weights = tgt_weights / tgt_weights.sum(axis=-1, keepdims=True)
tgt_attended = np.matmul(tgt_weights, tgt)
tgt = tgt + tgt_attended
# Cross-attention (tgt attends to src)
cross_scores = np.matmul(tgt, src.T) / np.sqrt(d_k)
cross_weights = np.exp(cross_scores - np.max(cross_scores, axis=-1, keepdims=True))
cross_weights = cross_weights / cross_weights.sum(axis=-1, keepdims=True)
cross_attended = np.matmul(cross_weights, src)
tgt = tgt + cross_attended
return tgt
# Test
np.random.seed(42)
layer = DecoderLayer(d_model=8, num_heads=2)
tgt = np.random.randn(3, 8) # 3 target tokens
src = np.random.randn(4, 8) # 4 source tokens
output = layer.forward(tgt, src)
assert output.shape == (3, 8), "Output shape"
print("✓ Decoder layer working")
if __name__ == "__main__":
print("Lab 2: DecoderLayer - PASSED")### Lab 3: Encoder-Decoder Forward Pass
import numpy as np
def encoder_decoder_forward(src, tgt, num_layers=2):
"""Simplified encoder-decoder forward"""
# Encoder
x = src
for _ in range(num_layers):
d_k = x.shape[-1]
# Self-attention
scores = np.matmul(x, x.T) / np.sqrt(d_k)
weights = np.exp(scores - np.max(scores, axis=-1, keepdims=True))
weights = weights / weights.sum(axis=-1, keepdims=True)
x = np.matmul(weights, x)
enc_output = x
# Decoder
y = tgt
for _ in range(num_layers):
d_k = y.shape[-1]
# Masked self-attention
scores = np.matmul(y, y.T) / np.sqrt(d_k)
mask = np.tril(np.ones_like(scores))
scores = np.where(mask, scores, -1e9)
weights = np.exp(scores - np.max(scores, axis=-1, keepdims=True))
weights = weights / weights.sum(axis=-1, keepdims=True)
y = np.matmul(weights, y)
# Cross-attention
c_scores = np.matmul(y, enc_output.T) / np.sqrt(d_k)
c_weights = np.exp(c_scores - np.max(c_scores, axis=-1, keepdims=True))
c_weights = c_weights / c_weights.sum(axis=-1, keepdims=True)
y = np.matmul(c_weights, enc_output)
return y
# Test
np.random.seed(42)
src = np.random.randn(5, 8)
tgt = np.random.randn(4, 8)
output = encoder_decoder_forward(src, tgt, num_layers=2)
assert output.shape == (4, 8), "Output shape"
print("✓ Encoder-decoder forward working")
if __name__ == "__main__":
print("Lab 3: EncoderDecoderForward - PASSED")### Lab 4: Beam Search Decoding
import numpy as np
def beam_search_decode(encoder_output, max_length=10, beam_width=3):
"""Simplified beam search decoding"""
sequences = [[]]
scores = [0.0]
for _ in range(max_length):
all_candidates = []
for i, seq in enumerate(sequences):
# Predict next token (simplified)
logits = np.random.randn(10) # 10 vocabulary
probs = np.exp(logits) / np.exp(logits).sum()
# Top-k candidates
top_indices = np.argsort(-probs)[:beam_width]
for idx in top_indices:
new_seq = seq + [idx]
new_score = scores[i] + np.log(probs[idx])
all_candidates.append((new_score, new_seq))
# Keep top beam_width
all_candidates.sort(reverse=True)
sequences = [seq for _, seq in all_candidates[:beam_width]]
scores = [score for score, _ in all_candidates[:beam_width]]
return sequences[0], scores[0]
# Test
np.random.seed(42)
enc_out = np.random.randn(5, 8)
seq, score = beam_search_decode(enc_out, max_length=10, beam_width=3)
assert isinstance(seq, list), "Sequence is list"
assert np.isfinite(score), "Score finite"
print("✓ Beam search working")
if __name__ == "__main__":
print("Lab 4: BeamSearch - PASSED")