Sequence-to-Sequence Models Encoder-Decoder Seq2seq
# Sequence-to-Sequence Models: Encoder-Decoder Seq2Seq
## Introduction & Motivation
Sequence-to-sequence: map variable-length input to variable-length output. Encoder: compress input into context vector. Decoder: generate output from context. Attention mechanism: focus on relevant input. Applications: machine translation, question answering, summarization, image captioning.
Motivation: Many tasks map sequences to sequences. Encoder-decoder framework generalizes across tasks.
Applications: Machine translation, summarization, dialogue.
---
## Core Concepts & Theory
### Context Vector
Fixed-size representation of input sequence.
### Encoder
Bidirectional RNN/LSTM; processes entire input.
### Decoder
Autoregressive generation from context.
### Beam Search
Multiple hypotheses; find best path.
---
## Mathematical Formulation
Encoder LSTM:
$$h_t = ext{LSTM}(x_t, h_{t-1})$$
Context vector:
$$c = h_T \quad ext{(final hidden state)}$$
Decoder LSTM:
$$h'_t = ext{LSTM}(y_{t-1}, h'_{t-1}, c)$$
Output probability:
$$p(y_t | c, y_{<t}) = ext{softmax}(W h'_t + b)$$
---
## Advanced Theory & Extensions
### Attention-Based Seq2Seq
Compute attention over encoder states.
### Multi-Encoder Architecture
Multiple encoders for different input modalities.
### Beam Search with Length Penalty
Avoid preferring short sequences.
---
## Computational Considerations
Encoder: O(T · d²) where T = sequence length, d = dimension.
Decoder: O(S · d²) where S = output sequence length.
Beam search: O(S · K · V) where K = beam width, V = vocabulary.
---
## Practical Implementation Strategies
### Teacher Forcing
Train with ground truth; avoid compounding error.
### Scheduled Sampling
Gradually switch to model predictions.
### Attention Mechanism
Focus on relevant encoder states.
---
## Benchmark Datasets & Evaluation
Machine Translation: WMT datasets; BLEU score.
Summarization: CNN/DailyMail; ROUGE metric.
Question Answering: SQuAD; exact match and F1.
---
## Key Challenges & Limitations
### Context Bottleneck
Fixed-size vector limits information flow.
### Exposure Bias
Train-test mismatch; teacher forcing vs. generation.
### Inference Latency
Autoregressive generation slow; O(S) tokens.
---
## Hyperparameter Tuning
Encoder layers: 2-4; bidirectional standard.
Decoder layers: 2-4; autoregressive.
Context dimension: 512-1024; model capacity.
---
## Real-World Applications & Case Studies
Google Neural Machine Translation: Production-scale seq2seq.
Abstractive Summarization: Document → concise summary.
Visual Question Answering: Image + question → answer.
---
## Integration with Other Methods
Seq2Seq + Attention → Transformer standard.
Seq2Seq + Copy Mechanism → pointer networks.
---
## Summary & Key Takeaways
Sequence-to-sequence encoder-decoder architectures enable flexible mapping from variable-length inputs to outputs through context vectors and autoregressive generation.
Principles:
1. Encoder: compress input.
2. Context: fixed-size representation.
3. Decoder: autoregressive generation.
4. Attention: focus mechanism.
5. Beam search: best path inference.
---
---
## Appendix: Practical Labs
### Lab 1: Simple Encoder-Decoder
import numpy as np
class SimpleSeq2Seq:
def __init__(self, input_size, hidden_size, output_size):
self.input_size = input_size
self.hidden_size = hidden_size
self.output_size = output_size
# Encoder weights (simplified)
self.W_enc = np.random.randn(hidden_size, input_size + hidden_size)
self.b_enc = np.zeros((hidden_size, 1))
# Decoder weights
self.W_dec = np.random.randn(output_size, hidden_size + hidden_size)
self.b_dec = np.zeros((output_size, 1))
def encode(self, input_seq):
"""Encode input sequence to context"""
h = np.zeros((self.hidden_size, 1))
for x_t in input_seq:
x_t = x_t.reshape(-1, 1)
concat = np.vstack([h, x_t])
h = np.tanh(np.dot(self.W_enc, concat) + self.b_enc)
return h # Context vector
def decode(self, context, seq_length):
"""Decode context to output sequence"""
outputs = []
h = context
for _ in range(seq_length):
# Simplified: concatenate hidden and context
concat = np.vstack([h, context])
logits = np.dot(self.W_dec, concat) + self.b_dec
outputs.append(logits)
# Update hidden (simplified)
h = np.tanh(logits)
return outputs
# Test
np.random.seed(42)
model = SimpleSeq2Seq(input_size=10, hidden_size=20, output_size=10)
input_seq = [np.random.randn(10) for _ in range(5)]
context = model.encode(input_seq)
outputs = model.decode(context, seq_length=5)
assert context.shape == (20, 1), "Context shape"
assert len(outputs) == 5, "Output sequence length"
print("✓ Simple seq2seq working")
if __name__ == "__main__":
print("Lab 1: SimpleSeq2Seq - PASSED")### Lab 2: Teacher Forcing
import numpy as np
def seq2seq_training_step(encoder_input, decoder_input, decoder_target, model):
"""Seq2seq training with teacher forcing"""
# Encode
context = model.encode(encoder_input)
# Decode with teacher forcing (use ground truth targets)
loss = 0
for i, (dec_in, target) in enumerate(zip(decoder_input, decoder_target)):
# Forward pass
concat = np.vstack([context, dec_in.reshape(-1, 1)])
logits = np.dot(np.random.randn(10, 21), concat) # Simplified
# Cross-entropy loss
probs = np.exp(logits) / np.exp(logits).sum()
loss -= np.log(probs[target] + 1e-8)
return loss / len(decoder_target)
# Test
np.random.seed(42)
class SimpleModel:
def encode(self, seq):
return np.random.randn(20, 1)
model = SimpleModel()
encoder_input = [np.random.randn(10) for _ in range(5)]
decoder_input = [np.random.randn(10) for _ in range(5)]
decoder_target = np.array([0, 1, 2, 3, 4])
loss = seq2seq_training_step(encoder_input, decoder_input, decoder_target, model)
assert np.isfinite(loss), "Loss finite"
print("✓ Teacher forcing working")
if __name__ == "__main__":
print("Lab 2: TeacherForcing - PASSED")### Lab 3: Beam Search Decoding
import numpy as np
def beam_search_decode(context, vocab_size=100, beam_width=3, max_length=10):
"""Beam search decoding for seq2seq"""
# Initialize with start token
sequences = [[]]
scores = [0.0]
for step in range(max_length):
all_candidates = []
for i, seq in enumerate(sequences):
# Predict next token (simplified)
logits = np.dot(np.random.randn(vocab_size), context.flatten())
probs = np.exp(logits - np.max(logits)) / np.exp(logits - np.max(logits)).sum()
# Top-k tokens
top_indices = np.argsort(-probs)[:beam_width]
for idx in top_indices:
candidate = seq + [idx]
candidate_score = scores[i] + np.log(probs[idx])
all_candidates.append((candidate_score, candidate))
# Keep top beam_width
all_candidates.sort(reverse=True)
sequences = [seq for _, seq in all_candidates[:beam_width]]
scores = [score for score, _ in all_candidates[:beam_width]]
return sequences[0], scores[0]
# Test
np.random.seed(42)
context = np.random.randn(20, 1)
seq, score = beam_search_decode(context, vocab_size=100, beam_width=3, max_length=10)
assert len(seq) == 10, "Sequence length"
assert np.isfinite(score), "Score finite"
print("✓ Beam search decoding working")
if __name__ == "__main__":
print("Lab 3: BeamSearchDecode - PASSED")### Lab 4: Seq2Seq Evaluation
import numpy as np
def compute_bleu_score(predictions, references, n_gram=4):
"""Simplified BLEU score"""
# Count matching n-grams
def get_ngrams(seq, n):
return set(zip(*[seq[i:] for i in range(n)]))
scores = []
for pred, ref in zip(predictions, references):
pred_ngrams = get_ngrams(pred, n_gram)
ref_ngrams = get_ngrams(ref, n_gram)
if len(ref_ngrams) == 0:
score = 0
else:
matches = len(pred_ngrams & ref_ngrams)
score = matches / len(ref_ngrams)
scores.append(score)
return np.mean(scores)
# Test
np.random.seed(42)
predictions = [[1, 2, 3, 4], [5, 6, 7, 8]]
references = [[1, 2, 3, 4], [5, 6, 7, 9]]
bleu = compute_bleu_score(predictions, references)
assert 0 <= bleu <= 1, "BLEU in [0,1]"
print("✓ BLEU score computation working")
if __name__ == "__main__":
print("Lab 4: BLEUScore - PASSED")