Sequence-to-Sequence Models in NLP Encoder-Decoder Architectures and Attention
# Sequence-to-Sequence Models in NLP: Encoder-Decoder Architectures and Attention
## 1. Introduction & Motivation
Sequence-to-sequence (Seq2Seq) models represent a fundamental paradigm for mapping input sequences to output sequences of potentially different lengths and vocabularies. These architectures underpin major NLP applications including machine translation, abstractive summarization, question answering, and semantic parsing.
The key innovation of Seq2Seq is decoupling input and output processing through an encoder-decoder framework, where the encoder compresses the input sequence into a fixed-size context vector, and the decoder generates output tokens conditioned on this representation. This separation enables training on aligned sequence pairs and inference through autoregressive generation.
The attention mechanism, introduced concurrently with early Seq2Seq work, addresses a critical limitation: the fixed-size context vector bottleneck. Attention allows the decoder to dynamically access and focus on relevant parts of the input sequence at each generation step, dramatically improving performance on long sequences and enabling interpretable alignment between source and target.
This article provides comprehensive coverage of Seq2Seq architectures, attention mechanisms, decoding algorithms, training objectives, and practical implementation strategies.
## 2. Core Concepts & Theory
### 2.1 Encoder-Decoder Framework
The Seq2Seq architecture comprises two components:
Encoder: Processes input sequence x = (x_1, ldots, x_T) to produce a sequence of hidden states:
$$h_t = ext{Encoder}(x_t, h_{t-1})$$
For RNN-based encoders (LSTM/GRU), this produces H = (h_1, ldots, h_T) representing the encoded input.
Decoder: Generates output sequence y = (y_1, ldots, y_M) autoregressively, conditioned on encoder outputs:
$$p(y_t | y_{<t}, H) = ext{softmax}(W_y \cdot s_t + b_y)$$
where s_t is the decoder hidden state at step t.
The joint probability factorizes as:
$$p(y | x) = \prod_{t=1}^{M} p(y_t | y_{<t}, x)$$
### 2.2 Context Vector Representation
Early Seq2Seq models used the final encoder hidden state as context:
$$c = h_T \quad ext{(last hidden state)}$$
This context is then used to initialize the decoder:
$$s_0 = W_c \cdot c$$
For bidirectional encoders, the context concatenates forward and backward finals:
$$c = ext{concat}(h_T^{ ightarrow}, h_0^{\leftarrow})$$
However, single context vectors suffer from information compression, particularly for long sequences where important information may be distributed across multiple time steps.
### 2.3 Attention Mechanism Fundamentals
Attention computes a weighted sum over encoder hidden states:
$$c_t = \sum_{s=1}^{T} \alpha_{t,s} h_s$$
where attention weights alpha_{t,s} are computed using a compatibility function:
$$e_{t,s} = ext{score}(s_t, h_s)$$
$$\alpha_{t,s} = \frac{\exp(e_{t,s})}{\sum_{s'=1}^{T} \exp(e_{t,s'})}$$
### 2.4 Attention Score Functions
Additive (Bahdanau) Attention:
$$e_{t,s} = v^T anh(W_1 s_t + W_2 h_s + b)$$
where v is a learned vector. This is the original attention mechanism from Bahdanau et al.
Multiplicative (Luong) Attention:
$$e_{t,s} = s_t^T W h_s$$
More computationally efficient, typically performs similarly to additive attention.
Scaled Dot-Product Attention:
$$e_{t,s} = \frac{s_t^T h_s}{\sqrt{d_k}}$$
The scaling factor sqrt(d_k) (where d_k is the key dimension) prevents extremely small gradients.
## 3. Mathematical Formulation
### 3.1 Encoder with Bidirectional LSTM
For bidirectional encoding:
$$\overrightarrow{h}_t = ext{LSTM}(x_t, \overrightarrow{h}_{t-1})$$
$$\overleftarrow{h}_t = ext{LSTM}(x_t, \overleftarrow{h}_{t+1})$$
$$h_t = ext{concat}(\overrightarrow{h}_t, \overleftarrow{h}_t)$$
The bidirectional context provides access to both past and future information, improving encoder representation quality.
### 3.2 Decoder with Attention
The decoder LSTM state update incorporates the context vector:
$$s_t = ext{LSTM}(y_{t-1}, s_{t-1}, c_t)$$
Some architectures feed context and input jointly:
$$s_t = ext{LSTM}( ext{concat}(y_{t-1}, c_t), s_{t-1})$$
Output probabilities are computed from the decoder state and context:
$$p(y_t | y_{<t}, x) = ext{softmax}(W_{sy}(s_t \oplus c_t) + b_y)$$
where (concat) denotes concatenation.
### 3.3 Training Objective: Negative Log-Likelihood
The model is trained to maximize the log-probability of reference sequences:
$$\mathcal{L} = -\sum_{t=1}^{M} \log p(y_t^* | y_{<t}^*, x)$$
where y* denotes reference tokens. This cross-entropy loss assumes access to ground truth history at training time.
### 3.4 Multi-Head Attention
Extending single attention heads to multiple independent attention mechanisms:
$$ ext{head}_i = ext{Attention}(Q W_i^Q, K W_i^K, V W_i^V)$$
$$ ext{MultiHead}(Q, K, V) = ext{concat}( ext{head}_1, \ldots, ext{head}_h) W^O$$
With h heads (typically 8-12), each head specializes in different types of alignments.
## 4. Advanced Theory & Extensions
### 4.1 Coverage Mechanism
Standard attention can produce misaligned output (e.g., repeating or skipping words). Coverage tracking penalizes redundant attention:
$$ ext{cov}_t = \sum_{t'=1}^{t-1} \alpha_{t'}$$
The coverage loss encourages attention diversity:
$$\mathcal{L}_{ ext{cov}} = \sum_s \min(\alpha_{t,s}, ext{cov}_t(s))$$
### 4.2 Copy Mechanism (Pointer Networks)
For tasks requiring output to directly copy input tokens (NER, abstractive summarization):
$$p_{ ext{copy}}(y_t) = \sigma( ext{score}(s_t, ext{avg}(H))) \cdot \max_s \alpha_{t,s}$$
The model learns when to copy versus generate, improving handling of OOV words and factual accuracy.
### 4.3 Hierarchical Attention
For long documents, apply hierarchical attention:
- Sentence-level attention identifies important sentences
- Word-level attention within selected sentences
$$c_{ ext{sent},i} = \sum_i \alpha_{ ext{sent},i} s_i$$
$$c_{ ext{word},i} = \sum_j \alpha_{ ext{word},(i,j)} h_{i,j}$$
### 4.4 Self-Attention and Transformer Encoders
Self-attention in the encoder allows each token to attend to all other tokens:
$$ ext{Attention}(Q, K, V) = ext{softmax}\left(\frac{QK^T}{\sqrt{d_k}} ight) V$$
Stacking self-attention layers with feed-forward networks creates the Transformer architecture, which has largely replaced RNN-based Seq2Seq in modern NLP.
## 5. Computational Considerations
### 5.1 Attention Complexity
Time complexity: Computing attention over sequence length T requires O(T^2) operations (Query-key dot products O(T^2), Softmax normalization O(T^2), Value aggregation O(T^2)). For long documents (T > 1000), quadratic complexity becomes prohibitive.
Space complexity: Storing attention weights requires O(T^2) memory (Full attention matrix T x T floats, Activation caching for backpropagation: additional O(T^2)).
### 5.2 Linear Attention Approximations
Kernel trick attention:
$$ ext{Attention}(Q, K, V) = \frac{\phi(Q)(\phi(K)^T V)}{\phi(Q)(\phi(K)^T \mathbf{1})}$$
Using kernel functions phi , reducing complexity to O(T).
Sparse attention patterns: Limit attention to local windows or strided patterns:
$$\alpha_{t,s} \propto \exp(s_t^T K_s) \quad ext{if } |t - s| \leq w$$
where w is window size.
### 5.3 Inference Optimization
Beam search complexity: Maintaining B hypotheses requires O(B * M * T) operations, where B is beam size and M is output length.
Cached attention: Reuse previous attention computations to avoid redundant matrix multiplications.
Batched inference: Process multiple sequences simultaneously, leveraging hardware parallelization.
## 6. Practical Implementation Strategies
### 6.1 Initialization and Preprocessing
Encoder initialization:
- Orthogonal initialization for recurrent weights
- Xavier uniform for feed-forward weights
- Typically better convergence than Gaussian initialization
Input preprocessing:
- Tokenization: Byte-pair encoding (BPE) for subword units
- Vocabulary: 30K-50K tokens typical for medium-sized datasets
- Special tokens: `<BOS>` (beginning), `<EOS>` (ending), `<UNK>` (unknown), `<PAD>` (padding)
Word embeddings:
- Random initialization often outperforms pretrained for some tasks
- Pretrained embeddings (Word2Vec, GloVe) provide benefit for low-resource tasks
- Typical embedding dimension: 128-512
### 6.2 Training Strategies
Teacher forcing: During training, decoder always uses ground-truth tokens:
$$s_t = ext{LSTM}(y_{t-1}^*, s_{t-1}, c_t)$$
This provides stable gradients but causes exposure bias at test time when the model generates incorrect tokens.
Scheduled sampling: Gradually transition from teacher forcing to model output:
$$p_{ ext{gold}}(t) = \epsilon \cdot \left(1 - \frac{t}{T_{ ext{max}}} ight)$$
With probability p_{text{gold}}(t) , feed ground-truth; otherwise feed model output.
Auxiliary objectives:
- Translation ranking: Score reference higher than alternatives
- Back-translation: Intermediate supervision from reverse direction
- Distillation: Train student model on large teacher predictions
### 6.3 Decoding Strategies
Greedy Decoding:
$$y_t = \arg\max_v p(v | y_{<t}, x)$$
Fast but prone to suboptimal sequences; early mistakes are irreversible.
Beam Search:
Maintain top-B hypotheses, expanding each at each step:
$$ ext{candidates} = \bigcup_{i=1}^{B} \{(y_i^{(t)}, y_t) | y_t \in V\}$$
Select top-B by cumulative score:
$$s(y) = \frac{1}{|y|^\alpha} \log p(y | x)$$
Length normalization with alpha in [0.6, 1.0] prevents brevity bias.
Temperature Sampling:
Control output randomness:
$$p_{ ext{temp}}(y_t) = \frac{p(y_t)^{1/ au}}{\sum_v p(v)^{1/ au}}$$
High temperature ( tau > 1 ) produces diverse outputs; low temperature ( tau < 1 ) concentrates probability.
### 6.4 Attention Visualization and Debugging
Attention matrix inspection:
- Plot attention heatmaps to verify meaningful alignments
- Check for degenerate patterns (uniform attention, diagonal concentration)
- Diagnose attention collapse (all probability on single position)
Saliency analysis:
- Identify which input tokens most influence output tokens
- Backward gradient computation through attention weights
- Useful for error analysis and interpretability
## 7. Benchmark Datasets & Evaluation
### 7.1 Machine Translation Benchmarks
WMT14 English-German:
- Size: 4.5M parallel sentence pairs
- Domain: News articles, mixed corpora
- Evaluation metric: BLEU score
- Baseline (Seq2Seq with attention): ~21 BLEU
- Transformer baseline: ~28 BLEU
IWSLT Chinese-English:
- Size: 250K sentences
- Domain: TED talks transcripts
- Baseline: ~18 BLEU
- Useful for medium-resource translation
### 7.2 Summarization Benchmarks
CNN/DailyMail:
- Size: 287K document-summary pairs
- Domain: News articles, extractive references
- Metrics: ROUGE-1/2/L (overlap with reference)
- Baseline Seq2Seq: ROUGE-1 34.5, ROUGE-L 31.1
- Transformer: ROUGE-1 41.2, ROUGE-L 38.8
SAMSum:
- Size: 16K dialogues with summaries
- Domain: Customer support conversations
- Baseline: ROUGE-1 42.5
- Modern fine-tuned models: ROUGE-1 47+
### 7.3 Question Answering
SQuAD 2.0:
- Size: 100K+ questions from Wikipedia
- Evaluation: Exact Match (EM), F1 score
- Seq2Seq baseline: ~60 EM, ~70 F1
- Modern models: ~90+ EM, ~97+ F1
### 7.4 Paraphrase Generation
PAWS:
- Size: 108K paraphrase pairs
- Evaluation: Semantic similarity (BLEU, BERTScore)
- Task: Generate alternative phrasings
- Baseline: ~45 BLEU
## 8. Key Challenges & Limitations
### 8.1 Attention is Not Explanation
While attention visualizations appear interpretable, research shows:
- Attention weights don't necessarily correspond to feature importance
- Multiple attention heads may have conflicting patterns
- Low attention to a token doesn't mean it's unimportant
Solution: Use gradient-based saliency for more reliable explanations.
### 8.2 Exposure Bias
Training with teacher forcing but testing with model output creates distribution mismatch:
$$p_{ ext{train}}(y_t | y_{<t}^*, x) eq p_{ ext{test}}(y_t | y_{<t}^{ ext{model}}, x)$$
Errors compound: initial mistakes condition on corrupted history, degrading final outputs. Scheduled sampling or reinforcement learning-based training provides mitigation.
### 8.3 Long Sequence Handling
- Quadratic complexity: Attention becomes expensive for sequences >500 tokens
- Information bottleneck: Compressing long inputs to fixed-size context loses information
- Gradient flow: Long-distance attention gradients weaken
Solutions: Hierarchical attention, local attention windows, or switching to constant-space RNN variants.
### 8.4 Rare Word and OOV Handling
Standard vocabularies (~50K tokens) leave ~10-15% of test words unseen (OOV):
$$p( ext{OOV} | ext{dataset}) \approx \frac{ ext{unique rare tokens}}{ ext{total tokens}}$$
Character-level models, subword tokenization (BPE, SentencePiece), and copy mechanisms address this.
## 9. Hyperparameter Tuning & Optimization
### 9.1 Architecture Hyperparameters
Encoder-Decoder dimensions:
- Typical: d_h = 512-1024 for attention-based Seq2Seq
- Larger models: d_h = 2048 with dropout and weight decay
- Embedding dimension: Usually same as d_h ; sometimes d_e = 0.5 d_h for large vocab
Attention dimension:
- Query/Key dimension:
$$ d_k = d_h / n_{ ext{heads}} $$
for multi-head attention
- Value dimension: d_v = d_k (typically)
- 8-12 heads standard; diminishing returns beyond 16
Layers:
- Typical: 2-4 encoder layers, 2-4 decoder layers
- Deeper models improve performance but increase training time
- Residual connections essential for >3 layers
### 9.2 Training Hyperparameters
Batch size: B = 32-128 typical
- Larger batches: Faster training, possibly worse generalization
- Smaller batches: Noisier gradients, better generalization
- Gradient accumulation simulates larger effective batch size
Learning rate: Highly dependent on optimizer and dataset
- Adam: eta = 1e^{-4} to 1e^{-3} - SGD with momentum: eta = 0.1 to 1.0
- Learning rate decay: eta_t = eta_0 * 0.95^{t} or warm restarts
Dropout:
- Typical: p = 0.2-0.3 on embeddings and states
- Higher for large models or small datasets
- Lower for small datasets risks overfitting but also underfitting
### 9.3 Decoding Hyperparameters
Beam size: B = 3-10 typical
- Larger beams improve quality but with diminishing returns
- B = 5 often optimal trade-off between quality and speed
- B > 10 rarely provides significant improvements
Length penalty:
- Affects brevity vs verbosity trade-off
- alpha = 0.6-0.8 typical; higher values encourage longer outputs
- Task-dependent: summarization prefers shorter ( alpha = 0.6 ), paraphrase longer ( alpha = 0.9 )
Temperature: tau = 1.0 (no scaling) standard for translation; tau = 1.5-2.0 for diversity
## 10. Real-World Applications & Case Studies
### 10.1 Neural Machine Translation (NMT)
Problem: Translate 1M sentence pairs from English to French
Architecture:
- Bidirectional LSTM encoder (2 layers, 512 units)
- Unidirectional LSTM decoder (2 layers, 512 units)
- Bahdanau attention mechanism
- Shared embeddings (20K vocabulary)
Results:
- Baseline: 24.3 BLEU
- With dropout (0.2): 26.1 BLEU (+1.8%)
- With coverage mechanism: 26.8 BLEU (+2.5%)
- Beam search (size 5): 27.2 BLEU (+2.9%)
Deployment considerations:
- Batched inference for throughput
- Cached attention to reduce computation
- Quantization to 8-bit for mobile devices
### 10.2 Abstractive Summarization
Problem: Generate concise summaries from news articles
Architecture:
- Input: Article text with 600-800 tokens
- BiLSTM encoder (2 layers, 256 units)
- LSTM decoder with copy mechanism
- Vocabulary: 50K from training set
Results:
- ROUGE-1: 38.2 (without copy mechanism)
- ROUGE-1: 40.1 (with copy mechanism)
- Average summary length: 85 tokens (compared to reference 75)
Key insights:
- Copy mechanism essential for factual accuracy (~2% improvement)
- Coverage mechanism prevents repetition
- Extractive baseline achieves ROUGE-1 40.5; Seq2Seq with copy competitive
### 10.3 Dialog Response Generation
Problem: Generate response in customer support conversation
Architecture:
- Multi-turn context encoding
- Hierarchical attention (dialog turn level + word level)
- Constraint: Response length < 150 tokens
Results:
- BLEU: 12.5 (automatic evaluation)
- Human evaluation: 4.1/5.0 (relevance), 3.8/5.0 (fluency)
- Inference speed: 150ms per response
Challenges:
- High variance in response acceptability
- Difficulty enforcing length constraints
- Tendency toward generic responses (addressed with diverse beam search)
### 10.4 Code Generation
Problem: Generate Python code from natural language docstrings
Architecture:
- LSTM encoder (400 units)
- LSTM decoder with syntax-aware attention
- Special tokens for language constructs (def, return, etc.)
Results:
- Exact match: 28.3% on test set
- Execution correctness: 35.7%
- Average prediction time: 280ms
Key techniques:
- Vocabulary includes common APIs and function names
- Copy mechanism for code identifiers
- Syntactic constraints during decoding (parse tree enforced)
## 11. Integration with Other Methods
### 11.1 Pre-trained Encoders
Combining Seq2Seq decoders with pre-trained encoders (BERT, RoBERTa):
$$h_t = ext{BERT}(x)_t$$
$$s_t = ext{LSTM}(y_{t-1}, s_{t-1}, ext{Attention}(s_t, h))$$
Pre-trained representations provide substantial improvements:
- Machine translation: +2-3 BLEU
- Summarization: +2-4 ROUGE-1
- Faster convergence to competitive performance
### 11.2 Reinforcement Learning Objectives
Combining maximum likelihood training with RL rewards:
$$\mathcal{L} = -\lambda \log p(y^* | x) + (1-\lambda) \mathbb{E}_{y \sim p_ heta}[-R(y, y^*)]$$
where R is a task-specific reward (e.g., BLEU for MT, task completion for dialog).
Benefits: Direct optimization toward evaluation metrics, handling task-specific constraints.
### 11.3 Multi-Task Learning
Training Seq2Seq on related tasks simultaneously:
$$\mathcal{L} = \sum_i w_i \mathcal{L}_i$$
Tasks: translation, paraphrase generation, back-translation. Shared encoder learns more robust representations.
### 11.4 Fusion with Knowledge Graphs
Incorporating external knowledge:
$$c_t = ext{Attention}(s_t, h) + ext{GraphAttention}(s_t, k)$$
where k represents knowledge base entities. Improves factuality and reduces hallucination.
## 12. Future Research Directions
### 12.1 Non-Autoregressive Generation
Traditional Seq2Seq generates tokens sequentially. Non-autoregressive models predict all tokens in parallel:
$$p(y | x) = \prod_t p(y_t | x)$$
Benefits: Constant-time inference independent of output length. Challenges: Loss of autoregressive modeling advantages.
### 12.2 Document-Level Modeling
Current models process sentences independently. Document-level Seq2Seq:
$$c_{ ext{doc}} = \sum_i \alpha_i h_i^{ ext{sent}}$$
Captures discourse coherence and long-range pronoun resolution.
### 12.3 Multimodal Seq2Seq
Combining text with vision:
$$h_{ ext{vision}} = ext{CNN}(x_{ ext{image}})$$
$$h = ext{concat}(h_{ ext{text}}, h_{ ext{vision}})$$
Applications: Visual question answering, image captioning, video understanding.
### 12.4 Interpretable and Faithful Generation
Current models may generate grammatical but nonsensical outputs. Future research:
- Constrained decoding enforcing semantic coherence
- Explanation generation alongside predictions
- Provenance tracking from input to output
## 13. Summary & Key Takeaways
Core Concepts:
- Encoder-decoder architecture decouples input and output processing
- Attention mechanisms allow dynamic focus on input regions
- Softmax attention creates interpretable (though not explanatory) alignments
Attention Mechanisms:
- Bahdanau (additive): More expressive but higher complexity
- Luong (multiplicative): Efficient, comparable performance
- Scaled dot-product: Foundation for Transformers
- Multi-head: Specializes different heads to different patterns
Training Strategies:
- Teacher forcing provides stable gradients but causes exposure bias
- Scheduled sampling mitigates exposure bias gradually
- Gradient clipping (norm 1-5) essential for stability
Decoding:
- Greedy decoding fast but suboptimal
- Beam search (size 5-10) good quality-speed trade-off
- Length normalization ( alpha = 0.6-0.8 ) prevents brevity bias
Practical Performance:
- Seq2Seq + attention: ~24 BLEU on WMT (2014 state-of-the-art)
- Modern transformers: ~28+ BLEU (better architecture)
- Pre-trained encoders: +2-3 BLEU improvement
- Copy mechanism: +1-2 BLEU for extractive tasks
Key Limitations:
- Quadratic attention complexity limits to ~1000 token sequences
- Exposure bias causes distribution mismatch between train and test
- Attention visualization doesn't reliably indicate importance
- Still generates hallucinated or nonsensical outputs despite fluency
Current Status:
Transformers have largely replaced RNN-based Seq2Seq for state-of-the-art NLP results, but the encoder-decoder + attention framework remains foundational to modern architectures. The principles learned from Seq2Seq (encoder-decoder separation, attention mechanisms, decoding strategies) directly transfer to Transformer-based models.
---
## Appendix: Practical Implementation Labs
### Lab 1: Bahdanau Attention Implementation
import torch
import torch.nn as nn
import torch.nn.functional as F
class BahdanauAttention(nn.Module):
def __init__(self, encoder_hidden_dim, decoder_hidden_dim, attention_dim):
super().__init__()
self.attention = nn.Linear(encoder_hidden_dim + decoder_hidden_dim,
attention_dim)
self.v = nn.Linear(attention_dim, 1, bias=False)
def forward(self, decoder_hidden, encoder_outputs):
# decoder_hidden: [batch, decoder_hidden_dim]
# encoder_outputs: [batch, seq_len, encoder_hidden_dim]
batch_size = encoder_outputs.shape[0]
seq_len = encoder_outputs.shape[1]
# Expand decoder_hidden to match sequence length
decoder_hidden_expanded = decoder_hidden.unsqueeze(1).expand(
batch_size, seq_len, -1)
# Concatenate and compute attention scores
combined = torch.cat([encoder_outputs, decoder_hidden_expanded], dim=2)
energy = torch.tanh(self.attention(combined)) # [B, T, attention_dim]
scores = self.v(energy).squeeze(-1) # [B, T]
# Apply softmax
attention_weights = F.softmax(scores, dim=1) # [B, T]
# Compute context vector
context = torch.bmm(attention_weights.unsqueeze(1),
encoder_outputs) # [B, 1, encoder_hidden_dim]
context = context.squeeze(1)
return context, attention_weights
# Test
attention = BahdanauAttention(512, 512, 256)
encoder_outputs = torch.randn(32, 50, 512)
decoder_hidden = torch.randn(32, 512)
context, weights = attention(decoder_hidden, encoder_outputs)
print(f"Context shape: {context.shape}, Weights shape: {weights.shape}")### Lab 2: Beam Search Decoding
import torch
import heapq
def beam_search(model, encoder_output, beam_size=5, max_length=50):
batch_size = encoder_output.shape[0]
device = encoder_output.device
# Initialize beam state: (score, sequence, hidden)
initial_token = torch.full((batch_size, 1), model.sos_token, device=device)
initial_hidden = model.init_hidden(batch_size)
beams = [[(-float('inf'), [model.sos_token], initial_hidden)]]
completed = []
for step in range(max_length):
new_beams = [[] for _ in range(batch_size)]
for batch_idx, beam in enumerate(beams[batch_idx] if batch_idx < len(beams) else []):
score, sequence, hidden = beam
if len(completed) >= beam_size:
break
# Get next token predictions
prev_token = torch.tensor([sequence[-1]], device=device).unsqueeze(0)
logits, hidden = model.decode_step(prev_token, hidden, encoder_output)
log_probs = torch.log_softmax(logits, dim=-1)[0]
# Get top candidates
top_log_probs, top_tokens = torch.topk(log_probs, beam_size)
for token_idx, top_token in enumerate(top_tokens):
new_score = score + top_log_probs[token_idx].item()
new_sequence = sequence + [top_token.item()]
if top_token.item() == model.eos_token:
completed.append((new_score, new_sequence))
else:
new_beams[batch_idx].append((new_score, new_sequence, hidden))
# Keep top beam_size hypotheses
for batch_idx in range(batch_size):
beams[batch_idx] = sorted(new_beams[batch_idx],
key=lambda x: x[0], reverse=True)[:beam_size]
# Return best sequence
best_score, best_sequence = max(completed, key=lambda x: x[0] / len(x[1]))
return best_sequence
# Usage in model
# sequences = beam_search(model, encoder_output, beam_size=5)### Lab 3: Seq2Seq with Attention Architecture
import torch
import torch.nn as nn
class Seq2SeqWithAttention(nn.Module):
def __init__(self, vocab_size, embedding_dim, hidden_dim, attention_dim,
num_layers=2, dropout=0.3):
super().__init__()
self.embedding = nn.Embedding(vocab_size, embedding_dim, padding_idx=0)
# Encoder
self.encoder = nn.LSTM(embedding_dim, hidden_dim, num_layers,
batch_first=True, dropout=dropout, bidirectional=True)
# Attention
self.attention = BahdanauAttention(hidden_dim * 2, hidden_dim, attention_dim)
# Decoder
self.decoder = nn.LSTM(embedding_dim + hidden_dim * 2, hidden_dim,
num_layers, batch_first=True, dropout=dropout)
self.output_proj = nn.Linear(hidden_dim, vocab_size)
self.dropout = nn.Dropout(dropout)
def encode(self, src):
embedded = self.dropout(self.embedding(src))
encoder_outputs, (hidden, cell) = self.encoder(embedded)
return encoder_outputs, hidden, cell
def decode_step(self, prev_token, hidden, cell, encoder_outputs):
embedded = self.dropout(self.embedding(prev_token))
decoder_output, (hidden, cell) = self.decoder(embedded, (hidden, cell))
# Apply attention
context, _ = self.attention(decoder_output[:, -1, :], encoder_outputs)
# Concatenate context and decoder output
combined = torch.cat([decoder_output[:, -1, :], context], dim=1)
logits = self.output_proj(combined)
return logits, (hidden, cell)
def forward(self, src, tgt):
encoder_outputs, hidden, cell = self.encode(src)
outputs = []
for t in range(tgt.shape[1]):
logits, (hidden, cell) = self.decode_step(
tgt[:, t:t+1], hidden, cell, encoder_outputs)
outputs.append(logits)
return torch.stack(outputs, dim=1)
# Test
model = Seq2SeqWithAttention(vocab_size=10000, embedding_dim=256,
hidden_dim=512, attention_dim=256)
src = torch.randint(1, 10000, (32, 50))
tgt = torch.randint(1, 10000, (32, 40))
output = model(src, tgt)
print(f"Output shape: {output.shape}") # [32, 40, 10000]### Lab 4: Training Loop with Scheduled Sampling
import torch
import torch.nn as nn
import numpy as np
def train_with_scheduled_sampling(model, train_loader, num_epochs,
initial_teacher_forcing=1.0, min_tf=0.1):
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
criterion = nn.CrossEntropyLoss(ignore_index=0)
total_steps = num_epochs * len(train_loader)
for epoch in range(num_epochs):
epoch_loss = 0
for batch_idx, (src, tgt) in enumerate(train_loader):
step = epoch * len(train_loader) + batch_idx
# Compute teacher forcing probability (linear decay)
p_teacher = max(min_tf, initial_teacher_forcing *
(1.0 - step / total_steps))
# Encode
encoder_outputs, hidden, cell = model.encode(src)
loss = 0
decoder_input = tgt[:, 0:1]
# Decode with scheduled sampling
for t in range(1, tgt.shape[1]):
logits, (hidden, cell) = model.decode_step(
decoder_input, hidden, cell, encoder_outputs)
loss += criterion(logits, tgt[:, t])
# Scheduled sampling: choose between ground truth and model output
if np.random.random() < p_teacher:
decoder_input = tgt[:, t:t+1]
else:
decoder_input = torch.argmax(logits, dim=1,
keepdim=True)
optimizer.zero_grad()
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
optimizer.step()
epoch_loss += loss.item()
if batch_idx % 100 == 0:
print(f"Epoch {epoch}, Batch {batch_idx}, Loss: {loss.item():.4f}, "
f"Teacher forcing prob: {p_teacher:.3f}")
print(f"Epoch {epoch} Average Loss: {epoch_loss / len(train_loader):.4f}")
# Usage
# train_with_scheduled_sampling(model, train_loader, num_epochs=20)