Machine Translation Seq2seq Neural Machine Translation
# Machine Translation: Seq2Seq & Neural Machine Translation
## Introduction & Motivation
Machine Translation: translate text between languages. Neural Machine Translation: end-to-end seq2seq with attention. Encoder-decoder architecture. Applications: automatic translation, multilingual NLP, cross-lingual understanding.
Motivation: Break language barriers; end-to-end learning.
Applications: Translation, multilingual, cross-lingual.
---
## Core Concepts & Theory
### Encoder-Decoder
Source language to target language.
### Attention Mechanism
Focus on relevant source tokens.
### Teacher Forcing
Ground truth as input during training.
---
## Mathematical Formulation
NMT objective:
$$L = -\sum_t \log P(y_t | y_{<t}, x)$$
Attention:
$$\alpha_{ij} = \frac{\exp(e_{ij})}{\sum_k \exp(e_{ik})}$$
$$c_i = \sum_j \alpha_{ij} h_j$$
Beam search:
$$y^* = \arg\max_y \sum_t \log P(y_t | y_{<t}, x)$$
---
## Advanced Theory & Extensions
### Transformer NMT
Attention-only architecture; parallel.
### Back-translation
Data augmentation; improve translation.
### Multilingual NMT
Multiple language pairs; shared encoder.
---
## Computational Considerations
Encoder: O(source_len·hidden_dim).
Decoder: O(target_len·hidden_dim).
Attention: O(source_len·target_len).
---
## Practical Implementation Strategies
### Byte-Pair Encoding (BPE)
Subword tokenization; shared vocabulary.
### Teacher Forcing
Use ground truth during training.
### Beam Search
K-best hypotheses; decoding trade-off.
---
## Benchmark Datasets & Evaluation
WMT: Workshop on Machine Translation.
BLEU Score: Standard metric.
METEOR, TER: Alternative metrics.
---
## Key Challenges & Limitations
### Low-Resource Languages
Limited parallel data.
### Domain Adaptation
Domain-specific translation.
### Rare Words
OOV and rare word handling.
---
## Hyperparameter Tuning
Beam size: 1-10; speed-accuracy trade-off.
Length penalty: Avoid short translations.
Learning rate: 1e-3 to 1e-4.
---
## Real-World Applications & Case Studies
Web Translation: Google Translate.
Multilingual NLP: Cross-lingual understanding.
Document Translation: CAT systems.
---
## Integration with Other Methods
NMT + Back-translation → augmentation.
NMT + Pivot → low-resource translation.
---
## Summary & Key Takeaways
Machine Translation via Neural Machine Translation enables end-to-end translation through encoder-decoder architectures with attention mechanism.
Principles:
1. Encoder-decoder: source to target.
2. Attention: focus on relevant parts.
3. Teacher forcing: training strategy.
4. Beam search: decoding.
5. BLEU: evaluation metric.
---
---
## Appendix: Practical Labs
### Lab 1: Attention Alignment
import numpy as np
def compute_attention_alignment(encoder_output, decoder_hidden, attention_matrix):
"""Compute attention alignment weights"""
# Decoder view of encoder
scores = decoder_hidden @ attention_matrix @ encoder_output.T
# Softmax
exp_scores = np.exp(scores - np.max(scores))
attention_weights = exp_scores / exp_scores.sum()
return attention_weights
# Test
np.random.seed(42)
encoder_output = np.random.randn(10, 128) # 10 source tokens, 128-dim
decoder_hidden = np.random.randn(128)
attention_matrix = np.random.randn(128, 128)
weights = compute_attention_alignment(encoder_output, decoder_hidden, attention_matrix)
assert weights.shape == (10,), "Attention shape"
assert np.isclose(weights.sum(), 1.0), "Weights sum to 1"
print("✓ Attention alignment working")
if __name__ == "__main__":
print("Lab 1: AttentionAlignment - PASSED")### Lab 2: Beam Search Decoding
import numpy as np
def beam_search_decode(scores, beam_size=3, max_len=20):
"""Simple beam search decoding"""
# Current hypotheses: (score, sequence)
hypotheses = [(0.0, [])]
for step in range(max_len):
all_candidates = []
for score, seq in hypotheses:
# Get scores for next token
if step < len(scores):
token_scores = scores[step]
else:
token_scores = np.zeros(100)
# Top tokens
top_token_ids = np.argsort(token_scores)[-beam_size:]
for token_id in top_token_ids:
new_score = score + token_scores[token_id]
new_seq = seq + [token_id]
all_candidates.append((new_score, new_seq))
# Keep top beam_size
hypotheses = sorted(all_candidates, key=lambda x: x[0], reverse=True)[:beam_size]
return hypotheses[0][1]
# Test
np.random.seed(42)
scores = np.random.randn(10, 100) # 10 steps, 100 vocab
sequence = beam_search_decode(scores, beam_size=3, max_len=5)
assert len(sequence) == 5, "Sequence length"
assert all(0 <= id < 100 for id in sequence), "Valid token IDs"
print("✓ Beam search working")
if __name__ == "__main__":
print("Lab 2: BeamSearch - PASSED")### Lab 3: BLEU Score
import numpy as np
def compute_bleu(reference, hypothesis, n=4):
"""Compute BLEU score (simplified)"""
ref_tokens = reference.split()
hyp_tokens = hypothesis.split()
# Precision for each n-gram
precisions = []
for ng in range(1, n + 1):
if len(hyp_tokens) < ng:
precisions.append(0)
continue
# Count n-grams
ref_ngrams = {}
for i in range(len(ref_tokens) - ng + 1):
ngram = tuple(ref_tokens[i:i+ng])
ref_ngrams[ngram] = ref_ngrams.get(ngram, 0) + 1
hyp_ngrams = {}
for i in range(len(hyp_tokens) - ng + 1):
ngram = tuple(hyp_tokens[i:i+ng])
hyp_ngrams[ngram] = hyp_ngrams.get(ngram, 0) + 1
# Match count
matches = sum(min(hyp_ngrams.get(ngram, 0), count)
for ngram, count in ref_ngrams.items())
total = max(len(hyp_tokens) - ng + 1, 0)
precision = matches / total if total > 0 else 0
precisions.append(precision)
# Geometric mean
geo_mean = np.exp(np.mean(np.log(np.array(precisions) + 1e-8)))
# Brevity penalty
bp = min(1, np.exp(1 - len(ref_tokens) / max(len(hyp_tokens), 1)))
bleu = bp * geo_mean
return bleu
# Test
reference = "the cat is on the mat"
hypothesis = "the cat is on the mat"
bleu = compute_bleu(reference, hypothesis)
assert 0 <= bleu <= 1, "BLEU in [0,1]"
print("✓ BLEU score working")
if __name__ == "__main__":
print("Lab 3: BLEUScore - PASSED")### Lab 4: Teacher Forcing
import numpy as np
def teacher_forcing_loss(encoder_output, target_sequence, vocab_size=1000):
"""Compute loss with teacher forcing"""
losses = []
decoder_hidden = encoder_output.mean(axis=0) # Initialize with encoder output
for t in range(len(target_sequence)):
# Decode with ground truth input
current_token = target_sequence[t]
# Simulate decoder logits (normally from neural net)
logits = np.random.randn(vocab_size)
# Cross-entropy loss
exp_logits = np.exp(logits - np.max(logits))
probs = exp_logits / exp_logits.sum()
loss = -np.log(probs[current_token] + 1e-8)
losses.append(loss)
return np.mean(losses)
# Test
np.random.seed(42)
encoder_output = np.random.randn(10, 128)
target = [1, 5, 3, 2, 7]
loss = teacher_forcing_loss(encoder_output, target)
assert np.isfinite(loss), "Loss finite"
assert loss > 0, "Loss positive"
print("✓ Teacher forcing working")
if __name__ == "__main__":
print("Lab 4: TeacherForcing - PASSED")