Mbart - Multilingual Denoising

# mBART - Multilingual Denoising

## Introduction & Motivation

mBART: multilingual BART for cross-lingual transfer. Denoising autoencoder on 25+ languages. Applications: machine translation, zero-shot transfer.

Motivation: Enable cross-lingual pre-training at scale.

Applications: Low-resource translation, multilingual understanding.

---

## Core Concepts & Theory

### Multilingual Masking

Mask tokens across multiple languages.

### Cross-Lingual Pretraining

Shared vocabulary across languages.

### Denoising Objective

Reconstruct from noisy input.

### Language-Agnostic Representations

Shared space across languages.

---

## Mathematical Formulation

Denoising Loss:
$$\mathcal{L}_{ ext{denoise}} = -\sum_t \log p(y_t | y_{<t}, z)$$

Cross-Lingual Alignment:
$$ ext{loss} = \mathcal{L}_{ ext{denoise}} + \lambda \cdot ext{alignment}$$

Language Mixing:
$$p( ext{lang}_i) = \frac{\lambda_i}{\sum_j \lambda_j}$$

---

## Advanced Theory & Extensions

### Backtranslation

Pseudo-parallel data generation.

### Pivoting

Bridge languages via high-resource language.

### Multilingual Fine-tuning

Task-specific adaptation.

---

## Computational Considerations

Vocabulary: O(N·L) for N words × L languages.

Cross-attention: O(T·S·D²).

Language tokens: Minimal overhead.

---

## Practical Implementation Strategies

### Shared Vocabulary

Common BPE across languages.

### Language Tags

Mark source/target languages.

### Batch Diversity

Mix languages in batches.

---

## Benchmark Datasets & Evaluation

FLORES: Machine translation benchmark.

TED Corpus: Multilingual translation.

XQuAD: Cross-lingual QA.

---

## Key Challenges & Limitations

### Low-Resource Languages

Insufficient training data.

### Language Imbalance

High-resource dominance.

### Vocabulary Explosion

Large cross-lingual vocabulary.

---

## Hyperparameter Tuning

Language mixing: Uniform or weighted.

Mask ratio: 0.15-0.25.

Learning rate: 1e-4 to 1e-3.

---

## Real-World Applications & Case Studies

Zero-Shot Translation: Translate unseen language pairs.

Cross-Lingual Transfer: Learn from one language, apply to another.

Multilingual Summarization: Summarize across languages.

---

## Integration with Other Methods

mBART + back-translation; + knowledge distillation for efficiency.

---

## Summary & Key Takeaways

mBART enables multilingual pre-training through denoising.

Principles:
1. Multilingual masking: Cross-language learning.
2. Denoising: Reconstruct from corruption.
3. Shared vocabulary: Common representation.
4. Language tags: Mark source/target.
5. Cross-lingual transfer: Zero-shot capabilities.

---

## Appendix: Practical Labs

### Lab 1: Multilingual Masking

import numpy as np

def multilingual_mask(tokens, languages, mask_prob=0.15):
 """Mask tokens across languages"""
 masked = tokens.copy()
 n_mask = int(len(tokens) * mask_prob)
 
 # Stratified sampling across languages
 lang_indices = {}
 for i, lang in enumerate(languages):
 if lang not in lang_indices:
 lang_indices[lang] = []
 lang_indices[lang].append(i)
 
 mask_indices = []
 for lang in lang_indices:
 lang_mask = min(n_mask // len(lang_indices), len(lang_indices[lang]))
 mask_indices.extend(np.random.choice(lang_indices[lang], lang_mask, replace=False))
 
 masked[mask_indices] = '[MASK]'
 return masked

tokens = ['hello', 'world', 'bonjour', 'monde']
langs = ['en', 'en', 'fr', 'fr']
masked = multilingual_mask(tokens, langs)
assert '[MASK]' in masked
print("✓ Multilingual masking working")

### Lab 2: Language Tag Injection

def inject_language_tags(tokens, source_lang, target_lang):
 """Add language tags to input"""
 tagged = [f'<{source_lang}>', *tokens, f'<{target_lang}>']
 return tagged

tokens = ['hello', 'world']
tagged = inject_language_tags(tokens, 'en_US', 'fr_FR')
assert '<en_US>' in tagged
print(f"✓ Language tags: {tagged}")

### Lab 3: Cross-Lingual Loss

import numpy as np

def cross_lingual_loss(reconstruction_loss, alignment_loss, alpha=0.5):
 """Combine reconstruction and alignment losses"""
 total = alpha * reconstruction_loss + (1 - alpha) * alignment_loss
 return total

recon = 2.3
align = 0.5
loss = cross_lingual_loss(recon, align)
assert loss > 0
print(f"✓ Cross-lingual loss: {loss:.3f}")

### Lab 4: Language Pair Sampling

import numpy as np

def sample_language_pairs(languages, weights=None):
 """Sample language pairs for training"""
 if weights is None:
 weights = np.ones(len(languages)) / len(languages)
 
 # Sample source and target
 source = np.random.choice(languages, p=weights)
 target = np.random.choice(languages, p=weights)
 
 return source, target

langs = ['en', 'fr', 'de', 'es', 'zh']
weights = np.array([0.4, 0.2, 0.15, 0.15, 0.1])
source, target = sample_language_pairs(langs, weights)
assert source in langs and target in langs
print(f"✓ Language pair: {source} -> {target}")

---