neural machine translation

**Neural Machine Translation (NMT)** is the **deep learning approach to machine translation that models the probability of a target-language sentence given a source-language sentence using an encoder-decoder neural network — where the transformer architecture with multi-head attention learns to align source and target words without explicit word alignment, achieving translation quality that approaches human parity on high-resource language pairs (English-German, English-Chinese) and enabling multilingual models that translate between 100+ languages with a single model**. **Architecture Evolution** **Sequence-to-Sequence with Attention (2014-2017)**: - Encoder: BiLSTM reads the source sentence and produces a sequence of hidden states. - Attention: At each decoder step, compute attention weights over encoder states — soft alignment indicates which source words are relevant for generating the current target word. - Decoder: LSTM generates target words one at a time, conditioned on attention context + previous target word. **Transformer (2017-present)**: - Replaces recurrence with self-attention. Encoder: 6-12 layers of multi-head self-attention + feedforward. Decoder: 6-12 layers of masked self-attention + cross-attention to encoder + feedforward. - Parallelizable (all positions computed simultaneously during training). Scales to much larger models and datasets than RNN-based NMT. - The dominant NMT architecture by a large margin. **Training** - **Data**: Parallel corpora — aligned sentence pairs (source, target). WMT datasets: 10-40M sentence pairs per language pair. For low-resource languages: data augmentation (back-translation, paraphrase mining). - **Back-Translation**: Train a reverse model (target→source). Translate monolingual target-language text to source language. Use the synthetic parallel data to augment training. Dramatically improves quality — leverages abundant monolingual data. - **Subword Tokenization**: BPE (Byte-Pair Encoding) or SentencePiece. Handles rare words by splitting into common subwords. Shared vocabulary between source and target enables cross-lingual sharing. - **Label Smoothing**: Replace hard one-hot targets with soft targets (0.9 for correct token, 0.1/V distributed to others). Prevents overconfidence and improves BLEU by 0.5-1.0 points. **Decoding** - **Beam Search**: Maintain top-K hypotheses at each step (beam size 4-8). Select the highest-scoring complete translation. Without beam search, greedy decoding is 0.5-2.0 BLEU worse. - **Length Normalization**: Divide hypothesis score by length^α (α=0.6-1.0) to prevent bias toward short translations. **Multilingual NMT** - **Many-to-Many Models**: A single model translates between all pairs of N languages. Prepend a target-language tag to the source: "[FR] Hello world" → "Bonjour le monde". Shared vocabulary and shared encoder enable cross-lingual transfer. - **NLLB (No Language Left Behind, Meta)**: 200 languages, 54B parameters. Specializes with language-specific routing and expert layers. State-of-the-art for low-resource language pairs. - **Zero-Shot Translation**: If trained on English↔French and English↔German, the model can translate French↔German (never seen during training) via shared interlingual representations. Quality is lower than direct training but often usable. Neural Machine Translation is **the technology that broke the language barrier at scale** — providing the quality and coverage that enables real-time translation of web pages, messages, and documents across hundreds of languages, connecting billions of people who speak different languages.