multilingual pre-training

**Multilingual Pre-training** is the **practice of training a single model on text from many different languages simultaneously (e.g., 100 languages)** — typified by mBERT and XLM-RoBERTa, allowing the model to learn universal semantic representations that align across languages. **Mechanism** - **Data**: Concatenate Wikipedia/CommonCrawl from 100 languages. - **Tokenizer**: Use a shared sentencepiece vocabulary (typically large, e.g., 250k tokens). - **Training**: Standard MLM. No explicit parallel data (translation pairs) is strictly needed, though it helps. - **Result**: A model that can process input in Swahili, English, or Chinese without specifying the language. **Why It Matters** - **Cross-Lingual Transfer**: You can fine-tune on English labeled data and run inference on German text. - **Low-Resource Support**: High-resource languages (English) help the model learn structures that transfer to low-resource languages (Swahili). - **Simplicity**: One model to deploy instead of 100 separate models. **Multilingual Pre-training** is **the Tower of Babel solved** — creating a single polyglot model that maps all languages into a shared semantic space.