multi-task pre-training

**Multi-Task Pre-training** is a **learning paradigm where a model is pre-trained simultaneously on a mixture of different objectives or datasets** — rather than just one task (like MLM), the model optimizes a weighted sum of losses from multiple tasks (e.g., MLM + NSP + Translation + Summarization) to learn a more general representation. **Examples** - **T5**: Trained on a "mixture" of unsupervised denoising, translation, summarization, and classification tasks. - **MT-DNN**: Multi-Task Deep Neural Network — combines GLUE tasks during pre-training. - **UniLM**: Trained on simultaneous bidirectional, unidirectional, and seq2seq objectives. **Why It Matters** - **Generalization**: Prevents overfitting to the idiosyncrasies of a single objective. - **Transfer**: Models pre-trained on many tasks transfer better to new, unseen tasks (Meta-learning). - **Efficiency**: A single model can handle ANY task without task-specific architectural changes. **Multi-Task Pre-training** is **cross-training for AI** — practicing many different skills simultaneously to build a robust, general-purpose model.