multi-task learning

**Multi-Task Learning (MTL)** is a **training paradigm where a single model is trained simultaneously on multiple related tasks** — leveraging shared representations to improve generalization, reduce overfitting, and reduce the total number of parameters compared to separate task-specific models. **Core Principle** - Inductive transfer: Learning auxiliary tasks acts as regularization for the primary task. - Shared features: Tasks share a common backbone; task-specific heads branch off. - More data effective: Combining data from multiple tasks provides more training signal. **MTL Architectures** **Hard Parameter Sharing**: - Shared encoder layers + separate output heads per task. - Most common: BERT fine-tuned with [CLS] → different linear heads for classification, NER, QA. - Risk: Task interference — conflicting gradients can hurt individual tasks. **Soft Parameter Sharing**: - Each task has its own model, but parameters are regularized to be similar. - Cross-stitch networks: Learn linear combination of feature maps across tasks. - Sluice networks: Generalization of cross-stitch with learnable sharing. **Task Balancing Challenges** - Dominant task problem: High-loss task dominates gradient → others undertrained. - Solutions: - **Uncertainty weighting (Kendall et al.)**: Weight losses by learned task uncertainty. - **GradNorm**: Normalize gradient magnitudes across tasks. - **PCGrad**: Project conflicting task gradients to prevent interference. **MTL in Foundation Models** - GPT/T5: Implicitly multi-task — trained on diverse text → encodes multi-task knowledge. - Gemini: Natively multi-modal — same model for text, image, audio. - Whisper: Multi-task speech — transcription, translation, language ID, timestamps. **When MTL Helps** - Tasks share low-level features (edge detection → object detection, grammar → semantics). - Limited data for primary task — auxiliary tasks provide regularization. - Tasks have complementary data distributions. Multi-task learning is **a powerful regularization and efficiency strategy** — the shared backbone learns richer representations than any single task would produce, and foundation models trained on diverse tasks generalize far better than narrow specialists on real-world distributions.