Multi-Task Learning (MTL) is the training paradigm where a single neural network is trained simultaneously on multiple related tasks (classification, detection, segmentation, depth estimation, etc.) with shared representations — improving generalization by leveraging the inductive bias that related tasks share common features, reducing overfitting on any single task, and enabling efficient deployment where one model replaces many task-specific models at a fraction of the total compute and memory cost.

Why Multi-Task Learning Works

• Implicit Data Augmentation: Each task provides a different view of the same data. Learning to predict depth and surface normals simultaneously forces features to capture 3D structure that benefits both tasks.
• Regularization: Shared parameters are constrained by multiple loss functions — harder to overfit to any single task's noise.
• Feature Sharing: Low-level features (edges, textures, shapes) are universal across vision tasks. Sharing these features across tasks avoids redundant computation and enables richer representations.

Architecture Patterns

Hard Parameter Sharing:

• Shared encoder (backbone), task-specific heads (decoders).
• Example: ResNet-50 shared backbone → classification head (FC + softmax), detection head (FPN + RPN + ROI), segmentation head (upsampling + per-pixel classifier).
• Advantage: Simple, parameter-efficient, strong regularization.
• Risk: Negative transfer — if tasks conflict, shared features compromise both tasks.

Soft Parameter Sharing:

• Each task has its own network, but parameters are regularized to be similar (L2 penalty on weight differences, or cross-stitch networks that learn linear combinations of task features).
• More flexible: tasks can learn distinct features where needed while sharing where beneficial.
• Cost: More parameters, more memory.

Loss Balancing

The total loss L = Σᵢ wᵢ × Lᵢ requires careful balancing of task weights wᵢ:

• Fixed Weights: Manually tuned. Fragile — different tasks have different loss scales and convergence rates.
• Uncertainty Weighting (Kendall et al.): Learn task weights based on homoscedastic uncertainty. Each weight is 1/(2σᵢ²) where σᵢ is a learned parameter. Tasks with higher uncertainty (harder tasks) receive lower weight — prevents hard tasks from dominating training.
• GradNorm: Dynamically adjust weights so that all tasks train at similar rates. Monitors gradient norms of each task's loss w.r.t. shared parameters and adjusts weights to equalize them.
• PCGrad (Project Conflicting Gradients): When task gradients conflict (negative cosine similarity), project one task's gradient onto the normal plane of the other. Prevents tasks from undoing each other's progress.

Applications

• Autonomous Driving: Detect objects + estimate depth + predict lane lines + segment drivable area — all from a shared backbone processing a single camera image. Tesla HydraNet processes 8 cameras with a shared backbone and 48 task-specific heads.
• NLP: Sentiment analysis + NER + POS tagging + parsing — shared transformer encoder, task-specific classification heads.
• Recommendation: Click prediction + conversion prediction + dwell time prediction — shared user/item embeddings, task-specific prediction towers.

Multi-Task Learning is the efficiency and generalization paradigm that replaces N separate models with one shared model — leveraging the insight that real-world tasks share structure, and correctly exploiting that structure produces representations superior to what any single task could learn alone.