Multi-Task Learning Shared Representations Task Weighting
# Multi-Task Learning: Shared Representations & Task Weighting
## Introduction & Motivation
Multi-task learning: train on multiple tasks jointly. Shared representations: earlier layers learn task-generic features. Task-specific heads: later layers specialize. Soft parameter sharing: related task interaction. Hard parameter sharing: full weight sharing. Applications: reduce overfitting, improve sample efficiency, transfer learning.
Motivation: Individual tasks limited data. Joint training leverages cross-task signal; shared representations improve generalization.
Applications: NLP (POS, NER, parsing), vision (detection, classification), medical (multiple diagnoses).
---
## Core Concepts & Theory
### Shared Encoder
Common early layers; extract task-agnostic features.
### Task-Specific Heads
Separate heads per task; specialize for task objective.
### Task Weighting
Scale losses; balance task contributions; learned or fixed.
---
## Mathematical Formulation
Multi-task loss:
$$L_{ ext{total}} = \sum_t w_t L_t(y_t, \hat{y}_t)$$
where w_t = task weight, L_t = task-specific loss.
Shared encoder:
$$\phi_{ ext{shared}} = ext{Encoder}(x)$$
Task-specific outputs:
$$\hat{y}_t = ext{Head}_t(\phi_{ ext{shared}})$$
Learned task weights (uncertainty):
$$w_t = \frac{1}{\sigma_t^2}, \quad L_{ ext{weighted}} = \sum_t \frac{1}{\sigma_t^2} L_t + \log \sigma_t$$
---
## Advanced Theory & Extensions
### Attention Mechanisms
Attention over tasks; dynamic feature specialization per task.
### Cross-Stitch Networks
Learned combination of task-specific and shared features.
### Progressive Neural Networks
Sequential task adaptation; avoid catastrophic forgetting.
---
## Computational Considerations
Training: O(T·forward + T·backward) for T tasks.
Inference: O(forward + T·task_heads) per sample.
Memory: O(shared + T·head_params).
---
## Practical Implementation Strategies
### Task Selection
Choose related tasks; unrelated tasks hurt generalization.
### Task Weighting
Data size: large weight to small-data tasks. Domain: empirical.
### Gradient Normalization
Normalize gradients per task; prevent dominance.
---
## Benchmark Datasets & Evaluation
MNIST Multi-Digit: Multiple digits per image; joint learning better.
NLP (GLUE): Multiple tasks; shared encoder improves performance.
Visual Domain: Multiple outputs (classification + detection); complementary.
---
## Key Challenges & Limitations
### Negative Transfer
Unrelated tasks; shared representation hurts. Task selection critical.
### Task Interference
Gradient conflicts; task weights or gradient normalization help.
### Hyperparameter Tuning
Task weights, architecture; many hyperparameters.
---
## Hyperparameter Tuning
Task weights: Inverse data size common; or learn via uncertainty.
Architecture: Shared depth: 60-70% total; task-specific: 30-40%.
Learning rate: Same LR for all tasks; decay together.
---
## Real-World Applications & Case Studies
BERT: Masked language + next sentence prediction; shared encoder.
Multi-Output Vision: Classification + localization + segmentation.
Medical: Multiple diagnoses; shared encoder improves robustness.
---
## Integration with Other Methods
MTL + Uncertainty → learned task weighting.
MTL + Regularization → shared representation regularization.
---
## Summary & Key Takeaways
Multi-task learning via shared encoders and task-specific heads improves sample efficiency and generalization through cross-task regularization.
Principles:
1. Shared encoder: task-generic early representations.
2. Task-specific heads: specialize for objectives.
3. Task weighting: balance gradient contributions.
4. Related tasks benefit; unrelated hurt (negative transfer).
5. Uncertainty weighting: learn task importance.
---
---
## Appendix: Practical Labs
### Lab 1: Multi-Task Architecture
import torch
import torch.nn as nn
class MultiTaskModel(nn.Module):
def __init__(self, input_dim=100, shared_dim=64, task1_dim=10, task2_dim=5):
super().__init__()
# Shared encoder
self.shared = nn.Sequential(
nn.Linear(input_dim, 128),
nn.ReLU(),
nn.Linear(128, shared_dim),
nn.ReLU()
)
# Task 1 head
self.head1 = nn.Linear(shared_dim, task1_dim)
# Task 2 head
self.head2 = nn.Linear(shared_dim, task2_dim)
def forward(self, x):
shared_feat = self.shared(x)
task1_out = self.head1(shared_feat)
task2_out = self.head2(shared_feat)
return task1_out, task2_out
# Test
model = MultiTaskModel()
x = torch.randn(32, 100)
out1, out2 = model(x)
assert out1.shape == (32, 10), "Task 1 output shape correct"
assert out2.shape == (32, 5), "Task 2 output shape correct"
print("✓ Multi-task architecture working")
if __name__ == "__main__":
print("Lab 1: Architecture - PASSED")### Lab 2: Multi-Task Loss
import torch
import torch.nn.functional as F
import numpy as np
def multitask_loss(out1, out2, y1, y2, w1=1.0, w2=1.0):
"""Compute weighted multi-task loss"""
loss1 = F.mse_loss(out1, y1)
loss2 = F.mse_loss(out2, y2)
total_loss = w1 * loss1 + w2 * loss2
return total_loss, loss1.item(), loss2.item()
# Test
np.random.seed(42)
out1 = torch.randn(32, 10, requires_grad=True)
out2 = torch.randn(32, 5, requires_grad=True)
y1 = torch.randn(32, 10)
y2 = torch.randn(32, 5)
total, l1, l2 = multitask_loss(out1, out2, y1, y2, w1=1.0, w2=0.5)
assert total > 0, "Loss should be positive"
assert abs(total - (l1 + 0.5 * l2)) < 1e-5, "Loss computation correct"
print("✓ Multi-task loss working")
if __name__ == "__main__":
print("Lab 2: Loss - PASSED")### Lab 3: Task Weighting
import numpy as np
def compute_task_weights(task_losses):
"""Compute task weights inversely proportional to loss"""
task_losses = np.array(task_losses)
weights = 1.0 / (task_losses + 1e-8)
weights = weights / weights.sum() * len(weights)
return weights
# Test
np.random.seed(42)
losses = [0.5, 2.0, 0.1]
weights = compute_task_weights(losses)
assert len(weights) == 3, "Should have 3 weights"
assert abs(weights.sum() - 3.0) < 1e-6, "Weights should sum to 3"
assert weights[2] > weights[1], "Lower loss should have higher weight"
print("✓ Task weighting working")
if __name__ == "__main__":
print("Lab 3: Weighting - PASSED")### Lab 4: Uncertainty Weighting
import torch
import numpy as np
class UncertaintyWeightedMTL:
def __init__(self, n_tasks=2):
self.log_vars = torch.nn.Parameter(torch.zeros(n_tasks))
def forward(self, losses):
"""Compute uncertainty-weighted loss"""
weighted_losses = []
for i, loss in enumerate(losses):
weighted_loss = torch.exp(-self.log_vars[i]) * loss + 0.5 * self.log_vars[i]
weighted_losses.append(weighted_loss)
return sum(weighted_losses)
# Test
np.random.seed(42)
uw_mtl = UncertaintyWeightedMTL(n_tasks=2)
loss1 = torch.tensor(1.0)
loss2 = torch.tensor(0.5)
total_loss = uw_mtl.forward([loss1, loss2])
assert total_loss > 0, "Total loss positive"
assert torch.isfinite(total_loss), "Loss should be finite"
print("✓ Uncertainty weighting working")
if __name__ == "__main__":
print("Lab 4: Uncertainty - PASSED")