multi-task learning parameter sharing negative transfer and representation learning

# Multi-Task Learning: Parameter Sharing, Negative Transfer, and Representation Learning

## 1. Introduction & Motivation

Multi-task learning (MTL) trains models on multiple related tasks simultaneously, leveraging shared structure to improve generalization. Rather than training separate models per task, MTL learns:

• Shared representations: Common features useful across tasks
• Task-specific components: Task-particular decision boundaries
• Inductive transfer: Knowledge from one task improves others

Applications span numerous domains:
- NLP: Multiple semantic tasks (POS tagging, NER, sentiment)
- Computer vision: Multiple visual tasks (segmentation, detection, depth)
- Healthcare: Predicting multiple related patient outcomes
- Recommendation: Multiple recommendation objectives (CTR, rating, retention)

MTL provides sample efficiency and better generalization but risks negative transfer when tasks conflict. This article comprehensively covers MTL architectures, sharing strategies, transfer mechanisms, and practical deployment.

## 2. Core Concepts & Theory

### 2.1 Task Relationships

Tasks vary in relatedness:

Positive transfer: Task A helps task B (improved B performance)

$$ ext{Perf}_B^{ ext{MTL}} > ext{Perf}_B^{ ext{single}}$$

Negative transfer: Task A hurts task B (reduced B performance)

$$ ext{Perf}_B^{ ext{MTL}} < ext{Perf}_B^{ ext{single}}$$

Understanding task relationships critical for successful MTL.

### 2.2 Hard Parameter Sharing

Simplest approach: Share lower layers across all tasks

$$ ext{Shared}(x) = f_{ ext{shared}}(x)$$
$$y_i = f_i( ext{Shared}(x))$$

Architecture: Shared backbone → task-specific heads

Intuition: Lower layers learn task-agnostic features; higher layers specialize.

Compression:

$$ O(d_{ ext{shared}} + \sum_i d_i) $$

instead of

$$ \sum_i d_{ ext{full}} $$

.

### 2.3 Soft Parameter Sharing

Tasks learn separate parameters with regularization toward shared center:

$$ ext{Loss} = \sum_i L_i( heta_i) + \lambda \sum_i \| heta_i - \bar{ heta}\|^2$$

where

$$ \bar{ heta} = \frac{1}{T} \sum_i heta_i $$

is mean parameters.

More flexible than hard sharing; allows task-specific divergence while encouraging similarity.

### 2.4 Loss Function Weighting

Balance task losses:

$$ ext{Loss}_{ ext{total}} = \sum_i w_i L_i$$

Static weighting: Fixed

$$ w_i $$

per task

Dynamic weighting: Learn

$$ w_i $$

during training

Uncertainty weighting:

$$ w_i \propto 1/ ext{Var}( ext{task}_i) $$

## 3. Mathematical Formulation

### 3.1 Hard Sharing Gradient Flow

Gradient backpropagation through shared layer:

$$\frac{\partial L}{\partial ext{Shared}} = \sum_i \frac{\partial L_i}{\partial ext{Shared}}$$

Shared layer receives gradient from all tasks. Conflicting gradients can cancel (positive) or amplify (negative transfer).

### 3.2 Soft Sharing with Task Regularization

Objective:

$$\min_{ heta_1, \ldots, heta_T} \sum_{i=1}^{T} L_i( heta_i) + \frac{\lambda}{2} \sum_{i=1}^{T} \| heta_i - \bar{ heta}\|_2^2$$

Lagrangian interpretation: Soft constraint that parameters stay near average.

More aggressive regularization (

$$ \lambda $$

larger) → more sharing.

### 3.3 Uncertainty Weighting for Task Balance

Task-dependent noise level:

$$ ext{Loss} = \sum_i \frac{1}{2\sigma_i^2} L_i + \frac{1}{2} \log \sigma_i^2$$

Learns task variances

$$ \sigma_i $$

automatically. Noisy tasks downweighted.

Equivalence to Gaussian likelihood maximization.

### 3.4 Attention-Based Task Fusion

Learn task importance per feature:

$$z = \sum_i a_i(x) \cdot z_i^{(t)}$$

where

$$ a_i(x) = ext{softmax}(f_i(x)) $$

is learned attention weight for task i.

Dynamic weighting based on input.

## 4. Advanced Theory & Extensions

### 4.1 Cross-Stitch Units

Learn connections between task-specific layers:

$$\alpha_i = W_i \mathbf{a}_i + W_j \mathbf{a}_j$$

Learnable mixing coefficient

$$ \alpha_i \in [0, 1] $$

determines information flow between tasks.

Provides fine-grained control over sharing.

### 4.2 Modulation Networks

Modulate shared features for each task:

$$x'_i = \gamma_i(x) \odot x + \beta_i(x)$$

where

$$ \gamma_i, \beta_i $$

are task-specific functions. Feature-wise adaptive scaling.

More flexible than hard sharing, less parameters than separate models.

### 4.3 Hierarchical Multi-Task Learning

Task hierarchy: Some tasks share more with each other.

Tree structure: Related tasks grouped, share within groups, less sharing across groups.

Example: Computer vision hierarchy
- Shared bottom layer (low-level features)
- Semantic-specific layers (detection vs segmentation)
- Fine-grained task-specific heads

### 4.4 Curriculum Learning in MTL

Gradually introduce tasks:

$$ ext{Task weight}_i(t) = w_i \cdot \max(0, 1 - (t - t_i) / au)$$

Early training: Easy task(s) only
Late training: All tasks

Improves convergence and final performance.

## 5. Computational Considerations

### 5.1 Computational Cost

Shared model size:

$$ O(d_{ ext{shared}} + \sum_i d_i) $$

vs separate models:

$$ \sum_i d_{ ext{full}} = T imes d_{ ext{full}} $$

Compression ratio:

$$ \approx 1 + \frac{\sum_i d_i}{T imes d_{ ext{full}}} \approx 1 + \frac{0.1 imes d_{ ext{full}}}{d_{ ext{full}}} = 1.1 $$

For

$$ T=100 $$

tasks, ~90x compression.

### 5.2 Training Time

Single-task: Time for one task

MTL: Time for T tasks sequentially =

$$ T imes $$

single-task

But: Shared learning amortizes cost. Effective time lower.

With sufficient GPU memory, train all tasks in parallel (same time as single task).

### 5.3 Gradient Magnitude Issues

Different tasks have different loss scales:

$$\frac{\partial L_1}{\partial heta} ext{ might be } 10x ext{ larger than } \frac{\partial L_2}{\partial heta}$$

Larger gradients dominate shared layer updates, harming smaller tasks.

Solution: Gradient normalization, uncertainty weighting, or batch normalization per task.

### 5.4 Memory Optimization

Shared backbone: Single copy, all tasks access

Task-specific heads: Stored once or replicated?
- Store once: Efficient memory, must handle scheduling
- Replicate: Simpler implementation, more memory

Typical: Replicate (simplicity > minor memory overhead).

## 6. Practical Implementation Strategies

### 6.1 Architecture Design

Identify shared structure:
- Which layers useful for all tasks?
- Which task-specific?

Typical pyramid:
- Base (100% shared): Low-level features
- Middle (50% shared): Mid-level patterns
- Top (0% shared): Task-specific decisions

### 6.2 Task Selection and Ordering

Task selection:
- Complementary tasks: Reduce negative transfer
- Related domains: Maximize positive transfer
- Avoid contradictory tasks

Task ordering:
- Curriculum: Easy → hard
- Frequency: Alternate tasks
- Inverse task frequency: Hard tasks more often

### 6.3 Loss Function Design

Weighting strategy:

*Fixed weights:*

$$ w_i = 1/T $$

(equal)

*Task frequency:*

$$ w_i = n_i / \sum_j n_j $$

(by dataset size)

*Uncertainty:*

$$ w_i = 1 / ext{Var}( ext{task}_i) $$

(learned)

*Dynamic:*

$$ w_i(t) = 1 + \alpha imes ext{Grad}_{norm}( ext{task}_i) $$

(gradient-based)

Typical: Start with equal, adjust if negative transfer observed.

### 6.4 Handling Negative Transfer

Detect:
- Performance drops for some task(s)
- Large gradient magnitudes
- Conflicting gradients

Mitigate:
1. Reduce task weight: Lower

$$ w_i $$

for conflicting task
2. Reduce sharing: Use soft instead of hard sharing
3. Task adaptation: Allow more task-specific parameters
4. Separate models: Resort to task-specific models if necessary

## 7. Benchmark Datasets & Evaluation

### 7.1 Multi-Task Benchmarks

NYU v2 Dataset (Vision):
- 4 tasks: Segmentation, Depth, Surface Normals, Boundaries
- 1449 RGB-D images from indoor scenes
- Shared encoder, task-specific decoders

Results:
- Single-task (avg): 39.4% mIoU (segmentation)
- MTL hard-sharing: 41.2% (+1.8%)
- MTL soft-sharing: 42.1% (+2.7%)
- Best ensemble: 43.1%

CelebA Dataset (Vision):
- 40 tasks: Binary attribute prediction
- 202K images
- Shared CNN backbone

Results:
- Single-task (avg): 88.5% accuracy
- MTL: 89.2% accuracy (+0.7%)
- Multi-headed: Best for per-attribute metrics

SQuAD + SuperGLUE (NLP):
- 9 language understanding tasks
- Shared BERT backbone, task-specific heads

Results:
- Single-task BERT: Avg 78.5%
- MTL BERT: Avg 79.8% (+1.3%)

### 7.2 Negative Transfer Analysis

Cross-Domain MTL (ImageNet classification):
- Task 1: CIFAR-10
- Task 2: STL-10
- Different scales, domains

Results:
- CIFAR alone: 95.2%
- CIFAR + STL (MTL): 94.1% (-1.1% negative transfer)
- With reduced sharing: 94.8% (mitigated)

### 7.3 Evaluation Metrics

Per-task performance: Metric for each task
- Allows identifying negative transfer
- Task-specific analysis

Average performance: Mean across tasks
-

$$ ext{AvgPerf} = \frac{1}{T} \sum_i ext{Perf}_i $$

Relative improvement: Compared to single-task
-

$$ ext{Improvement} = ( ext{MTL} - ext{Single}) / ext{Single} $$

## 8. Key Challenges & Limitations

### 8.1 Negative Transfer

MTL doesn't always help. Causes:

• Task conflict: Contradictory objectives
• Capacity constraints: Model can't learn all tasks well
• Imbalanced tasks: Large task dominates gradient

Mitigation: Task weighting, separate heads, curriculum learning.

### 8.2 Task Relationships Unknown

Prior: Don't know which tasks help which. Requires:

• Experimentation
• Domain expertise
• Automated task relation discovery

Current research: Learning task relationships from data.

### 8.3 Scalability to Many Tasks

Hard sharing problematic with T >> 1:

• Shared layer receives T different gradient signals
• Gradient conflicts increase
• Compromise solution hurts all tasks

Solution: Soft sharing, task clustering, hierarchical MTL.

### 8.4 Hyperparameter Complexity

More hyperparameters than single-task:

• Task weights

$$ w_i $$

• Sharing architecture
• Regularization strength
• Loss function design

Requires tuning; no universal configuration.

## 9. Hyperparameter Tuning & Optimization

### 9.1 Task Weighting

Fixed schemes:
- Equal:

$$ w_i = 1 $$

• Uniform loss:

$$ w_i = 1/T $$

• Inverse frequency:

$$ w_i = n_{ ext{total}}/n_i $$

(by dataset size)

Learned schemes:
- Uncertainty:

$$ w_i = 1/\sigma_i^2 $$

(learned variances)
- Gradient norm:

$$ w_i = G_i / \sum_j G_j $$

(normalized gradient norms)

Typical: Start equal, adjust if negative transfer detected.

### 9.2 Sharing Architecture

Shared layer size: Typically 60-80% of single-task

• Larger shared: More compression, risk of under-specification
• Smaller shared: Redundant parameters, less efficiency gain

Depth of sharing: How many layers shared?

• 100% sharing: All layers except final (high compression, risky)
• 50% sharing: First half shared (balance)
• 0% sharing (besides embedding): Task-specific except input

### 9.3 Soft Sharing Regularization

**Regularization strength

$$ \lambda $$

:** 0.001-0.1 typical

• Higher

$$ \lambda $$

: More sharing, less task specialization
- Lower

$$ \lambda $$

: Task independence, less transfer benefit

Tune via validation performance per task.

### 9.4 Learning Rate and Optimization

Single learning rate: Works well if tasks balanced

Per-task learning rates: More complex, sometimes better for imbalanced tasks

Gradient normalization: Clip or normalize gradients
- Prevents large-task domination
- Typical: Norm clipping to 1.0

## 10. Real-World Applications & Case Studies

### 10.1 NLP Multi-Task Fine-tuning

Problem: Fine-tune BERT on multiple text tasks

Setup:
- Tasks: Sentiment (SST), Similarity (STS), NLI (MNLI)
- Shared BERT encoder
- Task-specific classification heads
- 5K-10K training samples per task

Architecture:
- BERT backbone (frozen or fine-tuned)
- Task-specific linear layers
- Joint training with uncertainty weighting

Results:
- Single-task BERT: SST 92.2%, STS 88.5%, MNLI 82.1%
- MTL BERT: SST 93.1%, STS 89.2%, MNLI 83.4%
- Improvements: +0.9%, +0.7%, +1.3%

Deployment:
- Single model replaces 3 separate models
- 3x computational savings
- Shared knowledge improves all tasks

### 10.2 Computer Vision: Object Detection + Segmentation

Problem: Simultaneously detect objects and segment instances

Setup:
- Tasks: Object detection (classification + localization), Instance segmentation
- Shared ResNet backbone
- Task-specific heads (Faster RCNN for detection, Mask RCNN for segmentation)

Architecture:
- Shared CNN: ResNet-50
- Task 1 (Detection): RPN + classifier + bbox regressor
- Task 2 (Segmentation): Mask head
- Joint training

Results:
- Detection mAP: 40.2 (single) → 40.8 (MTL)
- Segmentation mAP: 35.1 (single) → 35.9 (MTL)
- Overall: 1-2% improvement, single model

### 10.3 Recommendation Systems: Multi-Objective Learning

Problem: Predict CTR, rating, and dwell time simultaneously

Setup:
- Tasks: Click-through rate (CTR), Rating (1-5), Dwell time
- Different objectives but related (all indicate engagement)
- Shared embedding layer
- Task-specific output layers

Architecture:
- Shared embeddings + dense layers
- Task-specific output layers
- Weighted loss:

$$ 0.5 L_{ ext{CTR}} + 0.3 L_{ ext{Rating}} + 0.2 L_{ ext{DwellTime}} $$

Results:
- CTR AUC: 0.80 (single) → 0.815 (MTL)
- Rating RMSE: 1.15 (single) → 1.10 (MTL)
- Dwell RMSE: 25.3 (single) → 24.1 (MTL)
- Business impact: +2% engagement

Key insight:
- CTR and engagement highly correlated
- Shared learning benefits both
- Better recommendation quality

### 10.4 Healthcare: Predicting Multiple Patient Outcomes

Problem: Predict mortality, readmission, LOS for ICU patients

Setup:
- Tasks: Mortality (binary), Readmission (binary), Length of Stay (regression)
- Patient data: Vitals, lab values, medications (~50 features)
- 10K+ patient trajectories

Architecture:
- Shared LSTM encoder (temporal)
- Task-specific heads
- Soft sharing + uncertainty weighting

Results:
- Mortality AUC: 0.81 (single) → 0.83 (MTL)
- Readmission AUC: 0.72 (single) → 0.75 (MTL)
- LOS RMSE: 3.2 (single) → 3.0 (MTL)

Clinical impact:
- Better predictions for clinicians
- Single model easier to deploy
- Shared knowledge improves all predictions

## 11. Integration with Other Methods

### 11.1 Meta-Learning for Task Adaptation

Learn initial weights that adapt quickly to new tasks:

$$ heta_i^* = heta - \alpha abla L_i( heta)$$

Combine with MTL: Pre-train on multiple source tasks, adapt to target task.

### 11.2 Multi-Task with Transfer Learning

Fine-tune pre-trained models on multiple tasks:
- BERT → Multiple NLP tasks
- ResNet → Multiple vision tasks

Benefits: Pre-training + multi-task transfer.

### 11.3 Domain Adaptation with MTL

Combine MTL with domain adaptation:
- Domain-invariant shared representations
- Task-specific heads
- Domain classifier (adversarial) for alignment

Improves both domain transfer and task learning.

### 11.4 Hierarchical MTL with Task Relationships

Learn task relationships:
- Task embedding space
- Soft parameter sharing based on task similarity
- Automatic discovery of task structure

## 12. Future Research Directions

### 12.1 Automatic Task Relation Discovery

Learning which tasks help which, without manual input:
- Task similarity from data
- Gradient alignment analysis
- Meta-learning approach

### 12.2 Handling Negative Transfer

Detecting and mitigating:
- Online detection of negative transfer
- Automatic weight adjustment
- Task decomposition for conflict resolution

### 12.3 Scaling to Hundreds of Tasks

Practical systems with >100 tasks:
- Efficient sharing mechanisms
- Task clustering
- Computation scheduling

### 12.4 Continual Multi-Task Learning

Adding new tasks without forgetting:
- Continual learning + MTL
- Task-incremental learning with knowledge consolidation

## 13. Summary & Key Takeaways

MTL Benefits:
- Sample efficiency: 10-30% improvement with limited data
- Model compression: 50-90% parameter reduction vs separate models
- Generalization: Shared representations often generalize better

Sharing Strategies:
- Hard sharing: Simple, 90% compression, risk of negative transfer
- Soft sharing: Flexible, 50% compression, better handles conflicts
- Attention-based: Dynamic weighting per input

Task Weighting:
- Equal weights:

$$ w_i = 1 $$

(baseline)
- Inverse frequency: By dataset size
- Uncertainty: Learns task variance automatically
- Gradient norm: Balances gradient magnitudes

Performance:
- Well-related tasks: 2-5% improvement typical
- Moderately related: 1-2% improvement
- Conflicting tasks: Negative transfer, avoid or separate

When to Use MTL:
- Multiple related tasks available
- Limited training data per task
- Seek model efficiency
- Tasks share low-level structure

When to Avoid:
- Tasks highly conflicting
- Single task most important
- No shared structure
- Separate models more efficient

Hyperparameters:
- Sharing depth: 50-100% (depend on task similarity)
- Regularization: λ = 0.01-0.1 for soft sharing
- Task weights: Start equal, adjust if negative transfer
- Architecture: Shared backbone + task-specific heads

Current Limitations:
- Negative transfer common, not well understood
- Task relationship discovery manual
- Scalability to 100+ tasks limited
- Hyperparameter tuning complex

Multi-task learning increasingly popular for practical systems. Provides meaningful benefits for related tasks; effectiveness depends critically on task selection and loss weighting. Recent work on automated task weighting and relation discovery showing promise.

---

## Appendix: Practical Implementation Labs

### Lab 1: Hard Parameter Sharing

import torch
import torch.nn as nn

class HardSharingMTL(nn.Module):
    def __init__(self, input_dim, shared_dim, num_tasks, task_dims):
        super().__init__()
        # Shared layers
        self.shared = nn.Sequential(
            nn.Linear(input_dim, shared_dim),
            nn.ReLU(),
            nn.Linear(shared_dim, shared_dim),
            nn.ReLU()
        )
        
        # Task-specific heads
        self.task_heads = nn.ModuleList([
            nn.Linear(shared_dim, task_dims[i]) for i in range(num_tasks)
        ])
    
    def forward(self, x):
        shared_rep = self.shared(x)
        task_outputs = [head(shared_rep) for head in self.task_heads]
        return task_outputs

# Training
model = HardSharingMTL(input_dim=50, shared_dim=100, num_tasks=3, 
                      task_dims=[10, 5, 1])
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

for epoch in range(100):
    outputs = model(X_batch)
    
    # Compute loss for each task
    loss_1 = criterion_1(outputs[0], y1_batch)
    loss_2 = criterion_2(outputs[1], y2_batch)
    loss_3 = criterion_3(outputs[2], y3_batch)
    
    total_loss = w1 * loss_1 + w2 * loss_2 + w3 * loss_3
    
    optimizer.zero_grad()
    total_loss.backward()
    optimizer.step()

### Lab 2: Soft Parameter Sharing

class SoftSharingMTL(nn.Module):
    def __init__(self, input_dim, hidden_dim, num_tasks, lambda_reg=0.01):
        super().__init__()
        self.num_tasks = num_tasks
        self.lambda_reg = lambda_reg
        
        # Task-specific parameters
        self.layers = nn.ModuleList([
            nn.Linear(input_dim, hidden_dim) for _ in range(num_tasks)
        ])
        self.output_layers = nn.ModuleList([
            nn.Linear(hidden_dim, 1) for _ in range(num_tasks)
        ])
    
    def forward(self, x):
        outputs = []
        for i in range(self.num_tasks):
            h = torch.relu(self.layers[i](x))
            y = self.output_layers[i](h)
            outputs.append(y)
        return outputs
    
    def regularization_loss(self):
        """Compute parameter similarity regularization"""
        # Compute mean parameters
        mean_weights = sum([self.layers[i].weight for i in range(self.num_tasks)]) / self.num_tasks
        
        reg_loss = 0
        for i in range(self.num_tasks):
            reg_loss += torch.norm(self.layers[i].weight - mean_weights) ** 2
        
        return self.lambda_reg * reg_loss

# Training
model = SoftSharingMTL(input_dim=50, hidden_dim=100, num_tasks=3, lambda_reg=0.1)

for epoch in range(100):
    outputs = model(X_batch)
    
    # Task losses
    task_losses = [criterion(outputs[i], y_batch[i]) for i in range(3)]
    total_loss = sum(task_losses) + model.regularization_loss()
    
    optimizer.zero_grad()
    total_loss.backward()
    optimizer.step()

### Lab 3: Uncertainty Weighting

class UncertaintyWeightedMTL(nn.Module):
    def __init__(self, num_tasks):
        super().__init__()
        self.log_vars = nn.Parameter(torch.zeros(num_tasks))
    
    def forward(self, task_losses):
        """Compute weighted loss with learned task variances"""
        loss = 0
        for i, task_loss in enumerate(task_losses):
            precision = torch.exp(-self.log_vars[i])
            loss += precision * task_loss + 0.5 * self.log_vars[i]
        return loss

# Training
model_mtl = HardSharingMTL(...)
uncertainty_model = UncertaintyWeightedMTL(num_tasks=3)
optimizer = torch.optim.Adam(list(model_mtl.parameters()) + 
                            list(uncertainty_model.parameters()), lr=0.001)

for epoch in range(100):
    outputs = model_mtl(X_batch)
    
    # Compute task losses
    loss_1 = criterion_1(outputs[0], y1_batch)
    loss_2 = criterion_2(outputs[1], y2_batch)
    loss_3 = criterion_3(outputs[2], y3_batch)
    
    total_loss = uncertainty_model([loss_1, loss_2, loss_3])
    
    optimizer.zero_grad()
    total_loss.backward()
    optimizer.step()

### Lab 4: Task-Weighted Ensemble

def evaluate_per_task_performance(model, test_loaders, metrics):
    """Evaluate MTL model performance per task"""
    results = {}
    
    for task_id, (loader, metric) in enumerate(zip(test_loaders, metrics)):
        task_perf = 0
        count = 0
        
        for x_batch, y_batch in loader:
            outputs = model(x_batch)
            task_output = outputs[task_id]
            task_perf += metric(task_output, y_batch).item()
            count += 1
        
        results[f'task_{task_id}'] = task_perf / count
    
    return results

def adjust_task_weights(results, current_weights):
    """Adjust weights based on performance"""
    # Lower weight if task performing well, increase if struggling
    new_weights = {}
    
    for task, perf in results.items():
        if perf > 0.9:  # Good performance
            new_weights[task] = current_weights[task] * 0.9
        elif perf < 0.7:  # Poor performance
            new_weights[task] = current_weights[task] * 1.1
        else:
            new_weights[task] = current_weights[task]
    
    # Normalize
    total = sum(new_weights.values())
    return {k: v/total for k, v in new_weights.items()}