Domain Adaptation Addressing Domain Shift Transfer
# Domain Adaptation: Addressing Domain Shift & Transfer
## Introduction & Motivation
Domain adaptation: source and target distributions differ. Unsupervised domain adaptation: no target labels. Adversarial domain adaptation: adversarial feature alignment. Maximum mean discrepancy: distribution distance metric. Self-training: pseudo-labels on target. Applications: zero-cost deployment, handling distribution shift, cross-dataset generalization.
Motivation: Models trained on source fail on target (domain shift). Adaptation aligns distributions; improves target performance.
Applications: Cross-camera adaptation, sim-to-real robotics, cross-domain recognition.
---
## Core Concepts & Theory
### Domain Shift
Covariate shift: P(X) differs. Label shift: P(Y) differs.
### Adversarial Alignment
Adversarial discriminator distinguishes domains; features domain-invariant.
### Maximum Mean Discrepancy (MMD)
Compute distance between distributions in RKHS.
---
## Mathematical Formulation
Maximum Mean Discrepancy:
$$ ext{MMD}^2 = \|\mathbb{E}_s[\phi(x_s)] - \mathbb{E}_t[\phi(x_t)]\|^2_H$$
measure distance in feature space.
Adversarial loss:
$$L = L_{ ext{task}} - \lambda L_{ ext{adversarial}}$$
feature extractor maximizes domain confusion.
Self-training:
$$L_{ ext{target}} = ext{CrossEntropy}(y_{ ext{pseudo}}, \hat{y}_{ ext{target}})$$
pseudo-labels from model predictions on target.
---
## Advanced Theory & Extensions
### Partial Domain Adaptation
Target classes subset of source classes.
### Open Set Domain Adaptation
Target contains unknown classes.
### Multi-source Domain Adaptation
Multiple sources; weighted alignment.
---
## Computational Considerations
MMD: O(n_s·n_t) pairwise distances.
Adversarial: O(forward + backward × 2) networks.
Self-training: O(forward) pseudo-labeling.
---
## Practical Implementation Strategies
### Adversarial Weight
λ controlling domain alignment; typically 0.1-1.0.
### Pseudo-Label Threshold
Confidence threshold; only high-confidence.
### Batch Composition
Mix source and target; domain diversity.
---
## Benchmark Datasets & Evaluation
Office-31: Standard benchmark; 3 domains, 31 classes.
VisDA: Large-scale; synthetic-to-real adaptation.
DomainNet: Multi-source; 6 domains, 345 classes.
---
## Key Challenges & Limitations
### Negative Transfer
Target-irrelevant source; hurts performance.
### Partial Domain Adaptation
Handles unknown target classes; harder problem.
### Label Noise
Pseudo-labels noisy; self-training hurts.
---
## Hyperparameter Tuning
λ (adversarial weight): 0.1-1.0; empirical.
Confidence threshold: 0.8-0.95; pseudo-label quality.
Batch ratio (source:target): 1:1 typical.
---
## Real-World Applications & Case Studies
Sim-to-Real: Simulation domain → real domain transfer.
Cross-Camera: Camera A → Camera B adaptation.
Autonomous Driving: Different cities, weather; domain adaptation.
---
## Integration with Other Methods
Domain Adaptation + Self-Supervised → double alignment.
Domain Adaptation + Ensemble → diverse domain models.
---
## Summary & Key Takeaways
Domain adaptation via adversarial alignment, MMD, and self-training addresses distribution shift, enabling robust cross-domain transfer.
Principles:
1. Covariate shift: align P(X) via adversarial.
2. MMD: measure distribution distance.
3. Adversarial: domain confusion via discriminator.
4. Self-training: pseudo-labels on target.
5. Partial DA: handle target class subset.
---
---
## Appendix: Practical Labs
### Lab 1: Maximum Mean Discrepancy
import numpy as np
def compute_mmd(X_s, X_t, kernel='rbf', sigma=1.0):
"""Compute MMD between source and target"""
def rbf_kernel(x1, x2, sigma):
dists = np.sum((x1[:, np.newaxis, :] - x2[np.newaxis, :, :]) ** 2, axis=2)
return np.exp(-dists / (2 * sigma ** 2))
n_s, n_t = len(X_s), len(X_t)
K_ss = rbf_kernel(X_s, X_s, sigma)
K_tt = rbf_kernel(X_t, X_t, sigma)
K_st = rbf_kernel(X_s, X_t, sigma)
mmd = np.mean(K_ss) - 2 * np.mean(K_st) + np.mean(K_tt)
return np.sqrt(max(mmd, 0))
# Test
np.random.seed(42)
X_s = np.random.randn(50, 20)
X_t = np.random.randn(50, 20)
mmd = compute_mmd(X_s, X_t)
assert mmd >= 0, "MMD should be non-negative"
assert np.isfinite(mmd), "MMD should be finite"
print("✓ MMD computation working")
if __name__ == "__main__":
print("Lab 1: MMD - PASSED")### Lab 2: Adversarial Domain Adaptation
import torch
import torch.nn as nn
import numpy as np
class DomainAdversarialModel(nn.Module):
def __init__(self, input_dim=20, hidden_dim=64):
super().__init__()
self.feature_extractor = nn.Sequential(
nn.Linear(input_dim, hidden_dim),
nn.ReLU()
)
self.classifier = nn.Linear(hidden_dim, 10)
self.domain_discriminator = nn.Sequential(
nn.Linear(hidden_dim, 64),
nn.ReLU(),
nn.Linear(64, 1),
nn.Sigmoid()
)
def forward(self, x):
features = self.feature_extractor(x)
class_out = self.classifier(features)
domain_out = self.domain_discriminator(features)
return class_out, domain_out, features
# Test
model = DomainAdversarialModel()
X_s = torch.randn(32, 20)
X_t = torch.randn(32, 20)
class_s, domain_s, feat_s = model(X_s)
class_t, domain_t, feat_t = model(X_t)
assert class_s.shape == (32, 10), "Class output shape correct"
assert domain_s.shape == (32, 1), "Domain output shape correct"
assert feat_s.shape == (32, 64), "Feature shape correct"
print("✓ Adversarial DA working")
if __name__ == "__main__":
print("Lab 2: AdversarialDA - PASSED")### Lab 3: Self-Training with Pseudo-Labels
import numpy as np
def self_training_pseudo_labels(model, X_target, confidence_threshold=0.8):
"""Generate pseudo-labels via self-training"""
predictions = model.predict(X_target)
confidences = predictions.max(axis=1)
# Filter by confidence
mask = confidences >= confidence_threshold
pseudo_labels = predictions[mask].argmax(axis=1)
pseudo_X = X_target[mask]
return pseudo_X, pseudo_labels, mask
# Test
np.random.seed(42)
class DummyModel:
def predict(self, X):
# Simulate model predictions
return np.random.dirichlet([1]*10, len(X))
model = DummyModel()
X_target = np.random.randn(100, 20)
pseudo_X, pseudo_y, mask = self_training_pseudo_labels(model, X_target, threshold=0.8)
assert len(pseudo_X) <= len(X_target), "Should filter samples"
assert len(pseudo_y) == len(pseudo_X), "Labels match samples"
print("✓ Self-training working")
if __name__ == "__main__":
print("Lab 3: SelfTraining - PASSED")### Lab 4: Domain Adaptation Evaluation
import numpy as np
def evaluate_domain_adaptation(y_true_t, y_pred_t):
"""Evaluate target domain accuracy"""
accuracy = np.mean(y_true_t == y_pred_t)
# Per-class accuracy
classes = np.unique(y_true_t)
class_accs = {}
for c in classes:
mask = y_true_t == c
class_accs[c] = np.mean(y_true_t[mask] == y_pred_t[mask])
return accuracy, class_accs
# Test
np.random.seed(42)
y_true = np.random.randint(0, 5, 100)
y_pred = y_true.copy()
y_pred[np.random.choice(100, 20, replace=False)] = np.random.randint(0, 5, 20)
acc, class_accs = evaluate_domain_adaptation(y_true, y_pred)
assert 0 <= acc <= 1, "Accuracy in [0,1]"
assert len(class_accs) == 5, "Should have 5 classes"
print("✓ DA evaluation working")
if __name__ == "__main__":
print("Lab 4: Evaluation - PASSED")