Few-Shot Learning Low-Data Adaptation One-Shot Learning

# Few-Shot Learning: Low-Data Adaptation & One-Shot Learning

## Introduction & Motivation

Few-shot learning: learn from minimal examples per class; 1-5 samples typical. Metric learning: learn comparison function. Matching networks: attention over support set. Relation networks: learnable similarity. Applications: rapid task adaptation, limited data scenarios, cold-start problems.

Motivation: Real-world often limited per-class data. Few-shot enables learning from scarce labels.

Applications: New disease diagnosis, rare object detection, new language translation.

---

## Core Concepts & Theory

### Support and Query Sets

Support: labeled few examples per class. Query: unlabeled; test classification.

### Metric Learning

Learn embedding space; classify via distance.

### Matching Networks

Compute attention weights over support; classify as weighted average.

---

## Mathematical Formulation

N-way K-shot task:
- N classes, K examples per class in support set S
- Query set Q: unlabeled samples
- Task: classify Q samples via S

Matching Networks:
$$P(y|x, S) = \sum_{i \in S} a(x, x_i) y_i$$

attention weight a(x, x_i) normalized.

Relation Network:
$$ ext{relation}_c(x) = ext{Net}( ext{concat}(x, c))$$

learnable relation score.

---

## Advanced Theory & Extensions

### Siamese Networks

Twin encoder branches; learn similarity via contrastive loss.

### Prototypical Networks

Meta-learning; learn metric space.

### Optimization-Based

MAML, learned optimizers; rapid gradient-based adaptation.

---

## Computational Considerations

Metric learning: O(K·N·forward) distance computation.

Matching networks: O(K·N) attention; reasonable.

Relation networks: O(K·N) relation computation.

---

## Practical Implementation Strategies

### Data Augmentation

Limited data; augment support set; common practice.

### Episodic Training

Simulate few-shot tasks during training; distribution match.

### Fine-Tuning

Adapt on support set; gradient or metric-based.

---

## Benchmark Datasets & Evaluation

miniImageNet: Standard; 5-way 1-shot, 5-way 5-shot.

omniglot: Character recognition; 60-way 1-shot.

CUB: Fine-grained birds; domain adaptation benchmark.

---

## Key Challenges & Limitations

### Domain Gap

Pretraining domain ≠ few-shot domain; hurts adaptation.

### Overfitting

Few examples; high variance; regularization critical.

### Scalability

Metric computation O(K·N) per query; expensive.

---

## Hyperparameter Tuning

Support size K: 1-5 typical; more = easier.

N-way: 5-20 typical; more = harder.

Learning rate: 0.001-0.1; few-shot typically high.

---

## Real-World Applications & Case Studies

Medical Imaging: Rare disease diagnosis; few labeled scans.

Object Detection: New object category; one-shot localization.

Machine Translation: New language; few parallel sentences.

---

## Integration with Other Methods

Few-Shot + Domain Adaptation → cross-domain low-shot.

Few-Shot + Contrastive Learning → better metric space.

---

## Summary & Key Takeaways

Few-shot learning via metric learning, matching networks, and meta-learning enables classification with minimal labeled examples through learned similarity and rapid adaptation.

Principles:
1. Support/Query split: learn from few, test on many.
2. Metric learning: learn comparison function.
3. Matching: attention over support set.
4. Episodic training: simulate few-shot tasks.
5. Data augmentation: critical with few examples.

---

---

## Appendix: Practical Labs

### Lab 1: Support and Query Sets

import numpy as np
from collections import defaultdict

def create_few_shot_task(X, y, n_way=5, n_shot=1, n_query=15):
 """Create N-way K-shot task"""
 classes = np.unique(y)
 selected_classes = np.random.choice(classes, n_way, replace=False)
 
 support_X, support_y = [], []
 query_X, query_y = [], []
 
 for c in selected_classes:
 class_indices = np.where(y == c)[0]
 selected_idx = np.random.choice(class_indices, n_shot + n_query, replace=False)
 
 support_X.append(X[selected_idx[:n_shot]])
 support_y.extend([c] * n_shot)
 
 query_X.append(X[selected_idx[n_shot:n_shot + n_query]])
 query_y.extend([c] * n_query)
 
 support_X = np.vstack(support_X)
 query_X = np.vstack(query_X)
 
 return support_X, support_y, query_X, query_y

# Test
np.random.seed(42)
X = np.random.randn(500, 64)
y = np.repeat(np.arange(10), 50)

support_X, support_y, query_X, query_y = create_few_shot_task(X, y, n_way=5, n_shot=1)

assert support_X.shape == (5, 64), "Support shape correct"
assert query_X.shape == (75, 64), "Query shape correct"
assert len(support_y) == 5, "Support labels correct"
print("✓ Few-shot task creation working")

if __name__ == "__main__":
 print("Lab 1: Task - PASSED")

### Lab 2: Matching Networks

import torch
import numpy as np

def matching_networks(query_feature, support_features, support_labels, n_way):
 """Matching networks: attention-based classification"""
 # Compute attention
 query_expanded = query_feature.unsqueeze(0) # [1, dim]
 dists = torch.cdist(query_expanded, support_features)[0] # [K]
 
 # Softmax attention
 attention = torch.softmax(-dists, dim=0) # Negative distance
 
 # Weighted average over support labels
 logits = torch.zeros(n_way)
 for c in range(n_way):
 mask = torch.tensor([label == c for label in support_labels])
 logits[c] = attention[mask].sum()
 
 return logits

# Test
np.random.seed(42)
query = torch.randn(64)
support_features = torch.randn(5, 64)
support_labels = [0, 1, 2, 3, 4]

logits = matching_networks(query, support_features, support_labels, n_way=5)

assert logits.shape == (5,), "Logits shape correct"
assert abs(logits.sum().item() - 1.0) < 1e-6, "Attention weights sum to 1"
print("✓ Matching networks working")

if __name__ == "__main__":
 print("Lab 2: Matching - PASSED")

### Lab 3: Siamese Networks

import torch
import torch.nn as nn

class SiameseNetwork(nn.Module):
 def __init__(self, input_dim=784, hidden_dim=128):
 super().__init__()
 self.encoder = nn.Sequential(
 nn.Linear(input_dim, hidden_dim),
 nn.ReLU(),
 nn.Linear(hidden_dim, hidden_dim),
 nn.ReLU(),
 nn.Linear(hidden_dim, hidden_dim)
 )
 self.classifier = nn.Sequential(
 nn.Linear(hidden_dim * 2, hidden_dim),
 nn.ReLU(),
 nn.Linear(hidden_dim, 1),
 nn.Sigmoid()
 )

 def forward(self, x1, x2):
 feat1 = self.encoder(x1)
 feat2 = self.encoder(x2)
 
 combined = torch.cat([feat1, feat2], dim=1)
 similarity = self.classifier(combined)
 
 return similarity

# Test
model = SiameseNetwork()
x1 = torch.randn(8, 784)
x2 = torch.randn(8, 784)

similarity = model(x1, x2)

assert similarity.shape == (8, 1), "Similarity shape correct"
assert (similarity >= 0).all() and (similarity <= 1).all(), "Similarity in [0,1]"
print("✓ Siamese network working")

if __name__ == "__main__":
 print("Lab 3: Siamese - PASSED")

### Lab 4: Few-Shot Evaluation Metrics

import numpy as np

def compute_few_shot_accuracy(query_labels, predicted_labels):
 """Compute accuracy for few-shot task"""
 correct = np.sum(query_labels == predicted_labels)
 total = len(query_labels)
 accuracy = correct / total
 
 return accuracy

def compute_few_shot_confidence_interval(accuracies, confidence=0.95):
 """Compute CI over multiple tasks"""
 mean_acc = np.mean(accuracies)
 std_acc = np.std(accuracies)
 n_tasks = len(accuracies)
 
 # 95% CI
 margin = 1.96 * std_acc / np.sqrt(n_tasks)
 
 return mean_acc, margin

# Test
np.random.seed(42)
# Simulate 100 tasks
accuracies = np.random.uniform(0.7, 1.0, 100)

mean_acc, margin = compute_few_shot_confidence_interval(accuracies)

assert 0 <= mean_acc <= 1, "Mean accuracy in [0,1]"
assert margin > 0, "Margin positive"
assert np.isfinite(mean_acc) and np.isfinite(margin), "All finite"
print("✓ Few-shot metrics working")

if __name__ == "__main__":
 print("Lab 4: Metrics - PASSED")