Active Learning Selective Annotation Strategy
# Active Learning: Selective Annotation Strategy
## Introduction & Motivation
Active Learning: select most informative samples for labeling. Reduce annotation cost. Uncertainty sampling, diversity sampling. Applications: limited budget, medical imaging, rare events.
Motivation: Maximize learning efficiency; minimize labeling cost.
Applications: Limited budget annotation, data selection.
---
## Core Concepts & Theory
### Uncertainty Sampling
Select high-uncertainty predictions.
### Diversity Sampling
Representative samples; avoid redundancy.
### Query Strategy
Committee voting, expected model change.
---
## Mathematical Formulation
Uncertainty (entropy):
$$H(y|x) = -\sum_c P(y=c|x) \log P(y=c|x)$$
BALD (Bayesian Active Learning by Disagreement):
$$ ext{BALD}(x) = H(y|x) - \mathbb{E}_{w|D}[H(y|x,w)]$$
Diversity (representativeness):
$$ ext{score}(x) = -\min_{x' \in L} ext{sim}(x, x')$$
---
## Advanced Theory & Extensions
### Query by Committee
Ensemble disagreement.
### Expected Model Change
Gradient-based sample importance.
### Core-set Approach
Geometric diversity selection.
---
## Computational Considerations
Uncertainty: O(N) for predictions.
Diversity: O(N²) for pairwise similarities.
Committee: O(N·ensemble_size).
---
## Practical Implementation Strategies
### Uncertainty + Diversity
Combine both criteria.
### Batch Selection
Select multiple samples per iteration.
### Cold Start
Initialize with diverse set.
---
## Benchmark Datasets & Evaluation
MNIST, CIFAR-10: Classification benchmarks.
Medical Imaging: Real-world annotation.
Text Classification: NLP applications.
---
## Key Challenges & Limitations
### Budget vs. Improvement
Diminishing returns; stopping criteria.
### Distribution Shift
Active samples may not represent future.
### Computational Cost
Training loop overhead.
---
## Hyperparameter Tuning
Batch size (per iteration): 10-100.
Uncertainty threshold: Task-specific.
Diversity weight: Balance criteria.
---
## Real-World Applications & Case Studies
Medical Diagnosis: Annotation-efficient learning.
NLP Labeling: Text annotation reduction.
Anomaly Detection: Rare event sampling.
---
## Integration with Other Methods
Active + Semi-supervised → label efficiency.
Active + Transfer → adaptation efficiency.
---
## Summary & Key Takeaways
Active Learning via selective sample annotation enables efficient learning through uncertainty and diversity-based selection strategies.
Principles:
1. Uncertainty: high-uncertainty selection.
2. Diversity: representative sampling.
3. Query strategy: sample importance.
4. Batch selection: multiple samples.
5. Cost efficiency: minimize annotations.
---
---
## Appendix: Practical Labs
### Lab 1: Uncertainty Sampling
import numpy as np
def uncertainty_sampling(probabilities, method='entropy', num_select=10):
"""Select uncertain samples"""
if method == 'entropy':
# Shannon entropy
uncertainty = -np.sum(probabilities * np.log(probabilities + 1e-8), axis=1)
elif method == 'margin':
# Margin between top-2 predictions
sorted_probs = np.sort(probabilities, axis=1)
uncertainty = sorted_probs[:, -1] - sorted_probs[:, -2]
uncertainty = 1 - uncertainty # Invert: lower margin = higher uncertainty
else:
raise ValueError(f"Unknown method: {method}")
# Select top uncertain
uncertain_indices = np.argsort(uncertainty)[-num_select:]
return uncertain_indices, uncertainty
# Test
np.random.seed(42)
probs = np.random.rand(100, 10)
probs = probs / probs.sum(axis=1, keepdims=True)
indices, unc = uncertainty_sampling(probs, num_select=10)
assert len(indices) == 10, "Correct number selected"
assert np.all((indices >= 0) & (indices < 100)), "Valid indices"
print("✓ Uncertainty sampling working")
if __name__ == "__main__":
print("Lab 1: UncertaintySampling - PASSED")### Lab 2: Diversity Sampling
import numpy as np
def diversity_sampling(features, num_select=10, metric='euclidean'):
"""Select diverse samples using core-set approach"""
selected_indices = []
remaining_indices = list(range(len(features)))
# Start with random sample
first_idx = np.random.choice(remaining_indices)
selected_indices.append(first_idx)
remaining_indices.remove(first_idx)
# Greedily select most distant samples
for _ in range(num_select - 1):
max_min_dist = 0
best_idx = remaining_indices[0]
for idx in remaining_indices:
# Minimum distance to selected
min_dist = min(
np.linalg.norm(features[idx] - features[sel_idx])
for sel_idx in selected_indices
)
if min_dist > max_min_dist:
max_min_dist = min_dist
best_idx = idx
selected_indices.append(best_idx)
remaining_indices.remove(best_idx)
return np.array(selected_indices)
# Test
np.random.seed(42)
features = np.random.randn(100, 64)
selected = diversity_sampling(features, num_select=10)
assert len(selected) == 10, "Correct number"
assert len(np.unique(selected)) == 10, "No duplicates"
print("✓ Diversity sampling working")
if __name__ == "__main__":
print("Lab 2: DiversitySampling - PASSED")### Lab 3: BALD Criterion
import numpy as np
def bald_criterion(mc_predictions, num_select=10):
"""Bayesian Active Learning by Disagreement"""
# Mean prediction (model prediction)
mean_pred = mc_predictions.mean(axis=0)
# Predictive entropy: H(y|x)
entropy_pred = -np.sum(mean_pred * np.log(mean_pred + 1e-8), axis=1)
# Expected entropy: E_w[H(y|x,w)]
expected_entropy = -np.sum(
mc_predictions * np.log(mc_predictions + 1e-8),
axis=2
).mean(axis=0)
# BALD = difference
bald = entropy_pred - expected_entropy
# Select high BALD
selected_indices = np.argsort(bald)[-num_select:]
return selected_indices, bald
# Test
np.random.seed(42)
mc_preds = np.random.rand(50, 100, 10) # 50 MC samples, 100 data, 10 classes
mc_preds = mc_preds / mc_preds.sum(axis=2, keepdims=True)
selected, bald_vals = bald_criterion(mc_preds, num_select=10)
assert len(selected) == 10, "Correct number"
assert np.isfinite(bald_vals).all(), "Finite BALD"
print("✓ BALD working")
if __name__ == "__main__":
print("Lab 3: BALD - PASSED")### Lab 4: Active Learning Loop
import numpy as np
def active_learning_step(unlabeled_indices, predictions, method='entropy', budget=10):
"""Select samples for next labeling iteration"""
probs = predictions[unlabeled_indices]
# Compute uncertainty
uncertainty = -np.sum(probs * np.log(probs + 1e-8), axis=1)
# Select top uncertain within budget
budget = min(budget, len(unlabeled_indices))
top_uncertain = np.argsort(uncertainty)[-budget:]
# Map back to original indices
selected_indices = unlabeled_indices[top_uncertain]
return selected_indices
# Test
np.random.seed(42)
unlabeled = np.arange(100, 200)
preds = np.random.rand(200, 10)
preds = preds / preds.sum(axis=1, keepdims=True)
selected = active_learning_step(unlabeled, preds, budget=10)
assert len(selected) == 10, "Correct budget"
assert np.all((selected >= 100) & (selected < 200)), "From unlabeled"
print("✓ Active learning step working")
if __name__ == "__main__":
print("Lab 4: ActiveLearningStep - PASSED")