Neural Architecture Search Nas Automl
# Neural Architecture Search: NAS & AutoML
## Introduction & Motivation
NAS automatically designs neural network architectures. Search space: operations, connections, hyperparameters. Search strategy: reinforcement learning, evolutionary algorithms, Bayesian optimization. Efficient sampling: weight sharing (ENAS), performance prediction (NWOT). Discovers non-intuitive architectures (MobileNet, EfficientNet).
Motivation: Manually designed architectures suboptimal; optimization is combinatorial. Automate design; let data guide architecture.
Applications: Mobile deployment, transfer learning, domain-specific networks.
---
## Core Concepts & Theory
### Search Space
Cells (repeated blocks): ops ∈ {conv, pool, skip}. DAG structure defines connectivity.
### Search Strategy
Reinforcement learning: RNN controller samples architectures. Evolutionary: population, mutation. Bayesian: model performance, query next.
### Performance Estimation
Train full network: expensive. Weight sharing (ENAS): share weights, small gradient updates.
---
## Mathematical Formulation
Architecture encoding (as DAG):
Nodes: {input, ops, output}. Edges: operations connecting nodes.
Controller RNN probability:
$$P(\mathcal{A}) = \prod_{t} P( ext{op}_t | ext{prev}_t, \mathcal{A}_{1:t-1}; heta_c)$$
ENAS loss:
$$\mathcal{L}_{ ext{ENAS}} = \mathcal{L}_{ ext{task}}( heta_w, \mathcal{A}) + \lambda \mathcal{L}_c( heta_c)$$
---
## Advanced Theory & Extensions
### Efficient NAS
Early stopping: predict performance early; discard unpromising.
### Multi-Objective NAS
Optimize accuracy + latency + memory jointly. Pareto frontier.
### Transferable NAS
NAS on proxy task; transfer to target.
---
## Computational Considerations
Random search: O(N_searches × train_time).
ENAS: O(N_searches × gradient_steps) (shared weights).
Bayesian: O(N_searches × (model update + acquisition)).
Evolutionary: O(N_population × generations × train_time).
---
## Practical Implementation Strategies
### Early Stopping
Monitor validation performance; stop unpromising architectures.
### Supernet Training
Share weights across all candidate architectures; efficient fine-tuning.
### Hardware-Aware Search
Include latency, energy as objectives; deploy to target devices.
---
## Benchmark Datasets & Evaluation
ImageNet: Standard; CIFAR-10 as proxy (faster).
DARTS Benchmark: 20 hours GPU training; reproducible.
Metrics: Top-1 accuracy, latency (ms), energy (mJ).
---
## Key Challenges & Limitations
### Computational Cost
Full NAS expensive (1000s of GPU hours); approximations trade accuracy.
### Transfer Across Tasks
Architecture optimized for ImageNet may not transfer.
### Reproducibility
Stochastic; variance in results.
---
## Hyperparameter Tuning
Search budget: 50-500 architecture evaluations.
Early stopping epochs: 5-20 (proxy task).
Population size (evolutionary): 20-100.
---
## Real-World Applications & Case Studies
MobileNet: NAS for mobile; 50% smaller, 30% faster.
EfficientNet: Scale network depth/width/resolution jointly.
Vision Transformer NAS: Automated transformer design.
---
## Integration with Other Methods
NAS + Transfer → pretrain on large, NAS on target.
NAS + Distillation → compress discovered architecture.
---
## Summary & Key Takeaways
Neural Architecture Search automates network design via search space, strategy (RL/evolution/Bayesian), and efficient sampling (weight sharing, early stopping).
Principles:
1. Search space: cell-based DAG architecture.
2. Strategy: RL (controller), evolution, Bayesian optimization.
3. ENAS: weight sharing reduces computation.
4. Early stopping, supernets accelerate search.
5. Multi-objective: accuracy vs. latency.
---
---
## Appendix: Practical Labs
### Lab 1: Simple Architecture Encoding
import numpy as np
class ArchitectureEncoding:
def __init__(self, n_cells=5, n_ops=3):
self.n_cells = n_cells
self.n_ops = n_ops
def random_architecture(self):
"""Generate random architecture"""
arch = np.random.randint(0, self.n_ops, size=self.n_cells)
return arch
def mutate(self, arch):
"""Mutate one operation"""
arch_mut = arch.copy()
idx = np.random.randint(len(arch))
arch_mut[idx] = np.random.randint(0, self.n_ops)
return arch_mut
encoder = ArchitectureEncoding(n_cells=5, n_ops=3)
arch1 = encoder.random_architecture()
arch2 = encoder.mutate(arch1)
print(f"Original arch: {arch1}")
print(f"Mutated arch: {arch2}")
assert len(arch1) == 5, "Should have 5 cells"
assert len(arch2) == 5, "Should have 5 cells"
print("✓ Architecture encoding working")
if __name__ == "__main__":
print("Lab 1: Encoding - PASSED")### Lab 2: Architecture Performance Prediction
import numpy as np
from sklearn.linear_model import LinearRegression
def predict_architecture_performance(architectures, known_performances, new_arch):
"""Predict performance of new architecture from neighbors"""
# Simple: use average of k-nearest architectures
distances = np.linalg.norm(architectures - new_arch, axis=1)
k = min(3, len(architectures))
nearest_idx = np.argsort(distances)[:k]
predicted_perf = known_performances[nearest_idx].mean()
return predicted_perf
# Data: architectures and their accuracies
archs = np.array([[1, 0, 2, 1, 0],
[2, 1, 0, 1, 2],
[0, 2, 1, 0, 1],
[1, 1, 2, 2, 0]])
perfs = np.array([0.85, 0.88, 0.83, 0.87])
new_arch = np.array([1, 0, 2, 1, 1])
pred_perf = predict_architecture_performance(archs, perfs, new_arch)
print(f"Predicted performance: {pred_perf:.3f}")
assert 0.80 <= pred_perf <= 0.90, "Prediction should be in range"
print("✓ Performance prediction working")
if __name__ == "__main__":
print("Lab 2: Prediction - PASSED")### Lab 3: Evolutionary Search
import numpy as np
def evaluate_architecture(arch):
"""Dummy evaluation: higher op sum = better (simplified)"""
return (arch.sum() + np.random.randn()) / len(arch)
def evolutionary_search(n_pop=10, n_gen=5, n_cells=5, n_ops=3):
"""Simple evolutionary NAS"""
# Initialize population
population = [np.random.randint(0, n_ops, size=n_cells) for _ in range(n_pop)]
fitnesses = [evaluate_architecture(arch) for arch in population]
history = []
for gen in range(n_gen):
# Select top performers
top_indices = np.argsort(fitnesses)[-n_pop//2:]
elite = [population[i] for i in top_indices]
# Mutate
new_pop = elite[:]
for _ in range(n_pop - len(elite)):
parent = elite[np.random.randint(len(elite))]
child = parent.copy()
idx = np.random.randint(len(child))
child[idx] = np.random.randint(0, n_ops)
new_pop.append(child)
population = new_pop
fitnesses = [evaluate_architecture(arch) for arch in population]
best_fit = max(fitnesses)
history.append(best_fit)
return history, population[np.argmax(fitnesses)]
history, best_arch = evolutionary_search(n_pop=10, n_gen=5)
print(f"Best fitness history: {[f'{f:.3f}' for f in history]}")
assert len(history) == 5, "Should have 5 generations"
assert len(best_arch) == 5, "Best arch should have 5 cells"
print("✓ Evolutionary search working")
if __name__ == "__main__":
print("Lab 3: Evolution - PASSED")### Lab 4: Architecture Comparison
import numpy as np
def compare_architectures(arch_list, performance_list):
"""Compare discovered vs. baseline architectures"""
baseline = np.mean(performance_list)
best_idx = np.argmax(performance_list)
improvement = (performance_list[best_idx] - baseline) / baseline * 100
return {
'baseline': baseline,
'best': performance_list[best_idx],
'improvement': improvement,
'best_arch_idx': best_idx
}
# Simulated: random architectures vs. discovered
random_archs = np.random.randint(0, 3, size=(5, 5))
random_perfs = np.random.uniform(0.80, 0.85, 5)
# Discovered (better)
discovered_perfs = np.array([0.88, 0.90, 0.85, 0.92, 0.87])
all_perfs = np.concatenate([random_perfs, discovered_perfs])
results = compare_architectures(range(len(all_perfs)), all_perfs)
print(f"Baseline accuracy: {results['baseline']:.3f}")
print(f"Best accuracy: {results['best']:.3f}")
print(f"Improvement: {results['improvement']:.1f}%")
assert results['improvement'] > 0, "Should find improvement"
print("✓ Architecture comparison working")
if __name__ == "__main__":
print("Lab 4: Comparison - PASSED")