Neural Architecture Search Nas
# Neural Architecture Search (NAS)
## Introduction & Motivation
Neural Architecture Search: automatically design neural network architectures. Hyperparameter optimization; AutoML. Applications: architecture design, resource-constrained models.
Motivation: Reduce manual architecture design; find optimal configurations.
Applications: AutoML, efficient architecture discovery.
---
## Core Concepts & Theory
### Search Space
Set of possible architectures.
### Search Strategy
Evolutionary algorithms, reinforcement learning, gradient-based.
### Performance Estimation
Predict accuracy without full training.
### Early Stopping
Reduce computational cost.
---
## Mathematical Formulation
Architecture Scoring:
$$ ext{score}(A) = ext{accuracy}(A) - \lambda \cdot ext{cost}(A)$$
Evolutionary Fitness:
$$f(A) = ext{accuracy}(A) - \alpha \cdot ext{params}(A)$$
RL Reward:
$$R(A) = ext{accuracy}(A) / ext{baseline}$$
---
## Advanced Theory & Extensions
### ENAS (Efficient NAS)
Parameter sharing across architectures.
### DARTS (Differentiable Architecture Search)
Gradient-based architecture search.
### Multi-Objective Optimization
Accuracy-latency trade-offs.
---
## Computational Considerations
Random search: O(N·train_time).
Evolutionary: O(pop·gen·train_time).
DARTS: O(architecture_params·iterations).
---
## Practical Implementation Strategies
### Early Stopping
Approximate accuracy with partial training.
### Cell-Based Search
Search for cell blocks, not full architectures.
### Weight Sharing
Reuse weights across architectures.
---
## Benchmark Datasets & Evaluation
ImageNet: Classification benchmark.
CIFAR-10: Small scale evaluation.
NAS-Bench: Precomputed architecture performance.
---
## Key Challenges & Limitations
### Computational Cost
High search cost.
### Generalization
Best architecture for one dataset may not transfer.
### Search Space Design
Critical design choice.
---
## Hyperparameter Tuning
Population size: 20-100.
Search budget: Hours to thousands of GPU hours.
Mutation rate: 0.1-0.3.
---
## Real-World Applications & Case Studies
MobileNetV3: NAS for mobile efficiency.
EfficientNet: Compound scaling via NAS.
AutoML: Automated model selection.
---
## Integration with Other Methods
NAS + knowledge distillation for compact models; + pruning for extreme efficiency.
---
## Summary & Key Takeaways
Neural Architecture Search via evolutionary and gradient-based methods enables automated architecture discovery.
Principles:
1. Search space: Definable set.
2. Evaluation: Predict or train.
3. Search strategy: Evolutionary or gradient.
4. Early stopping: Cost reduction.
5. Multi-objective: Efficiency-accuracy trade-off.
---
---
## Appendix: Practical Labs
### Lab 1: Architecture Scoring
import numpy as np
def score_architecture(accuracy, num_params, lambda_param=0.001):
"""Score architecture balancing accuracy and efficiency"""
# Penalize large models
score = accuracy - lambda_param * (num_params / 1e6)
return score
# Test
accuracy = 0.92
params = 10e6
score = score_architecture(accuracy, params, lambda_param=0.0001)
assert np.isfinite(score), "Score finite"
print("✓ Architecture scoring working")
if __name__ == "__main__":
print("Lab 1: ArchitectureScoring - PASSED")### Lab 2: Evolutionary Selection
import numpy as np
def evolutionary_selection(population, fitness_scores, survival_rate=0.5):
"""Select top architectures for next generation"""
n_keep = int(len(population) * survival_rate)
# Sort by fitness (descending)
sorted_indices = np.argsort(fitness_scores)[::-1]
# Keep top architectures
selected = [population[i] for i in sorted_indices[:n_keep]]
return selected
# Test
population = [{'layers': 3}, {'layers': 5}, {'layers': 4}, {'layers': 2}]
fitness = np.array([0.85, 0.92, 0.88, 0.80])
selected = evolutionary_selection(population, fitness, survival_rate=0.5)
assert len(selected) == 2, "Correct selection size"
print("✓ Evolutionary selection working")
if __name__ == "__main__":
print("Lab 2: EvolutionarySelection - PASSED")### Lab 3: Architecture Mutation
import numpy as np
def mutate_architecture(architecture, mutation_rate=0.1):
"""Randomly mutate architecture"""
mutated = architecture.copy()
# Mutate number of layers
if np.random.rand() < mutation_rate:
mutated['num_layers'] = np.clip(mutated['num_layers'] + np.random.randint(-1, 2), 2, 10)
# Mutate hidden dimension
if np.random.rand() < mutation_rate:
mutated['hidden_dim'] = np.clip(mutated['hidden_dim'] + np.random.randint(-32, 33), 64, 512)
return mutated
# Test
arch = {'num_layers': 5, 'hidden_dim': 256}
mutated = mutate_architecture(arch, mutation_rate=1.0)
assert isinstance(mutated, dict), "Mutated is dict"
print("✓ Architecture mutation working")
if __name__ == "__main__":
print("Lab 3: ArchitectureMutation - PASSED")### Lab 4: Early Stopping
import numpy as np
def early_stop_decision(val_losses, patience=3):
"""Decide whether to stop based on validation loss"""
if len(val_losses) < patience:
return False
# Check if latest loss is best
recent_losses = val_losses[-patience:]
best_loss = min(recent_losses)
if recent_losses[-1] > best_loss:
return True
return False
# Test
losses = [0.5, 0.4, 0.35, 0.36, 0.37, 0.38]
should_stop = early_stop_decision(losses, patience=3)
assert isinstance(should_stop, bool), "Boolean decision"
print("✓ Early stopping working")
if __name__ == "__main__":
print("Lab 4: EarlyStopping - PASSED")