neural architecture search nas automl
# Neural Architecture Search: NAS & AutoML
## Introduction & Motivation
Neural Architecture Search: automatically design architectures. NAS: search space, strategy, evaluation. Reinforcement learning NAS: ENAS, controller. Differentiable NAS: DARTS. Applications: model design, AutoML, architecture discovery.
Motivation: Manual design time-consuming; expertise required. Automated search finds novel architectures.
Applications: AutoML, model optimization.
---
## Core Concepts & Theory
### Search Space
Architecture components; building blocks.
### Search Strategy
Reinforcement learning, evolutionary, differentiable.
### Evaluation Strategy
Accuracy proxy; training cost reduction.
---
## Mathematical Formulation
Search objective:
$$ ext{Architecture}^* = \arg\max_a ext{Accuracy}(a) - \lambda \cdot ext{Cost}(a)$$
DARTS continuous relaxation:
$$\bar{o}^{(i,j)}(x) = \sum_{o \in \mathcal{O}} \frac{\exp(\alpha_o^{(i,j)})}{\sum_{o' \in \mathcal{O}} \exp(\alpha_{o'}^{(i,j)})} o(x)$$
---
## Advanced Theory & Extensions
### Predictor-Based NAS
Learn predictor of architecture performance.
### Weight Sharing
Share weights across architectures; ENAS.
### Multi-Objective Optimization
Pareto frontier; accuracy vs. efficiency.
---
## Computational Considerations
Grid Search: O(G^D) exponential; infeasible.
Evolutionary: O(P·G·T) population-based.
DARTS: O(D·T·U) differentiable; efficient.
---
## Practical Implementation Strategies
### Search Space Design
Meaningful blocks; reduction complexity.
### Early Stopping
Performance prediction; stop poor candidates.
### Transfer Learning
Initialize from pretrained weights.
---
## Benchmark Datasets & Evaluation
NASNet: ImageNet; state-of-the-art.
Efficient-Net: Pareto-optimal; various scales.
NAS Benchmarks: Standardized evaluation.
---
## Key Challenges & Limitations
### Search Cost
NAS expensive; GPU weeks.
### Reproducibility
Stochasticity; difficult to reproduce.
### Transfer
Optimal architecture task-specific.
---
## Hyperparameter Tuning
Search space size: Balance coverage and complexity.
Search budget: GPU-hours; cost constraint.
Evaluation accuracy: Proxy vs. full training.
---
## Real-World Applications & Case Studies
AutoML: Automatic model design.
Mobile: Efficient architecture; MobileNets.
Specialized: Domain-specific optimization.
---
## Integration with Other Methods
NAS + Transfer → Architecture adaptation.
NAS + Ensemble → Diverse models.
---
## Summary & Key Takeaways
Neural Architecture Search via NAS and AutoML enables automatic architecture discovery through search strategies and performance evaluation.
Principles:
1. Search space: design flexibility.
2. Strategy: RL, evolutionary, differentiable.
3. Evaluation: accuracy-cost tradeoff.
4. Weight sharing: efficiency.
5. Transferability: architecture generalization.
---
---
## Appendix: Practical Labs
### Lab 1: Search Space Definition
import numpy as np
from itertools import product
def generate_search_space(options_per_layer=3, num_layers=3):
"""Generate NAS search space"""
# Simple search space: kernel sizes and channels per layer
kernel_sizes = [1, 3, 5]
channels = [32, 64, 128]
architectures = []
for config in product(
*[[kernel_sizes, channels] for _ in range(num_layers)]
):
arch = []
for i in range(0, len(config), 2):
arch.append({"kernel": config[i], "channels": config[i+1]})
architectures.append(arch)
return architectures[:options_per_layer ** num_layers] # Limit
# Test
np.random.seed(42)
archs = generate_search_space(options_per_layer=3, num_layers=2)
assert len(archs) > 0, "Architectures generated"
print("✓ Search space generation working")
if __name__ == "__main__":
print("Lab 1: SearchSpace - PASSED")### Lab 2: Architecture Evaluation
import numpy as np
def estimate_architecture_performance(architecture, dataset_size=50000):
"""Estimate accuracy from architecture"""
# Proxy: depth + complexity correlation
depth = len(architecture)
complexity = sum(layer["channels"] for layer in architecture)
# Simplified performance model
base_accuracy = 0.7
depth_bonus = 0.01 * min(depth, 5) # Diminishing returns
complexity_penalty = 0.001 * complexity / 100
estimated_accuracy = base_accuracy + depth_bonus - complexity_penalty
return np.clip(estimated_accuracy, 0, 1)
# Test
np.random.seed(42)
architecture = [
{"kernel": 3, "channels": 32},
{"kernel": 3, "channels": 64},
{"kernel": 3, "channels": 128}
]
accuracy = estimate_architecture_performance(architecture)
assert 0 <= accuracy <= 1, "Accuracy in [0,1]"
print("✓ Architecture evaluation working")
if __name__ == "__main__":
print("Lab 2: ArchEvaluation - PASSED")### Lab 3: Random Search NAS
import numpy as np
def random_search_nas(num_trials=10):
"""Random architecture search"""
best_architecture = None
best_accuracy = 0
for trial in range(num_trials):
# Random architecture
num_layers = np.random.randint(2, 5)
architecture = []
for _ in range(num_layers):
layer = {
"kernel": np.random.choice([1, 3, 5]),
"channels": np.random.choice([32, 64, 128, 256])
}
architecture.append(layer)
# Evaluate
accuracy = 0.7 + np.random.rand() * 0.2 # Simplified
if accuracy > best_accuracy:
best_accuracy = accuracy
best_architecture = architecture
return best_architecture, best_accuracy
# Test
np.random.seed(42)
best_arch, best_acc = random_search_nas(num_trials=20)
assert best_arch is not None, "Architecture found"
assert 0 <= best_acc <= 1, "Accuracy valid"
print("✓ Random search NAS working")
if __name__ == "__main__":
print("Lab 3: RandomNAS - PASSED")### Lab 4: NAS Performance Prediction
import numpy as np
def predict_architecture_accuracy(architecture, weights):
"""Predict accuracy from architecture features"""
# Extract features
depth = len(architecture)
avg_channels = np.mean([layer["channels"] for layer in architecture])
avg_kernel = np.mean([layer["kernel"] for layer in architecture])
features = np.array([depth, avg_channels, avg_kernel])
# Predict: simple linear model
bias = weights[0]
pred = bias + np.dot(features, weights[1:])
return np.clip(pred, 0, 1)
# Test
np.random.seed(42)
architecture = [
{"kernel": 3, "channels": 64},
{"kernel": 3, "channels": 128}
]
weights = np.array([0.5, 0.01, 0.001, 0.01]) # Learned weights
accuracy = predict_architecture_accuracy(architecture, weights)
assert 0 <= accuracy <= 1, "Predicted accuracy in [0,1]"
print("✓ NAS prediction working")
if __name__ == "__main__":
print("Lab 4: NASPrediction - PASSED")