nas
**Neural Architecture Search (NAS)**
**What is NAS?**
Automated process of discovering optimal neural network architectures for given tasks, replacing manual architecture design.
**NAS Components**
**Search Space**
Define what architectures are possible:
```python
search_space = {
"num_layers": [4, 6, 8, 12],
"hidden_size": [256, 512, 768, 1024],
"num_heads": [4, 8, 12],
"activation": ["relu", "gelu", "swish"],
"dropout": [0.0, 0.1, 0.2]
}
```
**Search Strategy**
| Strategy | Description |
|----------|-------------|
| Random Search | Sample randomly from space |
| Grid Search | Exhaustive search (expensive) |
| Bayesian Optimization | Model-based search |
| Evolution | Genetic algorithms |
| Reinforcement Learning | RL controller picks architectures |
| Differentiable (DARTS) | Gradient-based search |
**Performance Estimation**
| Method | Speed | Accuracy |
|--------|-------|----------|
| Full training | Slow | High |
| Early stopping | Faster | Medium |
| Weight sharing | Fast | Variable |
| Predictors | Very fast | Variable |
**DARTS (Differentiable Architecture Search)**
```python
# Continuous relaxation of architecture choice
alpha = nn.Parameter(torch.randn(num_ops)) # Architecture weights
def forward(x):
ops_outputs = [op(x) for op in operations]
weights = F.softmax(alpha, dim=0)
return sum(w * o for w, o in zip(weights, ops_outputs))
# After training, select highest-weight operations
final_arch = alpha.argmax(dim=0)
```
**AutoML Platforms**
| Platform | Features |
|----------|----------|
| AutoGluon | Tabular, image, text |
| Auto-sklearn | Classical ML |
| H2O AutoML | Enterprise AutoML |
| Ludwig | Declarative deep learning |
| Ray Tune | Hyperparameter tuning |
**Use Cases**
- Find efficient architectures for deployment
- Discover architectures for new domains
- Optimize for specific hardware constraints
- Automate ML pipeline development
**Best Practices**
- Define search space based on domain knowledge
- Use early stopping for efficiency
- Validate on held-out data
- Consider transfer from similar tasks