Neural Architecture Search Nas

# Neural Architecture Search (NAS)

## Introduction & Motivation

Neural Architecture Search: automatically design neural network architectures. Hyperparameter optimization; AutoML. Applications: architecture design, resource-constrained models.

Motivation: Reduce manual architecture design; find optimal configurations.

Applications: AutoML, efficient architecture discovery.

---

## Core Concepts & Theory

### Search Space

Set of possible architectures.

### Search Strategy

Evolutionary algorithms, reinforcement learning, gradient-based.

### Performance Estimation

Predict accuracy without full training.

### Early Stopping

Reduce computational cost.

---

## Mathematical Formulation

Architecture Scoring:
$$ ext{score}(A) = ext{accuracy}(A) - \lambda \cdot ext{cost}(A)$$

Evolutionary Fitness:
$$f(A) = ext{accuracy}(A) - \alpha \cdot ext{params}(A)$$

RL Reward:
$$R(A) = ext{accuracy}(A) / ext{baseline}$$

---

## Advanced Theory & Extensions

### ENAS (Efficient NAS)

Parameter sharing across architectures.

### DARTS (Differentiable Architecture Search)

Gradient-based architecture search.

### Multi-Objective Optimization

Accuracy-latency trade-offs.

---

## Computational Considerations

Random search: O(N·train_time).

Evolutionary: O(pop·gen·train_time).

DARTS: O(architecture_params·iterations).

---

## Practical Implementation Strategies

### Early Stopping

Approximate accuracy with partial training.

### Cell-Based Search

Search for cell blocks, not full architectures.

### Weight Sharing

Reuse weights across architectures.

---

## Benchmark Datasets & Evaluation

ImageNet: Classification benchmark.

CIFAR-10: Small scale evaluation.

NAS-Bench: Precomputed architecture performance.

---

## Key Challenges & Limitations

### Computational Cost

High search cost.

### Generalization

Best architecture for one dataset may not transfer.

### Search Space Design

Critical design choice.

---

## Hyperparameter Tuning

Population size: 20-100.

Search budget: Hours to thousands of GPU hours.

Mutation rate: 0.1-0.3.

---

## Real-World Applications & Case Studies

MobileNetV3: NAS for mobile efficiency.

EfficientNet: Compound scaling via NAS.

AutoML: Automated model selection.

---

## Integration with Other Methods

NAS + knowledge distillation for compact models; + pruning for extreme efficiency.

---

## Summary & Key Takeaways

Neural Architecture Search via evolutionary and gradient-based methods enables automated architecture discovery.

Principles:
1. Search space: Definable set.
2. Evaluation: Predict or train.
3. Search strategy: Evolutionary or gradient.
4. Early stopping: Cost reduction.
5. Multi-objective: Efficiency-accuracy trade-off.

---

---

## Appendix: Practical Labs

### Lab 1: Architecture Scoring

import numpy as np

def score_architecture(accuracy, num_params, lambda_param=0.001):
 """Score architecture balancing accuracy and efficiency"""
 # Penalize large models
 score = accuracy - lambda_param * (num_params / 1e6)
 
 return score

# Test
accuracy = 0.92
params = 10e6

score = score_architecture(accuracy, params, lambda_param=0.0001)

assert np.isfinite(score), "Score finite"
print("✓ Architecture scoring working")

if __name__ == "__main__":
 print("Lab 1: ArchitectureScoring - PASSED")

### Lab 2: Evolutionary Selection

import numpy as np

def evolutionary_selection(population, fitness_scores, survival_rate=0.5):
 """Select top architectures for next generation"""
 n_keep = int(len(population) * survival_rate)
 
 # Sort by fitness (descending)
 sorted_indices = np.argsort(fitness_scores)[::-1]
 
 # Keep top architectures
 selected = [population[i] for i in sorted_indices[:n_keep]]
 
 return selected

# Test
population = [{'layers': 3}, {'layers': 5}, {'layers': 4}, {'layers': 2}]
fitness = np.array([0.85, 0.92, 0.88, 0.80])

selected = evolutionary_selection(population, fitness, survival_rate=0.5)

assert len(selected) == 2, "Correct selection size"
print("✓ Evolutionary selection working")

if __name__ == "__main__":
 print("Lab 2: EvolutionarySelection - PASSED")

### Lab 3: Architecture Mutation

import numpy as np

def mutate_architecture(architecture, mutation_rate=0.1):
 """Randomly mutate architecture"""
 mutated = architecture.copy()
 
 # Mutate number of layers
 if np.random.rand() < mutation_rate:
 mutated['num_layers'] = np.clip(mutated['num_layers'] + np.random.randint(-1, 2), 2, 10)
 
 # Mutate hidden dimension
 if np.random.rand() < mutation_rate:
 mutated['hidden_dim'] = np.clip(mutated['hidden_dim'] + np.random.randint(-32, 33), 64, 512)
 
 return mutated

# Test
arch = {'num_layers': 5, 'hidden_dim': 256}
mutated = mutate_architecture(arch, mutation_rate=1.0)

assert isinstance(mutated, dict), "Mutated is dict"
print("✓ Architecture mutation working")

if __name__ == "__main__":
 print("Lab 3: ArchitectureMutation - PASSED")

### Lab 4: Early Stopping

import numpy as np

def early_stop_decision(val_losses, patience=3):
 """Decide whether to stop based on validation loss"""
 if len(val_losses) < patience:
 return False
 
 # Check if latest loss is best
 recent_losses = val_losses[-patience:]
 best_loss = min(recent_losses)
 
 if recent_losses[-1] > best_loss:
 return True
 
 return False

# Test
losses = [0.5, 0.4, 0.35, 0.36, 0.37, 0.38]

should_stop = early_stop_decision(losses, patience=3)

assert isinstance(should_stop, bool), "Boolean decision"
print("✓ Early stopping working")

if __name__ == "__main__":
 print("Lab 4: EarlyStopping - PASSED")