Advanced Optimization Algorithms for ML

# Advanced Optimization Algorithms for ML

## Introduction & Motivation

Optimization algorithms drive ML model training. Advanced methods like Adam, RMSprop, and evolutionary algorithms balance convergence speed, stability, and computational efficiency for diverse engineering and scientific problems.

Motivation: Master optimization algorithms for effective ML training.

Applications: Model training, hyperparameter tuning, architecture search, process optimization.

---

## Core Concepts & Theory

### Gradient Descent

First-order optimization.

### Momentum

Accelerated convergence.

### Adaptive Learning Rates

Per-parameter step sizes.

### Second-Order Methods

Hessian-based optimization.

---

## Mathematical Formulation

Gradient Descent:
$$ heta_{t+1} = heta_t - \alpha abla L( heta_t)$$

Momentum:
$$v_t = \beta v_{t-1} + abla L( heta_t)$$

Adam:
$$ heta_t = heta_t - \alpha \frac{m_t}{\sqrt{v_t} + \epsilon}$$

---

## Advanced Theory & Extensions

### Natural Gradient

Fisher information weighting.

### Proximal Methods

Composite optimization.

### Coordinate Descent

Alternating minimization.

---

## Computational Considerations

SGD: O(D) per update.

Adam: O(D) memory and time.

Newton: O(D³) for Hessian.

---

## Practical Implementation Strategies

### Learning Rate Scheduling

Decay and annealing.

### Batch Normalization

Gradient flow improvement.

### Gradient Clipping

Training stability.

---

## Benchmark Datasets & Evaluation

Optimization Landscapes: Test functions.

ML Tasks: Model training benchmarks.

Convergence Studies: Comparative analysis.

---

## Key Challenges & Limitations

### Local Minima

Getting stuck in suboptimal solutions.

### Hyperparameter Selection

Learning rate and momentum.

### Computational Cost

Large-scale optimization.

---

## Hyperparameter Tuning

Learning rate: 1e-4 to 1e-1.

Momentum: 0.8-0.99.

Beta1, Beta2 (Adam): 0.9, 0.999.

---

## Real-World Applications & Case Studies

Neural Network Training: Deep learning.

Materials Optimization: Property tuning.

Process Control: Parameter optimization.

---

## Integration with Other Methods

Optimization + neural networks; + transfer learning; + Bayesian optimization.

---

## Summary & Key Takeaways

Advanced optimizers improve ML training efficiency.

Principles:
1. Gradient Descent: Foundation.
2. Momentum: Acceleration.
3. Adaptive: Per-parameter rates.
4. Scheduling: Dynamic adjustment.
5. Tuning: Problem-dependent optimization.

---

## Appendix: Practical Labs

### Lab 1: Gradient Descent Variants

import numpy as np

class SGDOptimizer:
 def __init__(self, lr=0.01, momentum=0.9):
 self.lr = lr
 self.momentum = momentum
 self.v = 0
 
 def update(self, grad):
 """Momentum update"""
 self.v = self.momentum * self.v + grad
 return -self.lr * self.v

class AdamOptimizer:
 def __init__(self, lr=0.001, beta1=0.9, beta2=0.999, epsilon=1e-8):
 self.lr = lr
 self.beta1 = beta1
 self.beta2 = beta2
 self.epsilon = epsilon
 
 self.m = 0
 self.v = 0
 self.t = 0
 
 def update(self, grad):
 """Adam update"""
 self.t += 1
 self.m = self.beta1 * self.m + (1 - self.beta1) * grad
 self.v = self.beta2 * self.v + (1 - self.beta2) * grad**2
 
 m_hat = self.m / (1 - self.beta1**self.t)
 v_hat = self.v / (1 - self.beta2**self.t)
 
 return -self.lr * m_hat / (np.sqrt(v_hat) + self.epsilon)

print(f"✓ Optimizers configured")

### Lab 2: Learning Rate Scheduling

import numpy as np

def exponential_decay(initial_lr, decay_rate, step):
 """Exponential learning rate decay"""
 return initial_lr * np.exp(-decay_rate * step)

def step_decay(initial_lr, decay_rate, step, decay_every=10):
 """Step-wise learning rate decay"""
 return initial_lr * (decay_rate ** (step // decay_every))

# Test
steps = np.arange(100)
lr_exp = np.array([exponential_decay(0.01, 0.05, s) for s in steps])
lr_step = np.array([step_decay(0.01, 0.1, s) for s in steps])

print(f"✓ Scheduling: exponential and step decay configured")

### Lab 3: Convergence Analysis

import numpy as np

def optimize_quadratic(x0, method='sgd', n_iter=100):
 """Minimize quadratic function"""
 x = x0.copy()
 losses = [np.sum(x**2)]
 
 for i in range(n_iter):
 grad = 2 * x
 
 if method == 'sgd':
 x -= 0.01 * grad
 elif method == 'momentum':
 if i == 0:
 v = grad
 else:
 v = 0.9 * v + grad
 x -= 0.01 * v
 
 losses.append(np.sum(x**2))
 
 return np.array(losses)

x0 = np.random.randn(5)
losses_sgd = optimize_quadratic(x0, method='sgd')
losses_momentum = optimize_quadratic(x0, method='momentum')

print(f"✓ SGD final loss: {losses_sgd[-1]:.6f}")
print(f"✓ Momentum final loss: {losses_momentum[-1]:.6f}")

### Lab 4: Hyperparameter Optimization

import numpy as np

class GridSearchOptimizer:
 def __init__(self):
 self.best_params = None
 self.best_score = float('inf')
 
 def search(self, param_grid, objective_fn):
 """Grid search optimization"""
 lrs = param_grid['learning_rate']
 momentums = param_grid['momentum']
 
 for lr in lrs:
 for momentum in momentums:
 score = objective_fn(lr, momentum)
 
 if score < self.best_score:
 self.best_score = score
 self.best_params = {'lr': lr, 'momentum': momentum}
 
 return self.best_params

def test_objective(lr, momentum):
 """Simple objective function"""
 return lr**2 + (momentum - 0.9)**2

searcher = GridSearchOptimizer()
grid = {
 'learning_rate': np.logspace(-4, -1, 5),
 'momentum': np.linspace(0.8, 0.99, 5)
}

best = searcher.search(grid, test_objective)
print(f"✓ Best params: {best}")

---