Initialization Strategies Xavier He Kaiming Initialization

# Initialization Strategies: Xavier, He & Kaiming Initialization

## Introduction & Motivation

Weight initialization: sets starting point for training. Random initialization critical; poor initialization → vanishing/exploding gradients, slow convergence. Xavier: uniform variance across layers; for sigmoid/tanh. He: larger variance; for ReLU networks. Kaiming: identical to He; popular in modern frameworks. Applications: all neural networks; often overlooked but essential for stability.

Motivation: Untrained weights → random behavior. Proper scale ensures gradients flow through early layers.

Applications: Neural network training, transfer learning.

---

## Core Concepts & Theory

### Xavier Initialization

Maintain activation variance across layers; uniform scaling.

### He Initialization

Account for ReLU sparsity; larger variance than Xavier.

### Kaiming Initialization

Variant of He; commonly used in PyTorch.

---

## Mathematical Formulation

Xavier (Uniform):
$$W \sim ext{Uniform}\left[-\sqrt{\frac{6}{n_{ ext{in}} + n_{ ext{out}}}}, \sqrt{\frac{6}{n_{ ext{in}} + n_{ ext{out}}}} ight]$$

Xavier (Normal):
$$W \sim \mathcal{N}\left(0, \frac{2}{n_{ ext{in}} + n_{ ext{out}}} ight)$$

He (Normal):
$$W \sim \mathcal{N}\left(0, \frac{2}{n_{ ext{in}}} ight)$$

Kaiming (identical to He):
$$ ext{std} = \sqrt{\frac{2}{n_{ ext{in}}}}$$

---

## Advanced Theory & Extensions

### Orthogonal Initialization

Orthogonal matrix; preserve norms through linear layers.

### LSUV (Layer-Sequential Unit-Variance)

Iteratively adjust variance per layer; ensure 1.0 output variance.

### Spectral Normalization

Normalize weights by largest singular value; stable training.

---

## Computational Considerations

Xavier/He: O(1) sampling; negligible overhead.

Orthogonal: O(n²) or O(n³) for QR decomposition; one-time cost.

LSUV: O(L·batch_size) forward passes; offline calibration.

---

## Practical Implementation Strategies

### Default Strategy

Use framework defaults (PyTorch: Kaiming for linear).

### Custom Initialization

Set manually for special layers (e.g., attention heads).

### Bias Initialization

Typically zero; some use small constants (0.01).

---

## Benchmark Datasets & Evaluation

ImageNet: He/Kaiming standard; ResNets use this.

MNIST: Less sensitive; any reasonable init works.

Deep Networks (>50 layers): Proper init critical; enables training.

---

## Key Challenges & Limitations

### Activation Dependency

Xavier for sigmoid/tanh; He for ReLU; wrong choice hurts.

### Skip Connections

Residuals reduce init dependency; less critical with residuals.

### Fine-tuning

Pretrained weights override init; less relevant.

---

## Hyperparameter Tuning

Xavier/He: Framework defaults usually optimal.

Bias initialization: 0 standard; 0.01-0.1 rarely needed.

Custom scaling: Rare; empirical tuning if needed.

---

## Real-World Applications & Case Studies

ResNets: He initialization; enables 100+ layer training.

Vision Transformers: Xavier common; relatively insensitive.

RNNs: Orthogonal for recurrent weights; He for input.

---

## Integration with Other Methods

Initialization + Normalization → robust gradient flow.

Initialization + Learning Rate → coupled sensitivity.

---

## Summary & Key Takeaways

Weight initialization via He, Xavier strategies controls gradient magnitudes, enabling stable deep network training with proper activation-dependent variance scaling.

Principles:
1. Xavier: for sigmoid/tanh; balance in/out variance.
2. He: for ReLU; account for sparsity via larger variance.
3. Kaiming: identical to He; PyTorch standard.
4. Bias: typically zero initialization.
5. Residuals: reduce init sensitivity; less critical.

---

---

## Appendix: Practical Labs

### Lab 1: Xavier Initialization

import torch
import numpy as np

def xavier_uniform(layer):
 """Xavier uniform initialization"""
 with torch.no_grad():
 fan_in = layer.weight.size(1)
 fan_out = layer.weight.size(0)
 limit = np.sqrt(6 / (fan_in + fan_out))
 layer.weight.uniform_(-limit, limit)
 if layer.bias is not None:
 layer.bias.zero_()

# Test
np.random.seed(42)
layer = torch.nn.Linear(100, 50)

xavier_uniform(layer)

fan_in, fan_out = 100, 50
limit = np.sqrt(6 / (fan_in + fan_out))

assert (layer.weight.abs() <= limit).all(), "Weights should be within bounds"
assert layer.bias is not None and (layer.bias == 0).all(), "Bias should be zero"
print("✓ Xavier uniform working")

if __name__ == "__main__":
 print("Lab 1: Xavier - PASSED")

### Lab 2: He Initialization

import torch
import numpy as np

def he_normal(layer):
 """He normal initialization"""
 with torch.no_grad():
 fan_in = layer.weight.size(1)
 std = np.sqrt(2.0 / fan_in)
 layer.weight.normal_(0, std)
 if layer.bias is not None:
 layer.bias.zero_()

# Test
np.random.seed(42)
layer = torch.nn.Linear(100, 50)

he_normal(layer)

fan_in = 100
expected_std = np.sqrt(2.0 / fan_in)

actual_std = layer.weight.std().item()
assert abs(actual_std - expected_std) < 0.01, "Std should match He"
assert (layer.bias == 0).all(), "Bias should be zero"
print("✓ He normal working")

if __name__ == "__main__":
 print("Lab 2: He - PASSED")

### Lab 3: Initialization Impact on Gradient Flow

import torch
import numpy as np

def measure_init_impact(init_fn, depth=20, n_trials=100):
 """Measure gradient magnitude through layers with different init"""
 grad_magnitudes = []

 for _ in range(n_trials):
 model = torch.nn.Sequential(
 *[torch.nn.Linear(100, 100) for _ in range(depth)]
 )
 
 for layer in model:
 init_fn(layer)

 X = torch.randn(32, 100)
 output = model(X)
 loss = output.sum()
 loss.backward()

 first_layer_grad = model[0][0].weight.grad.abs().mean().item()
 last_layer_grad = model[-1].weight.grad.abs().mean().item()
 
 grad_magnitudes.append((first_layer_grad, last_layer_grad))

 grad_magnitudes = np.array(grad_magnitudes)
 return grad_magnitudes.mean(axis=0)

# Test
np.random.seed(42)

def xavier_init(layer):
 fan_in, fan_out = layer.weight.size(1), layer.weight.size(0)
 limit = np.sqrt(6 / (fan_in + fan_out))
 layer.weight.uniform_(-limit, limit)

def he_init(layer):
 std = np.sqrt(2.0 / layer.weight.size(1))
 layer.weight.normal_(0, std)

xavier_grads = measure_init_impact(xavier_init, depth=20)
he_grads = measure_init_impact(he_init, depth=20)

assert all(np.isfinite(g) for g in xavier_grads), "All finite"
assert all(np.isfinite(g) for g in he_grads), "All finite"
print("✓ Initialization impact working")

if __name__ == "__main__":
 print("Lab 3: Impact - PASSED")

### Lab 4: Orthogonal Initialization

import torch
import numpy as np

def orthogonal_init(layer, gain=1.0):
 """Orthogonal initialization"""
 with torch.no_grad():
 weight = layer.weight.data
 rows, cols = weight.size(0), weight.size(1)
 
 # Generate random matrix
 random_matrix = torch.randn(rows, cols)
 
 # QR decomposition
 Q, _ = torch.linalg.qr(random_matrix)
 
 # Scale by gain
 weight.copy_(gain * Q[:rows, :cols])
 
 if layer.bias is not None:
 layer.bias.zero_()

# Test
np.random.seed(42)
layer = torch.nn.Linear(100, 50)

orthogonal_init(layer, gain=1.0)

# Check orthogonality: W @ W^T should be close to I
W = layer.weight
gram = W @ W.T

identity = torch.eye(W.size(0))
ortho_error = (gram - identity).norm().item()

assert ortho_error < 0.1, "Should be orthogonal"
assert (layer.bias == 0).all(), "Bias should be zero"
print("✓ Orthogonal initialization working")

if __name__ == "__main__":
 print("Lab 4: Orthogonal - PASSED")