Graph Convolutional Networks

# Weight Initialization Strategies

## Introduction & Motivation

Weight Initialization: set network parameter values. Xavier, He, LSUV. Applications: faster convergence, gradient flow stability.

Motivation: Enable effective training from random start.

Applications: Deep networks, convergence acceleration.

---

## Core Concepts & Theory

### Xavier Initialization

Maintain activation variance.

### He Initialization

Account for ReLU non-linearity.

### LSUV

Layer-sequential unit-variance.

### Orthogonal Initialization

Maintain spectral properties.

---

## Mathematical Formulation

Xavier: W \sim ext{Uniform}\left(-\sqrt{\frac{6}{n_{ ext{in}} + n_{ ext{out}}}}, \sqrt{\frac{6}{n_{ ext{in}} + n_{ ext{out}}}}
ight)

He Normal: W \sim \mathcal{N}\left(0, \sqrt{\frac{2}{n_{ ext{in}}}}
ight)

Orthogonal: QR decomposition

---

## Advanced Theory & Extensions

### Variance Preservation

Control activation statistics.

### Spectral Normalization

Lipschitz constraint.

### Hypercomplex Initialization

Quaternion networks.

---

## Computational Considerations

Xavier: O(n_in * n_out).

He: O(n_in).

Orthogonal: O((n_in * n_out)^1.5).

---

## Practical Implementation Strategies

### Per-Layer Initialization

Different schemes per layer.

### Gain Tuning

Activation-aware scaling.

### Bias Initialization

Often set to zero.

---

## Benchmark Datasets & Evaluation

ImageNet: Large-scale training.

CIFAR-10: Small network validation.

MNIST: Simple baseline.

---

## Key Challenges & Limitations

### Depth Sensitivity

Deeper networks need tuning.

### Activation Dependency

Scheme selection critical.

### BatchNorm Interaction

Normalization reduces sensitivity.

---

## Hyperparameter Tuning

Gain: 1.0-2.0.

Distribution: Normal/Uniform.

Fan mode: In/Out/Avg.

---

## Real-World Applications & Case Studies

Deep ResNets: He initialization standard.

GANs: Careful initialization for stability.

Transformers: Specific schemes per layer type.

---

## Integration with Other Methods

Initialization + learning rate scheduling for optimal convergence; + batch norm for robustness.

---

## Summary & Key Takeaways

Weight Initialization enables effective gradient flow.

Principles:
1. Xavier: Variance preservation.
2. He: ReLU-aware scaling.
3. LSUV: Unit variance layers.
4. Orthogonal: Spectral properties.
5. Tuning: Architecture-dependent.

---

## Appendix: Practical Labs

### Lab 1: Xavier Initialization

import numpy as np

def xavier_init(fan_in, fan_out):
 limit = np.sqrt(6 / (fan_in + fan_out))
 return np.random.uniform(-limit, limit, size=(fan_in, fan_out))

np.random.seed(42)
W = xavier_init(fan_in=1000, fan_out=500)
var = np.var(W)
assert 0.001 < var < 0.01, "Reasonable variance"
print("✓ Xavier initialization working")

### Lab 2: He Initialization

import numpy as np

def he_init(fan_in, fan_out):
 std = np.sqrt(2 / fan_in)
 return np.random.normal(0, std, size=(fan_in, fan_out))

np.random.seed(42)
W = he_init(fan_in=1000, fan_out=500)
var = np.var(W)
assert 0.001 < var < 0.005, "Reasonable variance"
print("✓ He initialization working")

### Lab 3: Orthogonal Initialization

import numpy as np

def orthogonal_init(fan_in, fan_out):
 A = np.random.normal(0, 1, size=(fan_in, fan_out))
 Q, R = np.linalg.qr(A)
 return Q[:fan_in, :fan_out]

np.random.seed(42)
W = orthogonal_init(fan_in=100, fan_out=100)
orthogonal = np.allclose(W.T @ W, np.eye(100), atol=1e-5)
assert orthogonal, "Orthogonal matrix"
print("✓ Orthogonal initialization working")

### Lab 4: Gain Scaling

import numpy as np

def scaled_initialization(fan_in, fan_out, gain=1.0):
 std = gain * np.sqrt(2 / (fan_in + fan_out))
 return np.random.normal(0, std, size=(fan_in, fan_out))

np.random.seed(42)
W_normal = scaled_initialization(100, 100, gain=1.0)
W_scaled = scaled_initialization(100, 100, gain=2.0)
assert np.var(W_scaled) > np.var(W_normal), "Gain increases variance"
print("✓ Gain scaling working")

---