Layer Normalization Techniques

# Layer Normalization Techniques

## Introduction & Motivation

Layer normalization: normalize across feature dimension. Works with variable batch sizes. Applications: transformers, recurrent networks, batch-independent normalization.

Motivation: Enable normalization independent of batch size.

Applications: Transformers, variable-length sequences, small batch training.

---

## Core Concepts & Theory

### Feature-Wise Normalization

Normalize across channels.

### Batch-Independent

No batch statistics needed.

### Learnable Affine

Scale and shift parameters.

### Transformer Standard

Used in all transformers.

---

## Mathematical Formulation

Layer Normalization:
$$y = \gamma \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}} + \beta$$

Where μ and σ computed across features

Efficiency:
$$O(D) ext{ computation per token}$$

---

## Advanced Theory & Extensions

### RMSNorm

Root mean square normalization.

### DeepNorm

Scale activations.

### ALiBi Integration

Combine with position bias.

---

## Computational Considerations

Training: O(D) per token.

Inference: O(D) per token.

Memory: Minimal overhead.

---

## Practical Implementation Strategies

### Initialization

Start with small gamma.

### Pre-Normalization

Apply before main computation.

### Post-Normalization

Apply after residual.

---

## Benchmark Datasets & Evaluation

Language Models: BERT, GPT.

Vision Transformers: ViT, Swin.

Perplexity: Language modeling.

---

## Key Challenges & Limitations

### Output Magnitude

May reduce gradient flow.

### Hyperparameter Sensitivity

Epsilon choice matters.

### Initialization

Careful setup needed.

---

## Hyperparameter Tuning

Epsilon: 1e-5 to 1e-3.

Gamma init: 1.0 or small.

Position: Pre vs post-norm.

---

## Real-World Applications & Case Studies

Transformers: Essential component.

Stability: Improve training stability.

Flexibility: Works with any batch size.

---

## Integration with Other Methods

Layer norm + self-attention; + residual connections.

---

## Summary & Key Takeaways

Layer normalization enables flexible training.

Principles:
1. Feature normalization: Across dimension.
2. Batch-independent: No batch stats.
3. Learnable affine: Scale and shift.
4. Stability: Improve convergence.
5. Efficiency: O(D) computation.

---

## Appendix: Practical Labs

### Lab 1: Layer Normalization

import numpy as np

def layer_norm(x, gamma, beta, eps=1e-5):
 """Layer normalization"""
 # Normalize across last dimension
 mu = np.mean(x, axis=-1, keepdims=True)
 sigma = np.std(x, axis=-1, keepdims=True)
 
 x_hat = (x - mu) / (np.sqrt(sigma**2 + eps))
 y = gamma * x_hat + beta
 
 return y

np.random.seed(42)
x = np.random.randn(10, 512)
gamma = np.ones(512)
beta = np.zeros(512)
y = layer_norm(x, gamma, beta)
assert y.shape == x.shape
print("✓ Layer norm working")

### Lab 2: RMSNorm

import numpy as np

def rms_norm(x, gamma, eps=1e-5):
 """RMS normalization (simpler than LayerNorm)"""
 rms = np.sqrt(np.mean(x**2, axis=-1, keepdims=True) + eps)
 y = gamma * (x / rms)
 return y

np.random.seed(42)
x = np.random.randn(10, 512)
gamma = np.ones(512)
y = rms_norm(x, gamma)
assert y.shape == x.shape
print("✓ RMSNorm working")

### Lab 3: Pre-Norm vs Post-Norm

import numpy as np

def pre_norm_block(x, ff_weight, norm_gamma, norm_beta):
 """Pre-normalization: normalize before main computation"""
 # Normalize first
 x_norm = layer_norm(x, norm_gamma, norm_beta)
 # Apply function
 x_transformed = x_norm @ ff_weight
 # Residual
 y = x + x_transformed
 return y

def post_norm_block(x, ff_weight, norm_gamma, norm_beta):
 """Post-normalization: normalize after main computation"""
 x_transformed = x @ ff_weight
 # Residual + normalize
 y = layer_norm(x + x_transformed, norm_gamma, norm_beta)
 return y

def layer_norm(x, gamma, beta, eps=1e-5):
 mu = np.mean(x, axis=-1, keepdims=True)
 sigma = np.std(x, axis=-1, keepdims=True)
 return gamma * (x - mu) / (sigma + eps) + beta

np.random.seed(42)
x = np.random.randn(10, 512)
w = np.random.randn(512, 512)
gamma = np.ones(512)
beta = np.zeros(512)

y_pre = pre_norm_block(x, w, gamma, beta)
y_post = post_norm_block(x, w, gamma, beta)
print("✓ Pre-norm vs post-norm comparison done")

### Lab 4: Epsilon Sensitivity

import numpy as np

def test_epsilon_effect(x, eps_values=[1e-7, 1e-5, 1e-3, 0.1]):
 """Analyze epsilon effect on normalization"""
 results = []
 for eps in eps_values:
 mu = np.mean(x, axis=-1, keepdims=True)
 sigma = np.std(x, axis=-1, keepdims=True)
 x_norm = (x - mu) / (np.sqrt(sigma**2 + eps))
 results.append(np.mean(np.abs(x_norm)))
 return results

np.random.seed(42)
x = np.random.randn(100, 512)
effects = test_epsilon_effect(x)
print(f"✓ Epsilon effects: {effects}")

---