layer normalization techniques
# Layer Normalization Techniques
## Introduction & Motivation
Layer normalization: normalize across feature dimension. Works with variable batch sizes. Applications: transformers, recurrent networks, batch-independent normalization.
Motivation: Enable normalization independent of batch size.
Applications: Transformers, variable-length sequences, small batch training.
---
## Core Concepts & Theory
### Feature-Wise Normalization
Normalize across channels.
### Batch-Independent
No batch statistics needed.
### Learnable Affine
Scale and shift parameters.
### Transformer Standard
Used in all transformers.
---
## Mathematical Formulation
Layer Normalization:
$$y = \gamma \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}} + \beta$$
Where μ and σ computed across features
Efficiency:
$$O(D) ext{ computation per token}$$
---
## Advanced Theory & Extensions
### RMSNorm
Root mean square normalization.
### DeepNorm
Scale activations.
### ALiBi Integration
Combine with position bias.
---
## Computational Considerations
Training: O(D) per token.
Inference: O(D) per token.
Memory: Minimal overhead.
---
## Practical Implementation Strategies
### Initialization
Start with small gamma.
### Pre-Normalization
Apply before main computation.
### Post-Normalization
Apply after residual.
---
## Benchmark Datasets & Evaluation
Language Models: BERT, GPT.
Vision Transformers: ViT, Swin.
Perplexity: Language modeling.
---
## Key Challenges & Limitations
### Output Magnitude
May reduce gradient flow.
### Hyperparameter Sensitivity
Epsilon choice matters.
### Initialization
Careful setup needed.
---
## Hyperparameter Tuning
Epsilon: 1e-5 to 1e-3.
Gamma init: 1.0 or small.
Position: Pre vs post-norm.
---
## Real-World Applications & Case Studies
Transformers: Essential component.
Stability: Improve training stability.
Flexibility: Works with any batch size.
---
## Integration with Other Methods
Layer norm + self-attention; + residual connections.
---
## Summary & Key Takeaways
Layer normalization enables flexible training.
Principles:
1. Feature normalization: Across dimension.
2. Batch-independent: No batch stats.
3. Learnable affine: Scale and shift.
4. Stability: Improve convergence.
5. Efficiency: O(D) computation.
---
## Appendix: Practical Labs
### Lab 1: Layer Normalization
import numpy as np
def layer_norm(x, gamma, beta, eps=1e-5):
"""Layer normalization"""
# Normalize across last dimension
mu = np.mean(x, axis=-1, keepdims=True)
sigma = np.std(x, axis=-1, keepdims=True)
x_hat = (x - mu) / (np.sqrt(sigma**2 + eps))
y = gamma * x_hat + beta
return y
np.random.seed(42)
x = np.random.randn(10, 512)
gamma = np.ones(512)
beta = np.zeros(512)
y = layer_norm(x, gamma, beta)
assert y.shape == x.shape
print("✓ Layer norm working")### Lab 2: RMSNorm
import numpy as np
def rms_norm(x, gamma, eps=1e-5):
"""RMS normalization (simpler than LayerNorm)"""
rms = np.sqrt(np.mean(x**2, axis=-1, keepdims=True) + eps)
y = gamma * (x / rms)
return y
np.random.seed(42)
x = np.random.randn(10, 512)
gamma = np.ones(512)
y = rms_norm(x, gamma)
assert y.shape == x.shape
print("✓ RMSNorm working")### Lab 3: Pre-Norm vs Post-Norm
import numpy as np
def pre_norm_block(x, ff_weight, norm_gamma, norm_beta):
"""Pre-normalization: normalize before main computation"""
# Normalize first
x_norm = layer_norm(x, norm_gamma, norm_beta)
# Apply function
x_transformed = x_norm @ ff_weight
# Residual
y = x + x_transformed
return y
def post_norm_block(x, ff_weight, norm_gamma, norm_beta):
"""Post-normalization: normalize after main computation"""
x_transformed = x @ ff_weight
# Residual + normalize
y = layer_norm(x + x_transformed, norm_gamma, norm_beta)
return y
def layer_norm(x, gamma, beta, eps=1e-5):
mu = np.mean(x, axis=-1, keepdims=True)
sigma = np.std(x, axis=-1, keepdims=True)
return gamma * (x - mu) / (sigma + eps) + beta
np.random.seed(42)
x = np.random.randn(10, 512)
w = np.random.randn(512, 512)
gamma = np.ones(512)
beta = np.zeros(512)
y_pre = pre_norm_block(x, w, gamma, beta)
y_post = post_norm_block(x, w, gamma, beta)
print("✓ Pre-norm vs post-norm comparison done")### Lab 4: Epsilon Sensitivity
import numpy as np
def test_epsilon_effect(x, eps_values=[1e-7, 1e-5, 1e-3, 0.1]):
"""Analyze epsilon effect on normalization"""
results = []
for eps in eps_values:
mu = np.mean(x, axis=-1, keepdims=True)
sigma = np.std(x, axis=-1, keepdims=True)
x_norm = (x - mu) / (np.sqrt(sigma**2 + eps))
results.append(np.mean(np.abs(x_norm)))
return results
np.random.seed(42)
x = np.random.randn(100, 512)
effects = test_epsilon_effect(x)
print(f"✓ Epsilon effects: {effects}")---