Normalizing Flows Invertible Transformations Density Estimation

# Normalizing Flows: Invertible Transformations & Density Estimation

## Introduction & Motivation

Normalizing flows: learn invertible transformations. Convert simple distribution → complex distribution. Density estimation: compute likelihood efficiently. Inverse: sample by transforming simple samples. Affine coupling: efficient invertible layers. Applications: generative modeling, variational inference, Bayesian deep learning.

Motivation: Likelihood-based: maximum likelihood training. Flows: tractable density + invertibility.

Applications: Generative modeling, variational inference, uncertainty quantification.

---

## Core Concepts & Theory

### Flow Transformation

z → x via invertible T; x = T(z), z = T^(-1)(x).

### Change of Variables

Density transformation via Jacobian determinant.

### Coupling Layers

Affine/nonlinear coupling; efficient invertible.

---

## Mathematical Formulation

Change of variables:
$$p_X(x) = p_Z(T^{-1}(x)) \left| \det \frac{dT^{-1}}{dx} ight|$$

Log probability:
$$\log p(x) = \log p(z) - \sum_{i=1}^K \log |\det J_i|$$

Affine coupling:
$$x^{(l)} = z^{(l)} \odot e^{s(z^{(l-1)})} + t(z^{(l-1)})$$

---

## Advanced Theory & Extensions

### Neural Spline Flows

Learned monotonic spline transformations; flexible.

### Invertible ResNets

Residual connections + invertibility; deep networks.

### Autoregressive Flows

Sequential transformation; tractable Jacobian.

---

## Computational Considerations

Forward pass: O(K·d) for K flows, d dimension.

Jacobian determinant: O(d³) via LU decomposition (or efficient).

Training: O(K·d) forward + backward; reasonable.

---

## Practical Implementation Strategies

### Coupling Strategy

Alternate feature masking; ensure invertibility.

### Jacobian Computation

Use trace estimators; avoid explicit computation.

### Prior Distribution

Standard normal common; flexible via flows.

---

## Benchmark Datasets & Evaluation

MNIST: Standard; generative quality assessment.

CelebA: Faces; likelihood vs sample quality tradeoff.

Density Estimation: Synthetic; known ground truth density.

---

## Key Challenges & Limitations

### Jacobian Computation

Expensive for large dimensions; efficient approximations needed.

### Expressiveness-Efficiency Tradeoff

Deep flows more expressive; costly Jacobian.

### Training Stability

Likelihood can diverge; careful regularization needed.

---

## Hyperparameter Tuning

Number of flows K: 4-8 typical; more = better.

Coupling architecture: Simple nets OK; 1-2 hidden layers.

Learning rate: 1e-3 standard; decay useful.

---

## Real-World Applications & Case Studies

Generative Modeling: High-dimensional density learning.

Variational Inference: Better posterior approximation.

Uncertainty Quantification: Predictive distributions.

---

## Integration with Other Methods

Flows + VAE → flexible posterior.

Flows + GAN → adversarial training of flows.

---

## Summary & Key Takeaways

Normalizing flows via invertible transformations enable tractable likelihood computation and flexible density modeling through sequential application of simple invertible layers.

Principles:
1. Invertible: both T and T^(-1) computable.
2. Change of variables: Jacobian determinant.
3. Coupling: efficient invertible parameterization.
4. Autoregressive: sequential, tractable Jacobian.
5. Tractable likelihood: maximum likelihood training.

---

---

## Appendix: Practical Labs

### Lab 1: Affine Coupling

import torch
import torch.nn as nn
import numpy as np

class AffineCoupling(nn.Module):
 def __init__(self, input_dim, hidden_dim):
 super().__init__()
 self.net = nn.Sequential(
 nn.Linear(input_dim // 2, hidden_dim),
 nn.ReLU(),
 nn.Linear(hidden_dim, input_dim)
 )

 def forward(self, x):
 """Forward: transform half, scale/shift other half"""
 x1, x2 = x[:, :x.shape[1]//2], x[:, x.shape[1]//2:]
 
 params = self.net(x1)
 s, t = params[:, :x2.shape[1]], params[:, x2.shape[1]:]
 
 y2 = x2 * torch.exp(s) + t
 y = torch.cat([x1, y2], dim=1)
 
 return y, s.sum(dim=1)

# Test
np.random.seed(42)
coupling = AffineCoupling(input_dim=20, hidden_dim=64)
x = torch.randn(8, 20)

y, log_det = coupling(x)

assert y.shape == x.shape, "Output shape matches"
print("✓ Affine coupling working")

if __name__ == "__main__":
 print("Lab 1: Coupling - PASSED")

### Lab 2: Change of Variables

import torch
import numpy as np

def change_of_variables(z, log_pz, log_det_jacobian):
 """Apply change of variables formula"""
 log_px = log_pz - log_det_jacobian.sum(dim=1)
 return log_px

# Test
np.random.seed(42)
z = torch.randn(32, 20)
log_pz = -0.5 * (z ** 2).sum(dim=1) - 10 * np.log(np.sqrt(2 * np.pi))
log_det = torch.randn(32)

log_px = change_of_variables(z, log_pz, log_det)

assert log_px.shape == (32,), "Output shape correct"
assert torch.isfinite(log_px).all(), "All finite"
print("✓ Change of variables working")

if __name__ == "__main__":
 print("Lab 2: Variables - PASSED")

### Lab 3: Invertible Transformation

import torch
import numpy as np

class SimpleInvertibleFlow:
 def __init__(self, scale=1.0):
 self.scale = scale

 def forward(self, z):
 """Forward transformation"""
 x = z * self.scale
 log_det = len(z) * np.log(self.scale)
 return x, log_det

 def inverse(self, x):
 """Inverse transformation"""
 z = x / self.scale
 return z

# Test
np.random.seed(42)
flow = SimpleInvertibleFlow(scale=2.0)

z = np.random.randn(10)
x, log_det = flow.forward(z)
z_recovered = flow.inverse(x)

assert np.allclose(z, z_recovered), "Should recover original"
print("✓ Invertible transformation working")

if __name__ == "__main__":
 print("Lab 3: Invertible - PASSED")

### Lab 4: Flow Composition

import torch
import numpy as np

def compose_flows(x, flows):
 """Apply sequence of flows"""
 log_det_total = 0
 
 for flow in flows:
 x, log_det = flow(x)
 log_det_total = log_det_total + log_det.sum()
 
 return x, log_det_total

# Test
np.random.seed(42)

class DummyFlow:
 def __call__(self, x):
 return x * 0.5, torch.ones(len(x)) * np.log(0.5)

flows = [DummyFlow(), DummyFlow()]
x = torch.randn(8, 20)

y, log_det = compose_flows(x, flows)

assert y.shape == x.shape, "Output shape correct"
assert np.isfinite(log_det), "Log det finite"
print("✓ Flow composition working")

if __name__ == "__main__":
 print("Lab 4: Composition - PASSED")