normalizing flow
**Normalizing Flow** is a **generative model that learns an invertible mapping between a simple base distribution (Gaussian) and a complex data distribution** — enabling exact likelihood computation and efficient sampling, unlike VAEs (approximate inference) or GANs (no likelihood).
**Core Idea**
- Learn invertible transformation $f_\theta: z \rightarrow x$ where $z \sim N(0,I)$.
- Change of variables: $\log p_X(x) = \log p_Z(z) + \log |\det J_{f^{-1}}(x)|$
- Train by maximizing log-likelihood directly — no approximation.
- Sample: $z \sim N(0,I)$, compute $x = f_\theta(z)$.
**Key Architectural Requirement**
- $f$ must be: (1) Invertible, (2) Differentiable, (3) Jacobian determinant efficiently computable.
- Most neural networks fail (2) and (3) — flows use special architectures.
**Major Flow Architectures**
**Coupling Layers (RealNVP)**:
- Split $x$ into $x_1, x_2$. $y_1 = x_1$; $y_2 = x_2 \odot \exp(s(x_1)) + t(x_1)$.
- Jacobian is triangular → det = product of diagonal.
- $s, t$: Arbitrary neural networks — no invertibility constraint.
- Inverse: $x_2 = (y_2 - t(y_1)) \odot \exp(-s(y_1))$ — trivially invertible.
**Autoregressive Flows (MAF, IAF)**:
- Each dimension conditioned on all previous.
- MAF: Fast training, slow sampling. IAF: Fast sampling, slow training.
**Continuous Flows (Neural ODE-based)**:
- Continuous Normalizing Flow (CNF): $dx/dt = f_\theta(x,t)$.
- Exact log-det via Hutchinson trace estimator.
- Flow Matching (2022): Simpler training for CNFs — straight-line trajectories.
**Applications**
- Density estimation: Anomaly detection (any outlier has low likelihood).
- Image generation: Glow (OpenAI, 2018) — high-quality image generation with flows.
- Variational inference: Richer posteriors than diagonal Gaussian.
- Protein structure: Boltzmann generators for molecular conformations.
Normalizing flows are **the theoretically elegant solution for exact generative modeling** — their tractable likelihood makes them uniquely suited for scientific applications requiring probability estimation, though diffusion models have superseded them for image generation quality.