maximum mean discrepancy
**Maximum Mean Discrepancy (MMD)** is a non-parametric statistical test and distance metric that measures the difference between two probability distributions by comparing their mean embeddings in a reproducing kernel Hilbert space (RKHS). In domain adaptation, MMD serves as a differentiable loss function that quantifies how different the source and target feature distributions are, enabling direct minimization of domain discrepancy without adversarial training.
**Why MMD Matters in AI/ML:**
MMD provides a **statistically principled, non-adversarial measure of distribution distance** that is differentiable, easy to compute, has well-understood theoretical properties, and directly plugs into neural network training as a regularization loss—making it the most mathematically grounded approach to domain alignment.
• **RKHS embedding** — Each distribution P is represented by its mean embedding μ_P = E_{x~P}[φ(x)] in a RKHS defined by kernel k; MMD²(P,Q) = ||μ_P - μ_Q||²_H = E[k(x,x')] - 2E[k(x,y)] + E[k(y,y')], where x,x' ~ P and y,y' ~ Q
• **Kernel choice** — The Gaussian RBF kernel k(x,y) = exp(-||x-y||²/2σ²) is most common; multi-kernel MMD uses a mixture of Gaussians with different bandwidths for robustness; the kernel must be characteristic (Gaussian, Laplacian) to guarantee that MMD=0 iff P=Q
• **Unbiased estimator** — Given source samples {x_i}ᵢ₌₁ᴺ and target samples {y_j}ⱼ₌₁ᴹ, the unbiased empirical MMD² = 1/(N(N-1))Σᵢ≠ⱼk(xᵢ,xⱼ) - 2/(NM)ΣᵢΣⱼk(xᵢ,yⱼ) + 1/(M(M-1))Σᵢ≠ⱼk(yᵢ,yⱼ) is computed from mini-batches during training
• **Multi-layer MMD (DAN)** — Deep Adaptation Network (DAN) minimizes MMD across multiple hidden layers simultaneously: L = L_task + λΣₗ MMD²(S_l, T_l), aligning representations at multiple abstraction levels for more robust adaptation
• **Conditional MMD** — Class-conditional MMD aligns source and target distributions per class: Σ_k MMD²(P_S(f|y=k), P_T(f|y=k)), preventing class confusion that can occur with marginal MMD alignment alone
| Variant | Kernel | Alignment Level | Complexity | Key Property |
|---------|--------|----------------|-----------|-------------|
| Single-kernel MMD | Gaussian RBF | Single layer | O(N²) | Simple, well-understood |
| Multi-kernel MMD (MK-MMD) | Mixture of RBFs | Single layer | O(N²) | Bandwidth-robust |
| DAN (multi-layer) | Multi-kernel | Multiple layers | O(L·N²) | Deep alignment |
| JAN (joint) | Multi-kernel | Joint distributions | O(N²) | Class-aware |
| Linear MMD | Linear kernel | Single layer | O(N·d) | Fast, less expressive |
| Conditional MMD | Any | Per-class | O(K·N²) | Prevents class confusion |
**Maximum Mean Discrepancy is the mathematically rigorous foundation for non-adversarial domain adaptation, providing a differentiable distribution distance in kernel space that enables direct minimization of domain discrepancy, with well-understood statistical properties, unbiased estimation from finite samples, and seamless integration as a regularization loss in deep neural network training.**