beta-vae
**β-VAE (Beta Variational Autoencoder)** is a modification of the standard VAE that introduces a hyperparameter β > 1 to upweight the KL divergence term in the ELBO objective, encouraging the model to learn more disentangled latent representations at the cost of reconstruction quality. The β-VAE objective L = E_q[log p(x|z)] - β·KL(q(z|x)||p(z)) pushes the encoder to produce a more structured, factorized posterior that aligns individual latent dimensions with independent factors of variation.
**Why β-VAE Matters in AI/ML:**
β-VAE demonstrated that **simple modification of the VAE objective can encourage disentangled representations**, providing the foundational approach for learning interpretable, factor-aligned latent spaces without explicit supervision on the underlying generative factors.
• **Information bottleneck** — Increasing β constrains the information flowing through the latent bottleneck (measured by KL divergence); under strong constraint, the model must efficiently encode only the most important, statistically independent factors, naturally producing disentanglement as the most efficient encoding strategy
• **Reconstruction-disentanglement tradeoff** — Higher β improves disentanglement metrics (β-VAE metric, MIG) but degrades reconstruction quality (blurry outputs); the optimal β balances interpretable latent structure against faithful reconstruction
• **Capacity annealing (β-VAE with controlled increase)** — Gradually increasing the KL capacity C: L = E_q[log p(x|z)] - β·|KL(q(z|x)||p(z)) - C| allows the model to first learn good reconstruction, then progressively constrain the latent space toward disentanglement
• **Factor discovery** — Without labeled factors, β-VAE discovers interpretable dimensions corresponding to azimuth, elevation, scale, shape, and color in synthetic datasets (dSprites, 3D Shapes), validating that unsupervised disentanglement is achievable
• **Relationship to rate-distortion** — β-VAE traces the rate-distortion curve: low β (high rate, low distortion, entangled) to high β (low rate, high distortion, disentangled), revealing the fundamental tradeoff between information compression and representation structure
| β Value | KL Weight | Reconstruction | Disentanglement | Use Case |
|---------|-----------|---------------|-----------------|----------|
| β = 0 | No regularization | Best | None (autoencoder) | Reconstruction only |
| β = 1 | Standard VAE | Good | Moderate | Standard generation |
| β = 2-4 | Mild pressure | Good | Improved | Balanced |
| β = 10-20 | Strong pressure | Moderate | Good | Disentanglement focus |
| β = 50-100 | Very strong | Poor (blurry) | Maximum | Analysis, discovery |
**β-VAE is the foundational method for unsupervised disentangled representation learning, demonstrating that simply upweighting the KL regularization in the VAE objective creates an information bottleneck that forces the model to discover efficient, factorized encodings aligned with the true generative factors of the data.**