probabilistic programming
**Probabilistic programming** expresses **probabilistic models as programs**, combining programming languages with probability theory to enable flexible modeling and inference — allowing developers to specify generative models with random variables, distributions, and conditional dependencies, while inference engines automatically compute posterior distributions given observed data.
**What Is Probabilistic Programming?**
- Traditional programming: Deterministic — same inputs always produce same outputs.
- **Probabilistic programming**: Programs include **random variables** and **probability distributions** — outputs are distributions, not single values.
- **Generative Models**: Programs describe how data is generated — the data-generating process.
- **Inference**: Given observed data, infer the values of unobserved (latent) variables — Bayesian inference.
**How Probabilistic Programming Works**
1. **Model Specification**: Write a program that describes the probabilistic model — how variables relate and what distributions they follow.
2. **Observations**: Provide observed data — condition the model on these observations.
3. **Inference**: The inference engine computes the posterior distribution — what values of latent variables are consistent with the observations.
4. **Sampling/Querying**: Draw samples from the posterior or query probabilities.
**Probabilistic Programming Languages**
- **Stan**: Specialized language for Bayesian inference — uses Hamiltonian Monte Carlo (HMC) for sampling.
- **Pyro**: Built on PyTorch — combines deep learning with probabilistic programming.
- **Edward**: TensorFlow-based probabilistic programming — now integrated into TensorFlow Probability.
- **Church/WebPPL**: Functional probabilistic languages based on Scheme/JavaScript.
- **Turing.jl**: Julia-based probabilistic programming with flexible inference.
- **PyMC**: Python library for Bayesian modeling and inference.
**Example: Probabilistic Program**
```python
import pyro
import pyro.distributions as dist
def coin_flip_model(observations):
# Prior: bias of the coin (unknown)
bias = pyro.sample("bias", dist.Beta(2, 2))
# Likelihood: observed coin flips
for i, obs in enumerate(observations):
pyro.sample(f"flip_{i}", dist.Bernoulli(bias), obs=obs)
return bias
# Observed data: 7 heads, 3 tails
observations = [1, 1, 1, 0, 1, 1, 1, 0, 1, 0]
# Inference: What is the posterior distribution of bias?
# (Use MCMC, variational inference, etc.)
```
**Key Concepts**
- **Prior Distribution**: What we believe before seeing data — encodes prior knowledge or assumptions.
- **Likelihood**: Probability of observing the data given model parameters.
- **Posterior Distribution**: Updated beliefs after seeing data — combines prior and likelihood via Bayes' rule.
- **Latent Variables**: Unobserved variables we want to infer — hidden states, parameters, causes.
- **Conditioning**: Fixing observed variables to their observed values — `obs=data`.
**Inference Methods**
- **Markov Chain Monte Carlo (MCMC)**: Sample from the posterior using random walks — Metropolis-Hastings, Hamiltonian Monte Carlo.
- **Variational Inference**: Approximate the posterior with a simpler distribution — optimization-based, faster than MCMC.
- **Importance Sampling**: Weight samples by their likelihood — simple but can be inefficient.
- **Sequential Monte Carlo**: Particle filters for sequential data — tracking over time.
**Applications**
- **Bayesian Machine Learning**: Probabilistic models with uncertainty quantification — Bayesian neural networks, Gaussian processes.
- **Causal Inference**: Modeling causal relationships and estimating causal effects.
- **Time Series Analysis**: Modeling temporal data with uncertainty — forecasting, anomaly detection.
- **Robotics**: Probabilistic state estimation, sensor fusion, planning under uncertainty.
- **Cognitive Science**: Modeling human cognition and decision-making as probabilistic inference.
- **Epidemiology**: Modeling disease spread with uncertainty.
**Benefits**
- **Uncertainty Quantification**: Probabilistic models naturally represent uncertainty — not just point estimates.
- **Modularity**: Separate model specification from inference algorithm — change inference method without changing model.
- **Flexibility**: Express complex models with hierarchies, dependencies, and constraints.
- **Interpretability**: Generative models are often more interpretable than discriminative models.
- **Prior Knowledge**: Incorporate domain knowledge through priors and model structure.
**Challenges**
- **Computational Cost**: Inference can be slow, especially for complex models — MCMC requires many samples.
- **Model Specification**: Designing good probabilistic models requires expertise in probability and statistics.
- **Convergence**: MCMC may not converge, or may converge slowly — diagnosing convergence is non-trivial.
- **Scalability**: Inference scales poorly with model complexity and data size.
**Probabilistic Programming + Deep Learning**
- **Variational Autoencoders (VAEs)**: Combine neural networks with probabilistic inference — learn latent representations.
- **Bayesian Neural Networks**: Neural networks with probabilistic weights — uncertainty in predictions.
- **Amortized Inference**: Use neural networks to approximate inference — fast inference after training.
Probabilistic programming is a **powerful paradigm for reasoning under uncertainty** — it makes sophisticated statistical modeling accessible to programmers and enables principled Bayesian inference in complex domains.