Probabilistic programming expresses probabilistic models as programs, combining programming languages with probability theory to enable flexible modeling and inference — allowing developers to specify generative models with random variables, distributions, and conditional dependencies, while inference engines automatically compute posterior distributions given observed data.
What Is Probabilistic Programming?
- Traditional programming: Deterministic — same inputs always produce same outputs.
- Probabilistic programming: Programs include random variables and probability distributions — outputs are distributions, not single values.
- Generative Models: Programs describe how data is generated — the data-generating process.
- Inference: Given observed data, infer the values of unobserved (latent) variables — Bayesian inference.
How Probabilistic Programming Works
1. Model Specification: Write a program that describes the probabilistic model — how variables relate and what distributions they follow.
2. Observations: Provide observed data — condition the model on these observations.
3. Inference: The inference engine computes the posterior distribution — what values of latent variables are consistent with the observations.
4. Sampling/Querying: Draw samples from the posterior or query probabilities.
Probabilistic Programming Languages
- Stan: Specialized language for Bayesian inference — uses Hamiltonian Monte Carlo (HMC) for sampling.
- Pyro: Built on PyTorch — combines deep learning with probabilistic programming.
- Edward: TensorFlow-based probabilistic programming — now integrated into TensorFlow Probability.
- Church/WebPPL: Functional probabilistic languages based on Scheme/JavaScript.
- Turing.jl: Julia-based probabilistic programming with flexible inference.
- PyMC: Python library for Bayesian modeling and inference.
Example: Probabilistic Program
``python
import pyro
import pyro.distributions as dist
def coin_flip_model(observations):
# Prior: bias of the coin (unknown)
bias = pyro.sample("bias", dist.Beta(2, 2))
# Likelihood: observed coin flips
for i, obs in enumerate(observations):
pyro.sample(f"flip_{i}", dist.Bernoulli(bias), obs=obs)
return bias
# Observed data: 7 heads, 3 tails
observations = [1, 1, 1, 0, 1, 1, 1, 0, 1, 0]
# Inference: What is the posterior distribution of bias?
# (Use MCMC, variational inference, etc.)
`
Key Concepts
- Prior Distribution: What we believe before seeing data — encodes prior knowledge or assumptions.
- Likelihood: Probability of observing the data given model parameters.
- Posterior Distribution: Updated beliefs after seeing data — combines prior and likelihood via Bayes' rule.
- Latent Variables: Unobserved variables we want to infer — hidden states, parameters, causes.
- Conditioning: Fixing observed variables to their observed values — obs=data`.
Inference Methods
- Markov Chain Monte Carlo (MCMC): Sample from the posterior using random walks — Metropolis-Hastings, Hamiltonian Monte Carlo.
- Variational Inference: Approximate the posterior with a simpler distribution — optimization-based, faster than MCMC.
- Importance Sampling: Weight samples by their likelihood — simple but can be inefficient.
- Sequential Monte Carlo: Particle filters for sequential data — tracking over time.
Applications
- Bayesian Machine Learning: Probabilistic models with uncertainty quantification — Bayesian neural networks, Gaussian processes.
- Causal Inference: Modeling causal relationships and estimating causal effects.
- Time Series Analysis: Modeling temporal data with uncertainty — forecasting, anomaly detection.
- Robotics: Probabilistic state estimation, sensor fusion, planning under uncertainty.
- Cognitive Science: Modeling human cognition and decision-making as probabilistic inference.
- Epidemiology: Modeling disease spread with uncertainty.
Benefits
- Uncertainty Quantification: Probabilistic models naturally represent uncertainty — not just point estimates.
- Modularity: Separate model specification from inference algorithm — change inference method without changing model.
- Flexibility: Express complex models with hierarchies, dependencies, and constraints.
- Interpretability: Generative models are often more interpretable than discriminative models.
- Prior Knowledge: Incorporate domain knowledge through priors and model structure.
Challenges
- Computational Cost: Inference can be slow, especially for complex models — MCMC requires many samples.
- Model Specification: Designing good probabilistic models requires expertise in probability and statistics.
- Convergence: MCMC may not converge, or may converge slowly — diagnosing convergence is non-trivial.
- Scalability: Inference scales poorly with model complexity and data size.
Probabilistic Programming + Deep Learning
- Variational Autoencoders (VAEs): Combine neural networks with probabilistic inference — learn latent representations.
- Bayesian Neural Networks: Neural networks with probabilistic weights — uncertainty in predictions.
- Amortized Inference: Use neural networks to approximate inference — fast inference after training.
Probabilistic programming is a powerful paradigm for reasoning under uncertainty — it makes sophisticated statistical modeling accessible to programmers and enables principled Bayesian inference in complex domains.