Probabilistic programming expresses probabilistic models as programs, combining programming languages with probability theory to enable flexible modeling and inference — allowing developers to specify generative models with random variables, distributions, and conditional dependencies, while inference engines automatically compute posterior distributions given observed data.

What Is Probabilistic Programming?

• Traditional programming: Deterministic — same inputs always produce same outputs.
• Probabilistic programming: Programs include random variables and probability distributions — outputs are distributions, not single values.
• Generative Models: Programs describe how data is generated — the data-generating process.
• Inference: Given observed data, infer the values of unobserved (latent) variables — Bayesian inference.

How Probabilistic Programming Works

1. Model Specification: Write a program that describes the probabilistic model — how variables relate and what distributions they follow.

2. Observations: Provide observed data — condition the model on these observations.

3. Inference: The inference engine computes the posterior distribution — what values of latent variables are consistent with the observations.

4. Sampling/Querying: Draw samples from the posterior or query probabilities.

Probabilistic Programming Languages

• Stan: Specialized language for Bayesian inference — uses Hamiltonian Monte Carlo (HMC) for sampling.
• Pyro: Built on PyTorch — combines deep learning with probabilistic programming.
• Edward: TensorFlow-based probabilistic programming — now integrated into TensorFlow Probability.
• Church/WebPPL: Functional probabilistic languages based on Scheme/JavaScript.
• Turing.jl: Julia-based probabilistic programming with flexible inference.
• PyMC: Python library for Bayesian modeling and inference.

Example: Probabilistic Program

import pyro
import pyro.distributions as dist

def coin_flip_model(observations):
    # Prior: bias of the coin (unknown)
    bias = pyro.sample("bias", dist.Beta(2, 2))
    
    # Likelihood: observed coin flips
    for i, obs in enumerate(observations):
        pyro.sample(f"flip_{i}", dist.Bernoulli(bias), obs=obs)
    
    return bias

# Observed data: 7 heads, 3 tails
observations = [1, 1, 1, 0, 1, 1, 1, 0, 1, 0]

# Inference: What is the posterior distribution of bias?
# (Use MCMC, variational inference, etc.)

Key Concepts

• Prior Distribution: What we believe before seeing data — encodes prior knowledge or assumptions.
• Likelihood: Probability of observing the data given model parameters.
• Posterior Distribution: Updated beliefs after seeing data — combines prior and likelihood via Bayes' rule.
• Latent Variables: Unobserved variables we want to infer — hidden states, parameters, causes.
• Conditioning: Fixing observed variables to their observed values — obs=data.

Inference Methods

• Markov Chain Monte Carlo (MCMC): Sample from the posterior using random walks — Metropolis-Hastings, Hamiltonian Monte Carlo.
• Variational Inference: Approximate the posterior with a simpler distribution — optimization-based, faster than MCMC.
• Importance Sampling: Weight samples by their likelihood — simple but can be inefficient.
• Sequential Monte Carlo: Particle filters for sequential data — tracking over time.

Applications

• Bayesian Machine Learning: Probabilistic models with uncertainty quantification — Bayesian neural networks, Gaussian processes.
• Causal Inference: Modeling causal relationships and estimating causal effects.
• Time Series Analysis: Modeling temporal data with uncertainty — forecasting, anomaly detection.
• Robotics: Probabilistic state estimation, sensor fusion, planning under uncertainty.
• Cognitive Science: Modeling human cognition and decision-making as probabilistic inference.
• Epidemiology: Modeling disease spread with uncertainty.

Benefits

• Uncertainty Quantification: Probabilistic models naturally represent uncertainty — not just point estimates.
• Modularity: Separate model specification from inference algorithm — change inference method without changing model.
• Flexibility: Express complex models with hierarchies, dependencies, and constraints.
• Interpretability: Generative models are often more interpretable than discriminative models.
• Prior Knowledge: Incorporate domain knowledge through priors and model structure.

Challenges

• Computational Cost: Inference can be slow, especially for complex models — MCMC requires many samples.
• Model Specification: Designing good probabilistic models requires expertise in probability and statistics.
• Convergence: MCMC may not converge, or may converge slowly — diagnosing convergence is non-trivial.
• Scalability: Inference scales poorly with model complexity and data size.

Probabilistic Programming + Deep Learning

• Variational Autoencoders (VAEs): Combine neural networks with probabilistic inference — learn latent representations.
• Bayesian Neural Networks: Neural networks with probabilistic weights — uncertainty in predictions.
• Amortized Inference: Use neural networks to approximate inference — fast inference after training.

Probabilistic programming is a powerful paradigm for reasoning under uncertainty — it makes sophisticated statistical modeling accessible to programmers and enables principled Bayesian inference in complex domains.