Differentiable programming is a programming paradigm where program components are differentiable functions, enabling gradient-based optimization through the entire program — extending automatic differentiation beyond neural networks to arbitrary programs, allowing optimization of complex computational pipelines end-to-end.

What Is Differentiable Programming?

<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,sans-serif">
  <rect x="0" y="0" width="760" height="470" fill="#0d1117"/>
  <text x="380" y="28" fill="#e6edf3" font-size="21" font-weight="700" text-anchor="middle">Differentiable Programming — Gradients Everywhere</text>
  <text x="380" y="48" fill="#8b98a5" font-size="12" text-anchor="middle">compose differentiable primitives into arbitrary programs — optimize end-to-end with backprop</text>

  <!-- Core concept -->
  <rect x="30" y="65" width="700" height="150" rx="6" fill="#080d14" stroke="#233043" stroke-width="1"/>
  <text x="380" y="86" fill="#e6edf3" font-size="11" font-weight="600" text-anchor="middle">The Idea: Programs as Differentiable Computation Graphs</text>

  <!-- Traditional program -->
  <rect x="55" y="100" width="190" height="90" rx="5" fill="#0b1220" stroke="#f87171" stroke-width="0.8"/>
  <text x="150" y="118" fill="#fca5a5" font-size="9" font-weight="600" text-anchor="middle">Traditional Program</text>
  <text x="150" y="138" fill="#8b98a5" font-size="8" text-anchor="middle">if/else, loops, discrete ops</text>
  <text x="150" y="155" fill="#8b98a5" font-size="8" text-anchor="middle">no gradients through logic</text>
  <text x="150" y="172" fill="#6b7684" font-size="7.5" text-anchor="middle">optimize by hand-tuning</text>

  <text x="270" y="145" fill="#f59e0b" font-size="14" text-anchor="middle">→</text>

  <!-- Differentiable program -->
  <rect x="295" y="100" width="200" height="90" rx="5" fill="#0b1220" stroke="#34d399" stroke-width="1.2"/>
  <text x="395" y="118" fill="#6ee7b7" font-size="9" font-weight="600" text-anchor="middle">Differentiable Program</text>
  <text x="395" y="138" fill="#8b98a5" font-size="8" text-anchor="middle">soft gates, continuous relaxation</text>
  <text x="395" y="155" fill="#8b98a5" font-size="8" text-anchor="middle">∂output/∂params exists everywhere</text>
  <text x="395" y="172" fill="#6b7684" font-size="7.5" text-anchor="middle">optimize with gradient descent!</text>

  <text x="520" y="145" fill="#f59e0b" font-size="14" text-anchor="middle">→</text>

  <!-- Result -->
  <rect x="545" y="100" width="165" height="90" rx="5" fill="#0b1220" stroke="#a78bfa" stroke-width="0.8"/>
  <text x="627" y="118" fill="#c4b5fd" font-size="9" font-weight="600" text-anchor="middle">End-to-End Learning</text>
  <text x="627" y="138" fill="#8b98a5" font-size="8" text-anchor="middle">physics + neural nets +</text>
  <text x="627" y="155" fill="#8b98a5" font-size="8" text-anchor="middle">solvers + renderers</text>
  <text x="627" y="172" fill="#6b7684" font-size="7.5" text-anchor="middle">all jointly optimized</text>

  <text x="380" y="205" fill="#8b98a5" font-size="8" text-anchor="middle">key enabler: automatic differentiation (autodiff) — computes gradients through arbitrary code</text>

  <!-- Examples panel -->
  <rect x="30" y="225" width="345" height="155" rx="6" fill="#080d14" stroke="#233043" stroke-width="1"/>
  <text x="202" y="245" fill="#e6edf3" font-size="10" font-weight="600" text-anchor="middle">Applications</text>

  <text x="50" y="268" fill="#60a5fa" font-size="8.5" font-weight="600">Physics simulation:</text>
  <text x="50" y="284" fill="#8b98a5" font-size="8">differentiable fluid/rigid-body sim → learn controllers</text>

  <text x="50" y="304" fill="#34d399" font-size="8.5" font-weight="600">Neural rendering (NeRF, 3DGS):</text>
  <text x="50" y="320" fill="#8b98a5" font-size="8">differentiable rasterizer → optimize 3D from 2D images</text>

  <text x="50" y="340" fill="#a78bfa" font-size="8.5" font-weight="600">Robotics:</text>
  <text x="50" y="356" fill="#8b98a5" font-size="8">diff. simulator → backprop through physics → learn policy</text>

  <text x="50" y="376" fill="#f59e0b" font-size="8.5" font-weight="600">Chip design (DREAMPlace):</text>
  <text x="50" y="392" fill="#8b98a5" font-size="8">diff. placement → gradient-based cell optimization</text>

  <!-- Frameworks panel -->
  <rect x="390" y="225" width="340" height="155" rx="6" fill="#080d14" stroke="#233043" stroke-width="1"/>
  <text x="560" y="245" fill="#e6edf3" font-size="10" font-weight="600" text-anchor="middle">Frameworks &amp; Autodiff Systems</text>

  <text x="410" y="268" fill="#60a5fa" font-size="8.5" font-weight="600">JAX:</text>
  <text x="448" y="268" fill="#8b98a5" font-size="8">jax.grad() — arbitrary Python + NumPy autodiff</text>

  <text x="410" y="288" fill="#34d399" font-size="8.5" font-weight="600">PyTorch:</text>
  <text x="470" y="288" fill="#8b98a5" font-size="8">autograd — define-by-run dynamic graphs</text>

  <text x="410" y="308" fill="#a78bfa" font-size="8.5" font-weight="600">Taichi:</text>
  <text x="455" y="308" fill="#8b98a5" font-size="8">differentiable GPU kernels for physics</text>

  <text x="410" y="328" fill="#f59e0b" font-size="8.5" font-weight="600">Swift for TF (deprecated):</text>
  <text x="410" y="344" fill="#8b98a5" font-size="8">language-level diff (∂ keyword)</text>

  <text x="410" y="368" fill="#8b98a5" font-size="8.5" font-weight="600">Key technique:</text>
  <text x="410" y="384" fill="#6b7684" font-size="8">reverse-mode AD (backprop) + custom VJPs for non-trivial ops</text>

  <text x="380" y="452" fill="#6b7684" font-size="11" text-anchor="middle">Differentiable programming erases the boundary between simulation and learning — anything computable is optimizable.</text>
</svg>

• Traditional programming: Functions map inputs to outputs — no notion of gradients.
• Differentiable programming: Functions are differentiable — you can compute gradients of outputs with respect to inputs and parameters.
• This enables gradient descent to optimize program parameters — the same technique that trains neural networks.
• Automatic differentiation (autodiff) computes gradients automatically — no need to derive them manually.

Why Differentiable Programming?

• End-to-End Optimization: Optimize entire pipelines, not just individual components — gradients flow through the whole computation.
• Inverse Problems: Given desired outputs, find inputs or parameters that produce them — optimization-based solution.
• Physics-Informed Learning: Incorporate physical laws as differentiable constraints — combine data-driven learning with domain knowledge.
• Unified Framework: Treat traditional algorithms and neural networks uniformly — both are differentiable functions.

How It Works

1. Differentiable Operations: Build programs from operations that have defined gradients — arithmetic, matrix operations, activation functions.

2. Automatic Differentiation: Frameworks (JAX, PyTorch, TensorFlow) automatically compute gradients using the chain rule.

3. Gradient-Based Optimization: Use gradients to adjust parameters — gradient descent, Adam, etc.

4. Backpropagation: Gradients flow backward through the computation graph — from outputs to inputs.

Differentiable Programming Frameworks

• JAX: Python library for high-performance numerical computing with autodiff — functional programming style, JIT compilation.
• PyTorch: Deep learning framework with eager execution and autodiff — widely used for research.
• TensorFlow: Google's framework with static and eager execution modes — production-focused.
• Julia (Zygote): Julia language with powerful autodiff capabilities — designed for scientific computing.

Applications

• Physics Simulations: Differentiable physics engines — optimize physical parameters, learn control policies.
• Example: Optimize robot design by backpropagating through physics simulation.

• Computer Graphics: Differentiable rendering — optimize 3D models to match 2D images.
• Example: Reconstruct 3D shapes from photographs.

• Robotics: Differentiable robot models — learn control policies end-to-end.
• Example: Train robot to manipulate objects by optimizing through forward kinematics.

• Scientific Computing: Solve inverse problems — parameter estimation, data assimilation.
• Example: Infer material properties from experimental measurements.

• Optimization: Solve complex optimization problems using gradient descent.
• Example: Optimize supply chain parameters.

Example: Differentiable Physics

import jax
import jax.numpy as jnp

def simulate_trajectory(initial_velocity, gravity=9.8, time=1.0):
    """Differentiable physics simulation."""
    t = jnp.linspace(0, time, 100)
    height = initial_velocity * t - 0.5 * gravity * t**2
    return height

# Compute gradient of final height w.r.t. initial velocity
grad_fn = jax.grad(lambda v: simulate_trajectory(v)[-1])
gradient = grad_fn(10.0)  # How does final height change with initial velocity?

Differentiable vs. Traditional Programming

• Traditional: Programs are discrete, symbolic — no gradients, optimization requires search or heuristics.
• Differentiable: Programs are continuous, differentiable — gradients enable efficient optimization.
• Hybrid: Combine both — differentiable components for optimization, discrete logic for control flow.

Challenges

• Discontinuities: Not all operations are differentiable — conditionals, discrete choices, non-smooth functions.
• Memory: Autodiff requires storing intermediate values for backpropagation — memory-intensive for long computations.
• Numerical Stability: Gradients can explode or vanish — requires careful numerical handling.
• Debugging: Gradient bugs can be subtle — incorrect gradients may not cause obvious errors.

Benefits

• Powerful Optimization: Gradient descent is highly effective — can optimize millions of parameters.
• Composability: Differentiable components compose — gradients flow through arbitrary compositions.
• Flexibility: Applicable to diverse domains — physics, graphics, robotics, optimization.
• Integration with Deep Learning: Seamlessly combine traditional algorithms with neural networks.

Differentiable Programming in AI

• Neural Architecture Search: Optimize neural network architectures using gradients.
• Meta-Learning: Learn learning algorithms themselves — optimize the optimization process.
• Inverse Graphics: Infer 3D scenes from 2D images using differentiable rendering.
• Differentiable Simulators: Train agents in simulation with gradients flowing through the simulator.

Differentiable programming is a paradigm shift — it extends the power of gradient-based optimization from neural networks to arbitrary programs, enabling end-to-end learning and optimization of complex systems.