stable diffusion, multimodal ai
Stable Diffusion generates images through latent space diffusion with text conditioning.
311 technical terms and definitions
Stable Diffusion generates images through latent space diffusion with text conditioning.
Stable Diffusion generates images from text. Latent diffusion for efficiency. Open source, highly customizable.
Stack AI is enterprise no-code AI platform.
Answer programming questions.
Train meta-model on predictions from base models.
Chemical treatment to enhance defects.
# Stanford HAI and Mathematics ## A Comprehensive Overview **Stanford HAI** (Stanford Institute for Human-Centered Artificial Intelligence), founded in 2019, engages with mathematics in multiple significant ways—tracking AI's mathematical capabilities, applying AI to mathematics education, reasoning, and scientific discovery. ## 1. AI Performance on Mathematical Benchmarks Stanford HAI's annual **AI Index Report** closely monitors how AI systems perform on mathematical tasks. ### 1.1 Mathematical Olympiad Performance - **Test-time compute breakthrough**: OpenAI's o1 model achieved $74.4\%$ on an International Mathematical Olympiad (IMO) qualifying exam, compared to GPT-4o's $9.3\%$ - AI models excel at IMO-style problems but struggle with complex reasoning benchmarks like **PlanBench** - The improvement represents a multiplicative factor of approximately: $$ \text{Improvement Factor} = \frac{74.4}{9.3} \approx 8.0\times $$ ### 1.2 MATH Benchmark Progress The MATH benchmark contains over 10,000 competition-level mathematics problems: | Year | Top Score | Human Expert Standard | |------|-----------|----------------------| | 2022 | $65\%$ | $90\%$ | | 2024 | $84\%$ | $90\%$ | - Performance gap reduction: $$ \Delta_{\text{gap}} = (90 - 65) - (90 - 84) = 25 - 6 = 19 \text{ percentage points} $$ ### 1.3 Benchmark Convergence (2023 → 2024) Performance gaps between top models narrowed dramatically: | Benchmark | Gap (End 2023) | Gap (End 2024) | Reduction | |-----------|----------------|----------------|-----------| | MMLU | $17.5\%$ | $0.3\%$ | $-17.2$ pp | | MMMU | $13.5\%$ | $8.1\%$ | $-5.4$ pp | | MATH | $24.3\%$ | $1.6\%$ | $-22.7$ pp | | HumanEval | $31.6\%$ | $3.7\%$ | $-27.9$ pp | - The Chatbot Arena Leaderboard Elo score difference between top and 10th-ranked model: - 2023: $11.9\%$ - 2025: $5.4\%$ - Top two models gap: $4.9\% \to 0.7\%$ ### 1.4 Current Limitations Despite progress, AI systems face persistent challenges: - **Complex reasoning**: LLMs struggle with benchmarks like MMMU - **Logic tasks**: Difficulty solving problems even when provably correct solutions exist - **Arithmetic & planning**: Reliability issues in high-stakes, accuracy-critical settings $$ \text{Reliability}_{\text{math}} = f(\text{problem complexity}, \text{reasoning depth}, \text{verification}) $$ ## 2. AI for Mathematics Education Stanford HAI funds research on AI tools for math teaching and learning. ### 2.1 Scaffolding for Math Teachers **Key Research Findings:** - LLMs help middle school math teachers structure tiered lessons for diverse skill levels - AI-generated "warmup" exercises rated **better than human-created ones** in: - Accessibility - Alignment with learning objectives - Teacher preference **Scaffolding Formula:** $$ \text{Effective Scaffolding} = \text{Prior Knowledge Activation} + \text{Skill-Level Differentiation} + \text{Curriculum Alignment} $$ ### 2.2 Research Team - **Dora Demszky** (Assistant Professor, Education Data Science) - Combines: Machine Learning + NLP + Linguistics + Practitioner Input - Partnership: Network of school districts - Goal: AI-powered resources for middle school math teachers ### 2.3 Current AI Limitations in Math Education | Capability | AI Performance | |------------|----------------| | Text-based content | ✅ Strong | | Story problems | ✅ Strong | | Written descriptions | ✅ Strong | | Visual approaches | ❌ Weak | | Diagrams | ❌ Weak | | Graphs | ❌ Weak | **Observed Issue:** > "The chatbot would produce perfect sentences that exhibited top-quality teaching techniques, such as positive reinforcement, but fail to get to the right mathematical answer." ## 3. Understanding Math Learning Disabilities with AI ### 3.1 Digital Twins Research HAI-funded study using AI to model children's mathematical cognition: - **Method**: AI + fMRI (functional Magnetic Resonance Imaging) - **Subjects**: 45 students, ages 7-9 - 21 with math learning disabilities - 24 typically developing ### 3.2 Key Finding: Training Requirements $$ \text{Training}_{\text{disability}} \approx 2 \times \text{Training}_{\text{typical}} $$ - AI twins modeling math learning disabilities required **nearly twice as much training** - Critical insight: They eventually reach **equivalent performance** ### 3.3 Implications - **Personalized learning plans**: Tailored to individual learning styles - **Predictive instruction**: AI can predict which instruction types work best - **Remediation strategies**: New hope for effective interventions **Research Team:** - Vinod Menon (Professor, Psychiatry & Behavioral Sciences) - Anthony Strock (Postdoctoral Scholar) - Percy Mistry (Research Scholar) ## 4. AI for Scientific/Mathematical Discovery ### 4.1 Breakthrough Applications | System | Application | Achievement | |--------|-------------|-------------| | **AlphaDev** | Algorithmic sorting | Up to $70\%$ faster for shorter sequences | | **GNoME** | Materials discovery | 2+ million new crystal structures | | **AlphaMissence** | Genetic classification | $89\%$ of 71 million missense mutations | ### 4.2 Computational Scale Previous human annotation capacity: $$ \text{Human Classification Rate} = 0.1\% \text{ of all missense mutations} $$ AI achievement: $$ \text{AI Classification Rate} \approx 89\% \text{ of all missense mutations} $$ Improvement factor: $$ \frac{89}{0.1} = 890\times $$ ### 4.3 AI in Mathematical Research **Theorem Proving with AI:** - Formal verification using proof assistants (Lean, Coq, Isabelle) - Autoformalization: Converting informal proofs to machine-verifiable formats **The Curry-Howard Correspondence:** $$ \text{Propositions} \cong \text{Types} $$ $$ \text{Proofs} \cong \text{Programs} $$ ## 5. HAI Graduate Fellows in Mathematics-Related Research ### 5.1 Current Fellows - **Victoria Delaney** - Focus: Mathematics education, computer science, learning sciences - Interest: Technology for pedagogy in advanced mathematics - **Faidra Monachou** - Method: Mathematical modeling + data-driven simulations - Focus: Socioeconomic problems, resource allocation, fair admissions policies - **Evan Munro** - Intersection: Machine learning $\cap$ Econometrics $\cap$ Mechanism design ## 6. Key Mathematical Insights from AI Index 2025 ### 6.1 Performance Trajectory The rate of AI improvement on mathematical tasks follows approximately: $$ P(t) = P_0 \cdot e^{kt} $$ where: - $P(t)$ = Performance at time $t$ - $P_0$ = Initial performance - $k$ = Growth rate constant ### 6.2 Benchmark Saturation Timeline For major benchmarks: | Benchmark | Saturation Year (Est.) | Notes | |-----------|------------------------|-------| | MMLU | 2024 | Gap $< 1\%$ | | MATH | 2024-2025 | Gap $\approx 1.6\%$ | | IMO Problems | 2024+ | Gold medal level achieved | | Complex Reasoning | Unknown | Significant challenges remain | ### 6.3 The Reasoning Gap $$ \text{Performance Gap} = \begin{cases} \text{Small} & \text{if } \text{problem} \in \{\text{pattern matching, calculation}\} \\ \text{Large} & \text{if } \text{problem} \in \{\text{novel reasoning, planning}\} \end{cases} $$ ## 7. Statistics ### 7.1 Quick Facts - **HAI Founded**: 2019 - **Annual Report**: AI Index (434+ pages) - **Seed Grants**: ~25 grants, up to \$75,000 each - **Focus Areas**: 1. Intelligence (novel AI technologies) 2. Applications (augmenting human capabilities) 3. Impact (societal effects of AI) ### 7.2 Mathematical AI Milestones ``` Timeline of Mathematical AI Progress: ├── 2022: MATH benchmark top score = 65% ├── 2023: New benchmarks introduced (MMMU, GPQA, SWE-bench) ├── 2024: │ ├── MATH score reaches 84% │ ├── o1 achieves 74.4% on IMO qualifying exam │ ├── AlphaProof achieves IMO silver medal level │ └── Performance gaps narrow to near-parity └── 2025: ├── Gold medal level on IMO problems ├── Complex reasoning remains challenging └── Focus shifts to reliability and verification ``` ## 8. Formulas and Equations Reference ### 8.1 Performance Metrics **Accuracy:** $$ \text{Accuracy} = \frac{\text{Correct Solutions}}{\text{Total Problems}} \times 100\% $$ **Improvement Rate:** $$ r = \frac{P_{\text{new}} - P_{\text{old}}}{P_{\text{old}}} \times 100\% $$ **Benchmark Gap:** $$ G = |P_{\text{human}} - P_{\text{AI}}| $$ ### 8.2 Learning Disability Model Training requirement ratio: $$ R = \frac{T_{\text{LD}}}{T_{\text{typical}}} \approx 2.0 $$ where: - $T_{\text{LD}}$ = Training iterations for learning disability model - $T_{\text{typical}}$ = Training iterations for typical model ### 8.3 Scientific Discovery Scale Classification improvement: $$ I = \frac{C_{\text{AI}}}{C_{\text{human}}} = \frac{0.89 \times 71\text{M}}{0.001 \times 71\text{M}} = 890 $$
# Stanford Materials Science & Mathematical Modeling ## Comprehensive Overview ## 1. Introduction Stanford University offers robust programs at the intersection of **materials science** and **mathematical modeling**, spanning multiple departments and institutes. The approach is distinctly interdisciplinary, connecting: - Mathematics - Physics - Computer Science - Engineering These disciplines work together to tackle materials challenges from **quantum** to **continuum** scales. ## 2. Department of Materials Science and Engineering (MSE) ### 2.1 Overview Stanford's MSE department has a dedicated research thrust in **Materials Computation, Theory & Design**. **Key Focus Areas:** - Development and application of methods to compute atomic and electronic structure of materials - Materials for electronic applications - Nano-electromechanics and energy - Leveraging statistics and machine learning to accelerate materials design ### 2.2 Research Themes | Theme | Description | |-------|-------------| | **Biomaterials & Bio-interfaces** | Materials interacting with biological systems | | **Electronic, Magnetic & Photonic Materials** | Functional electronic materials | | **Materials for Sustainability** | Eco-friendly materials development | | **Mechanical Behavior & Structural Materials** | Strength and deformation studies | | **Novel Characterization Methods** | Advanced imaging and spectroscopy | | **Novel Synthesis & Fabrication Methods** | New manufacturing approaches | | **Soft Matter & Hybrid Materials** | Polymers and composites | ## 3. Institute for Computational & Mathematical Engineering (ICME) ### 3.1 Mission ICME is a degree-granting (MS/PhD) interdisciplinary institute at the intersection of: - **Mathematics** - **Computing** - **Engineering** - **Applied Sciences** ### 3.2 Training Areas ICME trains students in: - Matrix computations - Computational probability - Combinatorial optimization - Optimization theory - Stochastics - Numerical solution of PDEs - Parallel computing algorithms ### 3.3 Research Areas - Aerodynamics and space applications - Fluid dynamics - Protein folding - Data science and machine learning - Ocean dynamics and climate modeling - Reservoir engineering - Computer graphics - Financial mathematics ## 4. Mathematical Modeling Methodologies ### 4.1 Density Functional Theory (DFT) DFT is a computational quantum mechanical modeling method used to investigate electronic structure of many-body systems. #### 4.1.1 Fundamental Equations The **Kohn-Sham equations** form the foundation: $$ \left[ -\frac{\hbar^2}{2m}\nabla^2 + V_{\text{eff}}(\mathbf{r}) \right] \psi_i(\mathbf{r}) = \epsilon_i \psi_i(\mathbf{r}) $$ where: - $\psi_i(\mathbf{r})$ — Kohn-Sham orbital - $\epsilon_i$ — Orbital energy - $V_{\text{eff}}(\mathbf{r})$ — Effective potential The effective potential is: $$ V_{\text{eff}}(\mathbf{r}) = V_{\text{ext}}(\mathbf{r}) + V_H(\mathbf{r}) + V_{\text{xc}}(\mathbf{r}) $$ where: - $V_{\text{ext}}$ — External potential (nuclei) - $V_H$ — Hartree potential (classical electron-electron) - $V_{\text{xc}}$ — Exchange-correlation potential #### 4.1.2 Electron Density $$ n(\mathbf{r}) = \sum_{i=1}^{N} |\psi_i(\mathbf{r})|^2 $$ #### 4.1.3 Applications - Electronic structure calculations - Band gap predictions - Defect analysis - Surface chemistry - Battery materials design ### 4.2 Molecular Dynamics (MD) #### 4.2.1 Equations of Motion Newton's equations govern atomic motion: $$ m_i \frac{d^2 \mathbf{r}_i}{dt^2} = \mathbf{F}_i = -\nabla_i U(\mathbf{r}_1, \mathbf{r}_2, \ldots, \mathbf{r}_N) $$ where: - $m_i$ — Mass of atom $i$ - $\mathbf{r}_i$ — Position vector - $\mathbf{F}_i$ — Force on atom $i$ - $U$ — Potential energy function #### 4.2.2 Velocity Verlet Algorithm Position update: $$ \mathbf{r}(t + \Delta t) = \mathbf{r}(t) + \mathbf{v}(t)\Delta t + \frac{1}{2}\mathbf{a}(t)\Delta t^2 $$ Velocity update: $$ \mathbf{v}(t + \Delta t) = \mathbf{v}(t) + \frac{1}{2}[\mathbf{a}(t) + \mathbf{a}(t + \Delta t)]\Delta t $$ #### 4.2.3 Statistical Mechanics Connection **Partition Function:** $$ Z = \int e^{-\beta H(\mathbf{p}, \mathbf{q})} d\mathbf{p} \, d\mathbf{q} $$ where $\beta = 1/(k_B T)$ **Ensemble Average:** $$ \langle A \rangle = \frac{1}{Z} \int A(\mathbf{p}, \mathbf{q}) e^{-\beta H} d\mathbf{p} \, d\mathbf{q} $$ ### 4.3 Phase-Field Modeling #### 4.3.1 Order Parameter Evolution The **Allen-Cahn equation** for non-conserved order parameter $\phi$: $$ \frac{\partial \phi}{\partial t} = -L \frac{\delta F}{\delta \phi} $$ The **Cahn-Hilliard equation** for conserved order parameter: $$ \frac{\partial c}{\partial t} = \nabla \cdot \left( M \nabla \frac{\delta F}{\delta c} \right) $$ where: - $L$ — Kinetic coefficient - $M$ — Mobility - $F$ — Free energy functional #### 4.3.2 Free Energy Functional The **Ginzburg-Landau** free energy: $$ F[\phi] = \int_\Omega \left[ f(\phi) + \frac{\kappa}{2}|\nabla \phi|^2 \right] dV $$ where: - $f(\phi)$ — Bulk free energy density (double-well potential) - $\kappa$ — Gradient energy coefficient - $\Omega$ — Domain #### 4.3.3 Double-Well Potential $$ f(\phi) = \frac{W}{4}\phi^2(1-\phi)^2 $$ where $W$ is the barrier height. #### 4.3.4 Applications - **Solidification**: Dendrite growth modeling - **Phase transformations**: Austenite → Ferrite - **Fracture mechanics**: Crack propagation - **Microstructure evolution**: Grain growth ### 4.4 Finite Element Method (FEM) #### 4.4.1 Weak Formulation For the **elasticity problem**, find $\mathbf{u} \in V$ such that: $$ \int_\Omega \boldsymbol{\sigma}(\mathbf{u}) : \boldsymbol{\varepsilon}(\mathbf{v}) \, dV = \int_\Omega \mathbf{f} \cdot \mathbf{v} \, dV + \int_{\Gamma_N} \mathbf{t} \cdot \mathbf{v} \, dS \quad \forall \mathbf{v} \in V_0 $$ where: - $\boldsymbol{\sigma}$ — Stress tensor - $\boldsymbol{\varepsilon}$ — Strain tensor - $\mathbf{f}$ — Body force - $\mathbf{t}$ — Traction on boundary $\Gamma_N$ #### 4.4.2 Constitutive Relations **Hooke's Law** (linear elasticity): $$ \boldsymbol{\sigma} = \mathbb{C} : \boldsymbol{\varepsilon} $$ In matrix form for isotropic materials: $$ \sigma_{ij} = \lambda \varepsilon_{kk} \delta_{ij} + 2\mu \varepsilon_{ij} $$ where: - $\lambda, \mu$ — Lamé parameters - $\delta_{ij}$ — Kronecker delta **Lamé parameters:** $$ \lambda = \frac{E\nu}{(1+\nu)(1-2\nu)}, \quad \mu = \frac{E}{2(1+\nu)} $$ #### 4.4.3 Strain-Displacement Relationship $$ \varepsilon_{ij} = \frac{1}{2}\left( \frac{\partial u_i}{\partial x_j} + \frac{\partial u_j}{\partial x_i} \right) $$ ### 4.5 Multiscale Modeling #### 4.5.1 Homogenization Theory For periodic microstructure with period $\epsilon$: $$ u^\epsilon(\mathbf{x}) = u_0(\mathbf{x}) + \epsilon u_1\left(\mathbf{x}, \frac{\mathbf{x}}{\epsilon}\right) + \epsilon^2 u_2\left(\mathbf{x}, \frac{\mathbf{x}}{\epsilon}\right) + \ldots $$ **Effective properties:** $$ \bar{\mathbb{C}}_{ijkl} = \frac{1}{|Y|} \int_Y \mathbb{C}_{ijkl} \left( \delta_{km} + \frac{\partial \chi_k^{mn}}{\partial y_m} \right) dY $$ where $\chi$ solves the cell problem. #### 4.5.2 Scale Hierarchy | Scale | Length | Method | |-------|--------|--------| | **Quantum** | $10^{-10}$ m | DFT, QMC | | **Atomistic** | $10^{-9}$ m | MD, MC | | **Mesoscale** | $10^{-6}$ m | Phase-field, KMC | | **Continuum** | $10^{-3}$ m | FEM, FDM | | **Macroscale** | $10^0$ m | Structural analysis | ## 5. Machine Learning for Materials ### 5.1 Neural Network Potentials #### 5.1.1 High-Dimensional Neural Network Potential The total energy is decomposed as: $$ E_{\text{total}} = \sum_{i=1}^{N} E_i(\mathbf{G}_i) $$ where $\mathbf{G}_i$ is the **symmetry function** descriptor for atom $i$. #### 5.1.2 Behler-Parrinello Symmetry Functions **Radial symmetry function:** $$ G_i^{\text{rad}} = \sum_{j \neq i} e^{-\eta(R_{ij} - R_s)^2} f_c(R_{ij}) $$ **Angular symmetry function:** $$ G_i^{\text{ang}} = 2^{1-\zeta} \sum_{j,k \neq i} (1 + \lambda \cos\theta_{ijk})^\zeta e^{-\eta(R_{ij}^2 + R_{ik}^2 + R_{jk}^2)} f_c(R_{ij}) f_c(R_{ik}) f_c(R_{jk}) $$ where: - $f_c(R)$ — Cutoff function - $\eta, \zeta, \lambda, R_s$ — Hyperparameters #### 5.1.3 Cutoff Function $$ f_c(R) = \begin{cases} \frac{1}{2}\left[\cos\left(\frac{\pi R}{R_c}\right) + 1\right] & R \leq R_c \\ 0 & R > R_c \end{cases} $$ ### 5.2 Graph Neural Networks for Materials #### 5.2.1 Message Passing Framework $$ \mathbf{h}_i^{(l+1)} = \sigma\left( \mathbf{W}_1^{(l)} \mathbf{h}_i^{(l)} + \sum_{j \in \mathcal{N}(i)} \mathbf{W}_2^{(l)} \mathbf{h}_j^{(l)} \odot \mathbf{e}_{ij} \right) $$ where: - $\mathbf{h}_i^{(l)}$ — Node features at layer $l$ - $\mathcal{N}(i)$ — Neighbors of node $i$ - $\mathbf{e}_{ij}$ — Edge features - $\sigma$ — Activation function ### 5.3 Applications - **Property Prediction**: Band gaps, formation energies, elastic moduli - **Structure Prediction**: Crystal structure search - **Inverse Design**: Target property → optimal composition - **Accelerated Screening**: High-throughput materials discovery ## 6. Recent Breakthroughs (2025) ### 6.1 Poisson Model for Heterogeneous Materials **Published**: October 2025, Physical Review Letters Stanford researchers solved the famous **Poisson model** for heterogeneous materials—a problem unsolved for decades in statistical physics. #### 6.1.1 Multipoint Correlation Functions For a Poisson tessellation, the $n$-point correlation function: $$ S_n(\mathbf{r}_1, \mathbf{r}_2, \ldots, \mathbf{r}_n) = \mathbb{P}[\text{all points in same phase}] $$ #### 6.1.2 Two-Point Correlation $$ S_2(r) = e^{-\lambda r} $$ where $\lambda$ is the line density parameter. #### 6.1.3 Stochastic Geometry Framework The **Poisson line process** intensity: $$ \Lambda(B) = \int_B \lambda(\mathbf{x}) d\mathbf{x} $$ ### 6.2 Applications - **Concrete optimization**: Design stronger, climate-friendlier materials - **Groundwater management**: Predict flow in porous media - **Nuclear waste storage**: Evaluate subsurface storage sites - **Composite materials**: Control microstructure for desired properties ## 7. Key Equations and Formulations ### 7.1 Thermodynamics #### 7.1.1 Gibbs Free Energy $$ G = H - TS = U + PV - TS $$ #### 7.1.2 Chemical Potential $$ \mu_i = \left(\frac{\partial G}{\partial n_i}\right)_{T,P,n_{j \neq i}} $$ #### 7.1.3 Phase Equilibrium At equilibrium between phases $\alpha$ and $\beta$: $$ \mu_i^\alpha = \mu_i^\beta \quad \forall i $$ ### 7.2 Transport Phenomena #### 7.2.1 Fick's Laws of Diffusion **First Law:** $$ \mathbf{J} = -D \nabla c $$ **Second Law:** $$ \frac{\partial c}{\partial t} = D \nabla^2 c $$ where: - $\mathbf{J}$ — Diffusion flux - $D$ — Diffusion coefficient - $c$ — Concentration #### 7.2.2 Heat Conduction (Fourier's Law) $$ \mathbf{q} = -k \nabla T $$ $$ \rho c_p \frac{\partial T}{\partial t} = k \nabla^2 T + Q $$ ### 7.3 Quantum Mechanics #### 7.3.1 Time-Independent Schrödinger Equation $$ \hat{H}\Psi = E\Psi $$ where the Hamiltonian: $$ \hat{H} = -\frac{\hbar^2}{2m}\nabla^2 + V(\mathbf{r}) $$ #### 7.3.2 Born-Oppenheimer Approximation Total wavefunction separation: $$ \Psi_{\text{total}}(\mathbf{r}, \mathbf{R}) \approx \psi_{\text{el}}(\mathbf{r}; \mathbf{R}) \chi_{\text{nuc}}(\mathbf{R}) $$ ## 8. Course Curriculum ### 8.1 Materials Science Courses | Course | Title | Topics | |--------|-------|--------| | **MATSCI 331** | Computational Materials Science at the Atomic Scale | DFT, tight-binding, empirical potentials, ML-based property prediction | | **MATSCI 165/175** | Nanoscale Materials Physics Computation Lab | Java-based atomistic simulations, Monte Carlo methods | | **MATSCI 142** | Quantum Mechanics of Materials | Electronic structure, band theory | | **MATSCI 143** | Materials Structure and Characterization | X-ray diffraction, electron microscopy | ### 8.2 ICME/CME Courses | Course | Title | Topics | |--------|-------|--------| | **CME 232** | Introduction to Computational Mechanics | FEM, BEM, variational methods | | **CME 302** | Numerical Linear Algebra | Matrix computations, eigenvalue problems | | **CME 306** | Mathematical Methods for Fluids, Solids and Interfaces | PDEs, level sets, Navier-Stokes | | **CME 356/ME 412** | Engineering Functional Analysis and Finite Elements | Sobolev spaces, convergence analysis | | **CME 216** | Machine Learning for Computational Engineering | Deep learning, physics-informed ML | ### 8.3 Mechanics Courses | Course | Title | Topics | |--------|-------|--------| | **ME 335A/B** | Finite Element Analysis | Continuum mechanics, numerical methods | | **ME 338** | Continuum Mechanics | Tensor analysis, constitutive equations | | **ME 340A** | Theory and Applications of Elasticity | Stress, strain, boundary value problems | ## 9. Research Groups ### 9.1 Reed Group (Prof. Evan Reed) **Focus Areas:** - Machine learning for materials property prediction - Battery materials and energy technologies - Fast algorithms for complex chemistry - Shock compression and phase transitions - DFT-based photoemission modeling ### 9.2 Computational Mechanics of Materials Lab (Prof. Christian Linder) **Focus Areas:** - Phase-field fracture modeling - Micromechanically motivated continuum approaches - Finite deformation mechanics - Ductile and brittle fracture ### 9.3 Living Matter Lab (Prof. Ellen Kuhl) **Focus Areas:** - Physics-informed neural networks - Bayesian inference for materials - Multiscale modeling with machine learning ### 9.4 Z-Energy Lab (Prof. X. L. Zheng) **Focus Areas:** - Machine learning for materials design - Data-driven feature-property relationships - Electrochemical materials ## 10. Summary ### 10.1 Mathematical Tools Overview | **Method** | **Scale** | **Key Equations** | **Applications** | |------------|-----------|-------------------|------------------| | DFT | Quantum | Kohn-Sham: $\hat{H}_{\text{KS}}\psi = \epsilon\psi$ | Electronic structure, band gaps | | MD | Atomistic | $m\ddot{\mathbf{r}} = -\nabla U$ | Phase transitions, diffusion | | Phase-Field | Mesoscale | $\partial_t \phi = -L \delta F/\delta\phi$ | Microstructure, fracture | | FEM | Continuum | $\int \boldsymbol{\sigma}:\boldsymbol{\varepsilon} \, dV = \int \mathbf{f}\cdot\mathbf{v} \, dV$ | Structural mechanics | | ML Potentials | Multi-scale | $E = \sum_i E_i(\mathbf{G}_i)$ | Accelerated simulations | | Stochastic | Multi-scale | $S_n(\mathbf{r}_1,\ldots,\mathbf{r}_n)$ | Heterogeneous materials | ### 10.2 Key Takeaways 1. **Interdisciplinary Approach**: Stanford integrates mathematics, physics, CS, and engineering 2. **Multi-scale Philosophy**: From quantum ($10^{-10}$ m) to continuum ($10^0$ m) 3. **Data-Driven Methods**: Machine learning increasingly central to materials discovery 4. **Theoretical Rigor**: Strong mathematical foundations across all methods 5. **Practical Applications**: Focus on energy, sustainability, and advanced manufacturing ## Symbol Glossary | Symbol | Description | Units | |--------|-------------|-------| | $\psi$ | Wavefunction | — | | $n(\mathbf{r})$ | Electron density | m$^{-3}$ | | $\boldsymbol{\sigma}$ | Stress tensor | Pa | | $\boldsymbol{\varepsilon}$ | Strain tensor | — | | $\phi$ | Phase-field order parameter | — | | $F$ | Free energy | J | | $D$ | Diffusion coefficient | m$^2$/s | | $k$ | Thermal conductivity | W/(m·K) | | $\mathbb{C}$ | Elastic stiffness tensor | Pa | | $\beta$ | Inverse temperature | J$^{-1}$ | ## Useful Constants | Constant | Symbol | Value | |----------|--------|-------| | Planck's constant | $\hbar$ | $1.055 \times 10^{-34}$ J·s | | Boltzmann constant | $k_B$ | $1.381 \times 10^{-23}$ J/K | | Electron mass | $m_e$ | $9.109 \times 10^{-31}$ kg | | Bohr radius | $a_0$ | $5.292 \times 10^{-11}$ m | | Hartree energy | $E_h$ | $4.360 \times 10^{-18}$ J |
Open-source code generation model.
Multi-domain image translation.
For startup ideas I help clarify problem, customer, value prop, business model, and go-to-market steps at a practical level.
State space models represent sequences through hidden states with linear recurrence.
# State Space Models and Time Series Analysis ## Introduction State space models (SSMs) provide a powerful and flexible framework for modeling dynamic systems and time series data. They represent systems through: - **Hidden states**: Latent variables that evolve over time according to system dynamics - **Observations**: Measured outputs that depend on the hidden states - **Probabilistic framework**: Explicit modeling of uncertainty in both dynamics and measurements ### Why State Space Models? * They unify many classical time series models under a common framework * They naturally handle multivariate data with complex dependencies * They provide principled uncertainty quantification * They accommodate irregular sampling and missing data * They bridge classical statistics and modern machine learning ## Mathematical Framework ### General State Space Representation The state space model consists of two fundamental equations: **State Equation (System/Transition Equation):** x_t = f(x_{t-1}, u_t, w_t) where: - x_t ∈ R^n is the state vector at time t - f(.) is the state transition function - u_t is the control input (optional) - w_t ~ N(0, Q_t) is process noise **Observation Equation (Measurement Equation):** y_t = h(x_t, v_t) where: - y_t ∈ R^m is the observation vector at time t - h(.) is the observation function - v_t ~ N(0, R_t) is measurement noise ### Linear Gaussian State Space Model For the special case of linear dynamics with Gaussian noise: **State Equation:** x_t = F_t × x_{t-1} + B_t × u_t + w_t **Observation Equation:** y_t = H_t × x_t + v_t where: - F_t ∈ R^{n×n} is the state transition matrix - H_t ∈ R^{m×n} is the observation matrix - B_t ∈ R^{n×p} is the control input matrix - w_t ~ N(0, Q_t) with covariance Q_t - v_t ~ N(0, R_t) with covariance R_t ### Initial Conditions Initial conditions: x_0 ~ N(mu_0, Sigma_0) ## Core Components ### The State Vector The state vector $\mathbf{x}_t$ contains all information needed to describe the system at time $t$: * **Components**: Can include levels, trends, seasonal components, regression effects * **Dimensionality**: Chosen based on model complexity and domain knowledge * **Interpretation**: May be directly interpretable (e.g., position, velocity) or abstract latent features ### Transition Dynamics The function $f(\cdot)$ or matrix $\mathbf{F}_t$ governs how states evolve: * **Time-invariant**: $\mathbf{F}_t = \mathbf{F}$ (constant dynamics) * **Time-varying**: $\mathbf{F}_t$ changes over time (adaptive systems) * **Nonlinear**: $f(\cdot)$ is nonlinear (e.g., neural network) * **Stochastic**: Process noise $\mathbf{w}_t$ captures unpredictable variations ### Observation Process The function $h(\cdot)$ or matrix $\mathbf{H}_t$ links hidden states to observations: * **Full observability**: $\mathbf{H}_t = \mathbf{I}$ (identity matrix) * **Partial observability**: Only some state components are measured * **Noisy measurements**: Measurement noise $\mathbf{v}_t$ represents sensor uncertainty * **Nonlinear observations**: $h(\cdot)$ can be arbitrarily complex ## Classical Time Series as State Space Models Many traditional time series models are special cases of state space models. ### 1. Random Walk **Model:** $$y_t = y_{t-1} + w_t, \quad w_t \sim \mathcal{N}(0, \sigma^2_w)$$ **State Space Form:** - State: $x_t = y_t$ - State equation: $x_t = x_{t-1} + w_t$ - Observation equation: $y_t = x_t$ **Matrices:** $$\mathbf{F} = 1, \quad \mathbf{H} = 1, \quad \mathbf{Q} = \sigma^2_w, \quad \mathbf{R} = 0$$ ### 2. Local Level Model **Model:** $$ \begin{aligned} y_t &= \mu_t + v_t, \quad v_t \sim \mathcal{N}(0, \sigma^2_v) \\ \mu_t &= \mu_{t-1} + w_t, \quad w_t \sim \mathcal{N}(0, \sigma^2_w) \end{aligned} $$ **State Space Form:** - State: $x_t = \mu_t$ (level) - State equation: $x_t = x_{t-1} + w_t$ - Observation equation: $y_t = x_t + v_t$ ### 3. Local Linear Trend Model **Model:** $$ \begin{aligned} y_t &= \mu_t + v_t \\ \mu_t &= \mu_{t-1} + \beta_{t-1} + w_t^{(1)} \\ \beta_t &= \beta_{t-1} + w_t^{(2)} \end{aligned} $$ **State Space Form:** - State: $\mathbf{x}_t = [\mu_t, \beta_t]^T$ (level and slope) $$ \begin{bmatrix} \mu_t \\ \beta_t \end{bmatrix} = \begin{bmatrix} 1 & 1 \\ 0 & 1 \end{bmatrix} \begin{bmatrix} \mu_{t-1} \\ \beta_{t-1} \end{bmatrix} + \begin{bmatrix} w_t^{(1)} \\ w_t^{(2)} \end{bmatrix} $$ $$ y_t = \begin{bmatrix} 1 & 0 \end{bmatrix} \begin{bmatrix} \mu_t \\ \beta_t \end{bmatrix} + v_t $$ ### 4. ARMA(p,q) Models **ARMA(p,q) Model:** $$y_t = \phi_1 y_{t-1} + \cdots + \phi_p y_{t-p} + \epsilon_t + \theta_1 \epsilon_{t-1} + \cdots + \theta_q \epsilon_{t-q}$$ **State Space Representation:** For ARMA(p,q) with $r = \max(p, q+1)$: $$ \mathbf{x}_t = \begin{bmatrix} y_t \\ y_{t-1} \\ \vdots \\ y_{t-r+1} \end{bmatrix} $$ $$ \mathbf{F} = \begin{bmatrix} \phi_1 & \phi_2 & \cdots & \phi_{r-1} & \phi_r \\ 1 & 0 & \cdots & 0 & 0 \\ 0 & 1 & \cdots & 0 & 0 \\ \vdots & \vdots & \ddots & \vdots & \vdots \\ 0 & 0 & \cdots & 1 & 0 \end{bmatrix}, \quad \mathbf{H} = \begin{bmatrix} 1 & 0 & \cdots & 0 \end{bmatrix} $$ ### 5. Seasonal Models **Seasonal Component (period $s$):** $$\gamma_t = -\sum_{j=1}^{s-1} \gamma_{t-j} + w_t^{(\gamma)}$$ **State Space Form:** $$ \mathbf{x}_t^{(\gamma)} = \begin{bmatrix} \gamma_t \\ \gamma_{t-1} \\ \vdots \\ \gamma_{t-s+2} \end{bmatrix} $$ $$ \mathbf{F}^{(\gamma)} = \begin{bmatrix} -1 & -1 & -1 & \cdots & -1 \\ 1 & 0 & 0 & \cdots & 0 \\ 0 & 1 & 0 & \cdots & 0 \\ \vdots & \vdots & \ddots & \ddots & \vdots \\ 0 & 0 & \cdots & 1 & 0 \end{bmatrix} $$ ## Estimation and Inference ### Kalman Filter (Linear Gaussian Case) The Kalman filter provides optimal recursive estimates of the state given observations. #### Prediction Step (Time Update) **State prediction:** $$\hat{\mathbf{x}}_{t|t-1} = \mathbf{F}_t \hat{\mathbf{x}}_{t-1|t-1} + \mathbf{B}_t \mathbf{u}_t$$ **Covariance prediction:** $$\mathbf{P}_{t|t-1} = \mathbf{F}_t \mathbf{P}_{t-1|t-1} \mathbf{F}_t^T + \mathbf{Q}_t$$ where: - $\hat{\mathbf{x}}_{t|t-1}$ is the predicted state (prior) - $\mathbf{P}_{t|t-1}$ is the predicted state covariance #### Update Step (Measurement Update) **Innovation (measurement residual):** $$\mathbf{e}_t = \mathbf{y}_t - \mathbf{H}_t \hat{\mathbf{x}}_{t|t-1}$$ **Innovation covariance:** $$\mathbf{S}_t = \mathbf{H}_t \mathbf{P}_{t|t-1} \mathbf{H}_t^T + \mathbf{R}_t$$ **Kalman gain:** $$\mathbf{K}_t = \mathbf{P}_{t|t-1} \mathbf{H}_t^T \mathbf{S}_t^{-1}$$ **State update:** $$\hat{\mathbf{x}}_{t|t} = \hat{\mathbf{x}}_{t|t-1} + \mathbf{K}_t \mathbf{e}_t$$ **Covariance update:** $$\mathbf{P}_{t|t} = (\mathbf{I} - \mathbf{K}_t \mathbf{H}_t) \mathbf{P}_{t|t-1}$$ #### Properties * **Optimality**: Minimizes mean squared error under linear Gaussian assumptions * **Recursive**: Only current state and covariance needed (no need to store history) * **Computational complexity**: $O(n^3 + m^3)$ per time step * **Stability**: Can be implemented in numerically stable forms (e.g., Joseph form, square root form) ### Kalman Smoother (RTS Smoother) For offline processing, the Rauch-Tung-Striebel (RTS) smoother provides optimal state estimates using all available data. **Backward recursion:** $$\hat{\mathbf{x}}_{t|T} = \hat{\mathbf{x}}_{t|t} + \mathbf{C}_t (\hat{\mathbf{x}}_{t+1|T} - \hat{\mathbf{x}}_{t+1|t})$$ $$\mathbf{P}_{t|T} = \mathbf{P}_{t|t} + \mathbf{C}_t (\mathbf{P}_{t+1|T} - \mathbf{P}_{t+1|t}) \mathbf{C}_t^T$$ where the smoother gain is: $$\mathbf{C}_t = \mathbf{P}_{t|t} \mathbf{F}_{t+1}^T \mathbf{P}_{t+1|t}^{-1}$$ ### Extended Kalman Filter (EKF) For nonlinear systems, the EKF linearizes around the current state estimate. **Linearization:** $$\mathbf{F}_t = \left. \frac{\partial f}{\partial \mathbf{x}} \right|_{\hat{\mathbf{x}}_{t-1|t-1}}, \quad \mathbf{H}_t = \left. \frac{\partial h}{\partial \mathbf{x}} \right|_{\hat{\mathbf{x}}_{t|t-1}}$$ Then apply standard Kalman filter equations with linearized matrices. **Limitations:** * Only accurate for mildly nonlinear systems * Can diverge if nonlinearity is strong * Requires computation of Jacobians ### Unscented Kalman Filter (UKF) The UKF uses the unscented transform to propagate mean and covariance through nonlinear functions. **Key idea:** * Select sigma points that capture the mean and covariance of the state distribution * Propagate sigma points through nonlinear function * Compute statistics of transformed points **Advantages:** * No Jacobian computation needed * Better accuracy than EKF for highly nonlinear systems * Similar computational cost to EKF ### Particle Filter (Sequential Monte Carlo) For highly nonlinear and/or non-Gaussian systems, particle filters approximate distributions with samples. **Algorithm:** 1. **Initialize**: Draw $N$ particles $\{\mathbf{x}_0^{(i)}\}_{i=1}^N$ from $p(\mathbf{x}_0)$ 2. **For each time step** $t$: * **Predict**: Propagate particles through state equation $$\mathbf{x}_t^{(i)} \sim p(\mathbf{x}_t | \mathbf{x}_{t-1}^{(i)})$$ * **Update**: Compute importance weights $$w_t^{(i)} \propto w_{t-1}^{(i)} \cdot p(\mathbf{y}_t | \mathbf{x}_t^{(i)})$$ * **Normalize**: $\tilde{w}_t^{(i)} = w_t^{(i)} / \sum_{j=1}^N w_t^{(j)}$ * **Resample**: Draw new particles according to weights (if effective sample size is low) **State estimate:** $$\hat{\mathbf{x}}_t = \sum_{i=1}^N \tilde{w}_t^{(i)} \mathbf{x}_t^{(i)}$$ **Challenges:** * Particle degeneracy (most weights become negligible) * High computational cost for high-dimensional states * Requires many particles for accurate approximation ### Parameter Estimation: Maximum Likelihood via EM When model parameters $\boldsymbol{\theta} = \{\mathbf{F}, \mathbf{H}, \mathbf{Q}, \mathbf{R}\}$ are unknown, we use Expectation-Maximization. **E-Step**: Run Kalman smoother to compute $p(\mathbf{x}_{1:T} | \mathbf{y}_{1:T}, \boldsymbol{\theta}^{(k)})$ **M-Step**: Maximize expected complete-data log-likelihood: $$\boldsymbol{\theta}^{(k+1)} = \arg\max_{\boldsymbol{\theta}} \mathbb{E}_{\mathbf{x}_{1:T}} [\log p(\mathbf{y}_{1:T}, \mathbf{x}_{1:T} | \boldsymbol{\theta})]$$ **Closed-form M-step solutions** exist for linear Gaussian SSMs: $$\mathbf{Q}^{(k+1)} = \frac{1}{T} \sum_{t=1}^T \mathbb{E}[(\mathbf{x}_t - \mathbf{F}\mathbf{x}_{t-1})(\mathbf{x}_t - \mathbf{F}\mathbf{x}_{t-1})^T]$$ $$\mathbf{R}^{(k+1)} = \frac{1}{T} \sum_{t=1}^T \mathbb{E}[(\mathbf{y}_t - \mathbf{H}\mathbf{x}_t)(\mathbf{y}_t - \mathbf{H}\mathbf{x}_t)^T]$$ ## Modern Deep Learning Connections ### Recurrent Neural Networks as State Space Models RNNs can be viewed as nonlinear state space models: **State equation:** $$\mathbf{h}_t = \tanh(\mathbf{W}_{hh} \mathbf{h}_{t-1} + \mathbf{W}_{xh} \mathbf{x}_t + \mathbf{b}_h)$$ **Observation equation:** $$\mathbf{y}_t = \mathbf{W}_{hy} \mathbf{h}_t + \mathbf{b}_y$$ This perspective: * Connects classical signal processing to deep learning * Enables hybrid models with interpretable structure * Suggests principled initialization strategies ### Structured State Space Models (S4) Recent breakthrough architecture for efficient long-range sequence modeling. **Continuous-time formulation:** $$\frac{d\mathbf{x}(t)}{dt} = \mathbf{A}\mathbf{x}(t) + \mathbf{B}u(t)$$ $$y(t) = \mathbf{C}\mathbf{x}(t) + \mathbf{D}u(t)$$ **Discretization** (with step size $\Delta$): $$\mathbf{\bar{A}} = (\mathbf{I} - \Delta/2 \cdot \mathbf{A})^{-1}(\mathbf{I} + \Delta/2 \cdot \mathbf{A})$$ $$\mathbf{\bar{B}} = (\mathbf{I} - \Delta/2 \cdot \mathbf{A})^{-1} \Delta \mathbf{B}$$ **Key innovations:** * **HiPPO initialization**: Specially designed matrix $\mathbf{A}$ that memorizes history efficiently * **Structured matrices**: Diagonal plus low-rank structure for efficient computation * **Convolutional view**: Can be computed as convolution with learned kernel * **Linear time complexity**: $O(L)$ for sequence length $L$ (vs $O(L^2)$ for attention) **Performance:** * Matches or exceeds Transformers on long-range tasks * Much more efficient for very long sequences (10K+ tokens) * Better extrapolation to longer sequences than seen during training ### Mamba Architecture Evolution of S4 with selective state updates. **Selective SSM:** $$ \begin{aligned} \mathbf{B}_t &= s_B(\mathbf{x}_t) \\ \mathbf{C}_t &= s_C(\mathbf{x}_t) \\ \Delta_t &= \tau_{\Delta}(\text{Parameter} + s_{\Delta}(\mathbf{x}_t)) \end{aligned} $$ where $s_B, s_C, s_{\Delta}$ are input-dependent functions. **Key features:** * **Input-dependent selection**: Different inputs get different dynamics * **Hardware-aware design**: Optimized for GPU memory hierarchy * **Linear scaling**: Maintains $O(L)$ complexity with selective mechanism * **State-of-the-art performance**: Competitive with large Transformers **Applications:** * Language modeling * Long document understanding * Time series forecasting * DNA sequence modeling ### Differentiable Kalman Filters Combining Kalman filtering with neural networks: **Approach 1: Neural Transition/Observation Functions** $$\mathbf{x}_t = f_{\theta}(\mathbf{x}_{t-1}, \mathbf{u}_t) + \mathbf{w}_t$$ $$\mathbf{y}_t = h_{\phi}(\mathbf{x}_t) + \mathbf{v}_t$$ where $f_{\theta}$ and $h_{\phi}$ are neural networks. **Approach 2: Neural Kalman Gain** Learn the Kalman gain directly: $$\mathbf{K}_t = g_{\psi}(\mathbf{y}_t, \hat{\mathbf{x}}_{t|t-1}, \text{context})$$ **Training:** * End-to-end via backpropagation through time * Loss functions: prediction error, negative log-likelihood, KL divergence * Maintains interpretability of state space structure **Benefits:** * Flexibility of neural networks * Uncertainty quantification from Kalman filter * Data efficiency from inductive bias ## Applications ### 1. Economics and Finance **Macroeconomic Forecasting:** * GDP growth decomposition (trend, cycle, seasonal) * Inflation modeling with latent price pressures * Unobserved components models **Example: Dynamic Factor Model for GDP** $$ \begin{aligned} y_{it} &= \lambda_i f_t + \epsilon_{it} \quad \text{(multiple indicators)} \\ f_t &= \phi f_{t-1} + \eta_t \quad \text{(latent factor)} \end{aligned} $$ **Financial Applications:** * Volatility estimation (stochastic volatility models) * Yield curve dynamics (affine term structure models) * Portfolio optimization with regime switching ### 2. Engineering and Control **Tracking Systems:** * Target tracking with radar/sonar * GPS navigation and sensor fusion * Robot localization (SLAM) **Example: Constant Velocity Model** $$ \mathbf{x}_t = \begin{bmatrix} p_t \\ v_t \end{bmatrix} = \begin{bmatrix} 1 & \Delta t \\ 0 & 1 \end{bmatrix} \begin{bmatrix} p_{t-1} \\ v_{t-1} \end{bmatrix} + \mathbf{w}_t $$ where $p_t$ is position and $v_t$ is velocity. **Control Applications:** * Linear Quadratic Gaussian (LQG) control * Model predictive control (MPC) * Adaptive control systems ### 3. Neuroscience **Neural Decoding:** * Estimating hand position from neural spike trains * Brain-machine interfaces * Population dynamics analysis **State Space Model:** $$ \begin{aligned} \mathbf{x}_t &= \mathbf{A}\mathbf{x}_{t-1} + \mathbf{w}_t \quad \text{(latent neural state)} \\ \mathbf{n}_t &\sim \text{Poisson}(\exp(\mathbf{C}\mathbf{x}_t)) \quad \text{(spike counts)} \end{aligned} $$ ### 4. Epidemiology **Disease Dynamics:** * SIR/SEIR models with unobserved compartments * Nowcasting with reporting delays * Intervention effect estimation **Example: SEIR Model** $$ \begin{aligned} S_t &= S_{t-1} - \beta S_{t-1} I_{t-1} / N \\ E_t &= E_{t-1} + \beta S_{t-1} I_{t-1} / N - \sigma E_{t-1} \\ I_t &= I_{t-1} + \sigma E_{t-1} - \gamma I_{t-1} \\ R_t &= R_{t-1} + \gamma I_{t-1} \end{aligned} $$ with observations of reported cases (subset of true infections). ### 5. Climate Science **Temperature Reconstruction:** * Combining proxy data (tree rings, ice cores) with instrumental records * Missing data handling * Long-term trend estimation **Dynamic Linear Model:** $$ \begin{aligned} T_t &= T_{t-1} + \alpha \cdot \text{CO2}_t + w_t \quad \text{(temperature evolution)} \\ y_{it} &= \beta_i T_t + v_{it} \quad \text{(multiple proxies)} \end{aligned} $$ ### 6. Speech Processing **Speech Recognition:** * Hidden Markov Models (HMMs) are discrete state space models * Acoustic feature extraction with temporal dynamics * Noise-robust recognition **Modern approach:** * Deep learning for feature extraction * State space models for temporal dynamics * Hybrid architectures ## Advantages and Challenges ### Advantages of State Space Models #### 1. Unified Framework * Single mathematical formulation encompasses many models * Common algorithms (Kalman filter, EM) apply broadly * Facilitates model comparison and selection #### 2. Uncertainty Quantification * Full posterior distributions over states: $p(\mathbf{x}_t | \mathbf{y}_{1:t})$ * Prediction intervals: $p(\mathbf{y}_{t+h} | \mathbf{y}_{1:t})$ for horizon $h$ * Parameter uncertainty via Bayesian inference #### 3. Missing Data and Irregularity * Natural handling of missing observations * Irregular sampling intervals (non-uniform time steps) * Ragged arrays and unbalanced panels **Example**: If $y_t$ is missing, skip the update step: $$\hat{\mathbf{x}}_{t|t} = \hat{\mathbf{x}}_{t|t-1}, \quad \mathbf{P}_{t|t} = \mathbf{P}_{t|t-1}$$ #### 4. Multivariate Modeling * Joint modeling of multiple related time series * Cross-series dependencies and spillovers * Dimension reduction through low-dimensional states #### 5. Interpretability * Explicit separation: system dynamics vs. measurement process * States often have clear meanings (level, trend, cycle) * Parameters have physical/economic interpretations #### 6. Forecasting * Multi-step ahead predictions via iterating state equation * Uncertainty grows naturally with forecast horizon * Scenario analysis through control inputs #### 7. Causal Structure * Can incorporate known constraints (e.g., physical laws) * Identifiable causal effects in some cases * Counterfactual analysis possible ### Challenges and Limitations #### 1. Computational Complexity **Kalman Filter:** * Time: $O(Tn^3)$ for $T$ time steps, state dimension $n$ * Space: $O(n^2)$ for covariance matrices * Becomes prohibitive for very high dimensions **Particle Filter:** * Exponential scaling with state dimension (curse of dimensionality) * Requires $N \propto \exp(n)$ particles for good approximation * Parallelization helps but doesn't eliminate the problem #### 2. Model Specification **Challenges:** * Choosing state dimension $n$ is non-trivial * Specifying $\mathbf{F}, \mathbf{H}$ structure requires domain knowledge * Overfitting risk with too many parameters * Underfitting risk with oversimplified dynamics **Approaches:** * Cross-validation for model selection * Information criteria (AIC, BIC) * Bayesian model averaging * Sparse priors on parameters #### 3. Identifiability **Fundamental issue:** Multiple state space representations can produce identical observations. **Example:** For any invertible matrix $\mathbf{T}$: $$ \begin{aligned} \tilde{\mathbf{x}}_t &= \mathbf{T} \mathbf{x}_t \\ \tilde{\mathbf{F}} &= \mathbf{T} \mathbf{F} \mathbf{T}^{-1} \\ \tilde{\mathbf{H}} &= \mathbf{H} \mathbf{T}^{-1} \end{aligned} $$ yields the same observations. **Implications:** * Parameter estimates may be non-unique * Need constraints for identification (e.g., diagonal $\mathbf{Q}$) * Interpretation of states can be ambiguous #### 4. Nonlinearity **Extended/Unscented Kalman Filters:** * Only work well for mildly nonlinear systems * Can diverge for strong nonlinearity * No optimality guarantees **Particle Filters:** * Curse of dimensionality * Sensitive to proposal distribution choice * Require careful tuning **Neural State Space Models:** * Loss of interpretability * Require large datasets * Black-box dynamics #### 5. Parameter Estimation **Convergence issues:** * EM may converge slowly * Multiple local optima * Sensitive to initialization **Computational cost:** * Each EM iteration requires full Kalman filter/smoother pass * For $T$ observations and $n$-dimensional state: $O(Tn^3)$ per iteration * Many iterations often needed **Alternatives:** * Markov Chain Monte Carlo (MCMC) for Bayesian inference * Variational inference for approximate posteriors * Gradient-based optimization with automatic differentiation #### 6. Model Misspecification **Robustness concerns:** * Kalman filter optimal only under correct specification * Misspecified dynamics or noise can lead to poor performance * Difficult to diagnose misspecification **Approaches:** * Robust filtering (e.g., Huber loss) * Model diagnostics (innovation analysis) * Adaptive filtering #### 7. Real-time Constraints **Latency issues:** * Smoothing requires all data (offline processing) * Filtering is online but may still have latency * Hardware constraints in embedded systems **Solutions:** * Fixed-lag smoothing (compromise between filter and smoother) * Approximate methods for speed * Hardware acceleration (GPUs, FPGAs) ## Modern Trends and Future Directions ### 1. Deep State Space Models **Hybrid Approaches:** * Neural networks for $f(\cdot), h(\cdot)$ * Keep probabilistic structure for uncertainty * End-to-end differentiable training **Examples:** * Deep Kalman Filters * Variational RNNs * Neural ODEs with state space structure ### 2. Structured State Space Models (S4/Mamba) **Recent advances:** * Efficient long-range modeling * Competitive with Transformers * Better scaling properties **Applications expanding:** * Language modeling * Time series forecasting * Multi-modal learning ### 3. Causal Inference **State space models for causality:** * Synthetic control methods * Interrupted time series analysis * Dynamic treatment effects **Advantages:** * Temporal structure aids identification * Counterfactual predictions * Heterogeneous effects over time ### 4. High-Dimensional Problems **Approaches:** * Sparse state space models * Low-rank approximations * Hierarchical models * Ensemble Kalman filters ### 5. Online Learning **Adaptive models:** * Time-varying parameters * Regime switching * Concept drift handling **Methods:** * Bayesian online change-point detection * Adaptive forgetting factors * Sequential model selection ### 6. Integration with Other ML Paradigms **Combinations:** * State space + attention mechanisms * State space + graph neural networks * State space + reinforcement learning **Benefits:** * Best of both worlds * Structured inductive biases * Data efficiency ## Summary State space models provide a powerful, flexible, and principled framework for modeling dynamic systems and time series data. Their key strengths include: * **Mathematical elegance**: Unified treatment of diverse models * **Principled inference**: Optimal filtering and smoothing algorithms * **Uncertainty quantification**: Full posterior distributions * **Practical flexibility**: Handle missing data, irregularity, multivariate series * **Modern relevance**: Connections to deep learning (S4, Mamba) Despite computational and specification challenges, state space models remain essential tools at the intersection of classical statistics, signal processing, control theory, and modern machine learning. As deep learning continues to evolve, the integration of state space structure with neural networks promises even more powerful and interpretable models for sequential data across domains from finance to neuroscience to natural language processing.
Alternative to Transformers using structured state representations.
Fixed quantization parameters.
Model parameter distributions.
Embed patterns in token distribution.
Biologically-inspired learning rule.
Apply external forces to explore transitions.
Models perpetuating stereotypes.
STL decomposes time series using locally weighted regression separating seasonal trend and remainder components robustly.
Add noise to differential equation models.
Update with one example at a time.
Stochastic volatility models treat variance as latent stochastic process capturing volatility clustering and heavy tails.
Stock-outs occur when demand exceeds available inventory disrupting production or sales.
Stop sequences terminate generation when encountered in output.
Stochastic Temporal Optimal control using Recurrent Networks combines control theory with stochastic RNNs.
Straight fin heat sinks use parallel plates optimized for unidirectional airflow.
Vertical leads.
Straight-through estimators approximate gradients through non-differentiable quantization operations.
Straight-through Gumbel-Softmax enables gradient-based learning with discrete variables.
Apply mechanical strain to increase carrier mobility.
Silicon with stress to improve mobility.
Strategic sourcing aligns procurement with business objectives optimizing cost quality and risk.
Strategy adaptation modifies approaches based on success feedback.
Streaming generation outputs tokens incrementally as generated improving perceived latency.
Process infinite sequences.
Model thermal stress effects.
Relate Raman shift to stress.
Attend to every k-th position.
# Structural Time Series Models ## STS Structural time series (STS) models, also called **state space models** or **unobserved components models**, decompose a time series into interpretable components—each representing a distinct source of variation. ## 1. Core Components A structural time series model decomposes an observed series $y_t$ into additive components: $$ y_t = \mu_t + \gamma_t + \psi_t + X_t\beta + \varepsilon_t $$ Where: - $\mu_t$ — Trend component - $\gamma_t$ — Seasonal component - $\psi_t$ — Cyclical component - $X_t\beta$ — Regression/explanatory effects - $\varepsilon_t$ — Irregular (white noise) component ## 2. Component Specifications ### 2.1 Trend Component ($\mu_t$) The trend captures the underlying level and growth pattern of the series. #### Local Level Model (Random Walk) $$ \mu_t = \mu_{t-1} + \eta_t, \quad \eta_t \sim N(0, \sigma_\eta^2) $$ - Level evolves as a random walk - No slope/growth rate component - Suitable for series without systematic growth #### Local Linear Trend Model $$ \begin{aligned} \mu_t &= \mu_{t-1} + \nu_{t-1} + \eta_t, \quad \eta_t \sim N(0, \sigma_\eta^2) \\ \nu_t &= \nu_{t-1} + \zeta_t, \quad \zeta_t \sim N(0, \sigma_\zeta^2) \end{aligned} $$ - $\mu_t$ — Stochastic level - $\nu_t$ — Stochastic slope (growth rate) - Both level and slope evolve over time - When $\sigma_\zeta^2 = 0$: slope is fixed (deterministic growth) - When $\sigma_\eta^2 = 0$: smooth trend (integrated random walk) #### Smooth Trend (Integrated Random Walk) $$ \begin{aligned} \mu_t &= \mu_{t-1} + \nu_{t-1} \\ \nu_t &= \nu_{t-1} + \zeta_t, \quad \zeta_t \sim N(0, \sigma_\zeta^2) \end{aligned} $$ - Level changes are smooth (no level disturbance) - Only slope receives stochastic shocks #### Deterministic Trend $$ \mu_t = \alpha + \beta t $$ - Fixed intercept $\alpha$ and slope $\beta$ - No stochastic evolution ### 2.2 Seasonal Component ($\gamma_t$) Captures recurring patterns at fixed intervals. #### Dummy Variable Form $$ \gamma_t = -\sum_{j=1}^{s-1} \gamma_{t-j} + \omega_t, \quad \omega_t \sim N(0, \sigma_\omega^2) $$ - $s$ — Number of seasons (e.g., $s=12$ for monthly data) - Seasonal effects sum to zero over a complete cycle - When $\sigma_\omega^2 = 0$: deterministic (fixed) seasonality #### Trigonometric/Fourier Form $$ \gamma_t = \sum_{j=1}^{[s/2]} \gamma_{j,t} $$ Each harmonic $j$ follows: $$ \begin{bmatrix} \gamma_{j,t} \\ \gamma_{j,t}^* \end{bmatrix} = \begin{bmatrix} \cos \lambda_j & \sin \lambda_j \\ -\sin \lambda_j & \cos \lambda_j \end{bmatrix} \begin{bmatrix} \gamma_{j,t-1} \\ \gamma_{j,t-1}^* \end{bmatrix} + \begin{bmatrix} \omega_{j,t} \\ \omega_{j,t}^* \end{bmatrix} $$ Where: - $\lambda_j = \frac{2\pi j}{s}$ — Frequency of harmonic $j$ - $\omega_{j,t}, \omega_{j,t}^* \sim N(0, \sigma_\omega^2)$ - Allows different variances for different harmonics - More parsimonious when few harmonics are needed ### 2.3 Cyclical Component ($\psi_t$) Captures medium-term fluctuations not tied to fixed calendar periods. $$ \begin{bmatrix} \psi_t \\ \psi_t^* \end{bmatrix} = \rho \begin{bmatrix} \cos \lambda_c & \sin \lambda_c \\ -\sin \lambda_c & \cos \lambda_c \end{bmatrix} \begin{bmatrix} \psi_{t-1} \\ \psi_{t-1}^* \end{bmatrix} + \begin{bmatrix} \kappa_t \\ \kappa_t^* \end{bmatrix} $$ Where: - $\lambda_c \in (0, \pi)$ — Cycle frequency - $\rho \in (0, 1)$ — Damping factor (ensures stationarity) - $\kappa_t, \kappa_t^* \sim N(0, \sigma_\kappa^2)$ - Period of cycle: $\frac{2\pi}{\lambda_c}$ time units ### 2.4 Regression Component ($X_t\beta$) Incorporates explanatory variables: $$ \text{Regression effect} = \sum_{k=1}^{K} \beta_k x_{k,t} $$ Common applications: - **Intervention effects**: Step functions, pulse dummies, ramp effects - **Calendar effects**: Trading days, holidays, leap years - **Explanatory variables**: Economic indicators, weather, etc. #### Time-Varying Coefficients (Optional) $$ \beta_t = \beta_{t-1} + \xi_t, \quad \xi_t \sim N(0, \sigma_\xi^2) $$ ### 2.5 Irregular Component ($\varepsilon_t$) $$ \varepsilon_t \sim N(0, \sigma_\varepsilon^2) $$ - White noise (serially uncorrelated) - Captures measurement error and short-term fluctuations - Also called "observation noise" ## 3. State Space Representation ### 3.1 General Form Any structural time series model can be written in state space form: **Observation Equation:** $$ y_t = Z_t \alpha_t + \varepsilon_t, \quad \varepsilon_t \sim N(0, H_t) $$ **State Equation:** $$ \alpha_{t+1} = T_t \alpha_t + R_t \eta_t, \quad \eta_t \sim N(0, Q_t) $$ Where: - $y_t$ — Observed data (scalar or vector) - $\alpha_t$ — State vector (unobserved components) - $Z_t$ — Observation matrix (links states to observations) - $T_t$ — Transition matrix (governs state evolution) - $R_t$ — Selection matrix - $H_t$ — Observation noise variance - $Q_t$ — State noise covariance matrix ### 3.2 Example: Local Linear Trend + Seasonal State vector: $$ \alpha_t = \begin{bmatrix} \mu_t \\ \nu_t \\ \gamma_t \\ \gamma_{t-1} \\ \vdots \\ \gamma_{t-s+2} \end{bmatrix} $$ ## 4. Estimation via Kalman Filter ### 4.1 Kalman Filter Recursions **Prediction Step:** $$ \begin{aligned} \alpha_{t|t-1} &= T_t \alpha_{t-1|t-1} \\ P_{t|t-1} &= T_t P_{t-1|t-1} T_t' + R_t Q_t R_t' \end{aligned} $$ **Update Step:** $$ \begin{aligned} v_t &= y_t - Z_t \alpha_{t|t-1} \quad \text{(prediction error)} \\ F_t &= Z_t P_{t|t-1} Z_t' + H_t \quad \text{(prediction error variance)} \\ K_t &= P_{t|t-1} Z_t' F_t^{-1} \quad \text{(Kalman gain)} \\ \alpha_{t|t} &= \alpha_{t|t-1} + K_t v_t \\ P_{t|t} &= (I - K_t Z_t) P_{t|t-1} \end{aligned} $$ Where: - $\alpha_{t|t-1}$ — Predicted state (prior) - $\alpha_{t|t}$ — Filtered state (posterior) - $P_{t|t-1}$ — Predicted state covariance - $P_{t|t}$ — Filtered state covariance ### 4.2 Kalman Smoother Refines estimates using full sample (backward pass): $$ \begin{aligned} \alpha_{t|n} &= \alpha_{t|t} + P_{t|t} T_{t+1}' P_{t+1|t}^{-1} (\alpha_{t+1|n} - \alpha_{t+1|t}) \\ P_{t|n} &= P_{t|t} + P_{t|t} T_{t+1}' P_{t+1|t}^{-1} (P_{t+1|n} - P_{t+1|t}) P_{t+1|t}^{-1} T_{t+1} P_{t|t} \end{aligned} $$ Where $n$ is the total number of observations. ## 5. Hyperparameter Estimation ### 5.1 Maximum Likelihood The log-likelihood is computed via prediction error decomposition: $$ \log L(\theta) = -\frac{n}{2} \log(2\pi) - \frac{1}{2} \sum_{t=1}^{n} \left( \log |F_t| + v_t' F_t^{-1} v_t \right) $$ Where: - $\theta$ — Vector of hyperparameters (variance terms) - $v_t$ — Prediction errors from Kalman filter - $F_t$ — Prediction error variances Optimization methods: - Quasi-Newton (BFGS, L-BFGS) - EM algorithm - Scoring algorithms ### 5.2 Bayesian Estimation $$ p(\theta | y_{1:n}) \propto p(y_{1:n} | \theta) \cdot p(\theta) $$ Common approaches: - **MCMC**: Gibbs sampling, Hamiltonian Monte Carlo - **Variational inference**: Faster approximation - **Integrated nested Laplace approximation (INLA)** Common priors: - Inverse-gamma for variance parameters - Half-Cauchy or half-normal for scale parameters ## 6. Model Selection and Diagnostics ### 6.1 Information Criteria $$ \begin{aligned} \text{AIC} &= -2 \log L + 2k \\ \text{BIC} &= -2 \log L + k \log n \\ \text{AICc} &= \text{AIC} + \frac{2k(k+1)}{n-k-1} \end{aligned} $$ Where $k$ is the number of hyperparameters. ### 6.2 Diagnostic Checks Standardized prediction errors should be: - **Zero mean**: $E[v_t / \sqrt{F_t}] = 0$ - **Unit variance**: $\text{Var}[v_t / \sqrt{F_t}] = 1$ - **Serially uncorrelated**: Check with Ljung-Box test - **Normally distributed**: Check with Jarque-Bera test ### 6.3 Auxiliary Residuals - **Observation residuals**: Detect outliers - **State residuals**: Detect structural breaks $$ \begin{aligned} e_t &= \frac{y_t - Z_t \alpha_{t|n}}{\sqrt{\text{Var}(y_t - Z_t \alpha_{t|n})}} \\ r_t &= \frac{\eta_t}{\sqrt{\text{Var}(\eta_t)}} \end{aligned} $$ ## 7. Comparison | Approach | Philosophy | Strengths | Limitations | |:---------|:-----------|:----------|:------------| | **ARIMA** | Reduced-form; models stationary transformations | Parsimonious, well-understood | Components not interpretable | | **Exponential Smoothing** | Weighted averages with decay | Simple, effective | Less flexible seasonality | | **Structural TS** | Explicit component decomposition | Interpretable, handles missing data | More parameters | | **Prophet** | Additive trend + seasonality + holidays | User-friendly | Less rigorous uncertainty | | **Deep Learning** | Learn patterns from data | Powerful with big data | Black box, data hungry | ## 8. Topics ### 8.1 Handling Missing Data The Kalman filter naturally handles missing observations: - When $y_t$ is missing, skip the update step - Prediction step proceeds normally - Smoother propagates information through gaps ### 8.2 Multivariate Extensions For vector $y_t \in \mathbb{R}^p$: $$ y_t = Z_t \alpha_t + \varepsilon_t, \quad \varepsilon_t \sim N(0, H_t) $$ Applications: - Common trends across multiple series - Factor models - Dynamic factor analysis ### 8.3 Non-Gaussian Extensions - **Student-t errors**: Heavy tails, robust to outliers - **Mixture models**: Regime switching - **Non-linear state space**: Extended Kalman filter, particle filters ## 9. Software Implementations ### R Packages ```r KFAS - Kalman Filter and Smoother library(KFAS) model <- SSModel(y ~ SSMtrend(2, Q = list(NA, NA)) + SSMseasonal(12, Q = NA), H = NA) fit <- fitSSM(model, inits = rep(0, 4)) bsts - Bayesian Structural Time Series library(bsts) ss <- AddLocalLinearTrend(list(), y) ss <- AddSeasonal(ss, y, nseasons = 12) model <- bsts(y, state.specification = ss, niter = 1000) dlm - Dynamic Linear Models library(dlm) build <- function(theta) { dlmModPoly(2, dV = exp(theta[1]), dW = exp(theta[2:3])) + dlmModSeas(12, dV = 0, dW = exp(theta[4])) } fit <- dlmMLE(y, parm = rep(0, 4), build = build) ``` ### Python ```python statsmodels from statsmodels.tsa.statespace.structural import UnobservedComponents model = UnobservedComponents( y, level='local linear trend', seasonal=12, stochastic_seasonal=True ) results = model.fit() TensorFlow Probability import tensorflow_probability as tfp trend = tfp.sts.LocalLinearTrend(observed_time_series=y) seasonal = tfp.sts.Seasonal(num_seasons=12, observed_time_series=y) model = tfp.sts.Sum([trend, seasonal], observed_time_series=y) ``` ## 11. Structural time series models Structural time series models provide: - **Interpretability**: Each component has clear economic/statistical meaning - **Flexibility**: Add/remove components based on domain knowledge - **Robustness**: Natural handling of missing data and irregular spacing - **Uncertainty quantification**: Full probability distributions for components and forecasts - **Intervention analysis**: Easy incorporation of known breaks and policy changes The state space framework unifies estimation, filtering, smoothing, and forecasting within a coherent probabilistic structure, making structural time series models a powerful tool for understanding and predicting temporal phenomena.
Predefined sparsity patterns.
Structured output constrains generation to follow specified formats like JSON or schemas.
Structured pruning removes entire channels layers or blocks enabling hardware-efficient acceleration.
Remove entire channels layers or attention heads.
Student is smaller model learning from larger teacher. Teacher provides richer signal than hard labels.
Match style statistics.
Combine styles from different images.
Style mixing combines latent codes at different scales creating hybrid generations.
Match style of reference image.