kolmogorov arnold network kan

**Kolmogorov-Arnold Networks (KANs): Learnable Activation Functions — replacing fixed activations with spline-based univariate functions** Kolmogorov-Arnold Networks (KANs, Liu et al., 2024) challenge the MLP paradigm by making activation functions learnable rather than fixed. KAN networks achieve competitive or superior accuracy while offering interpretability advantages. **Kolmogorov-Arnold Theorem and KAN Formulation** Kolmogorov-Arnold theorem: multivariate continuous functions decompose as: f(x_1,...,x_n) = Σ_{q=0}^{2n} Φ_q(Σ_{p=1}^n φ_{q,p}(x_p)) (superposition of univariate functions). KAN leverages this: each weight is learnable univariate function (not scalar). Spline basis (B-splines): φ_{q,p}(x) = Σ_j c_j B_j(x) (linear combination of B-spline bases). Grid size (number of spline nodes) controls expressivity; larger grids fit finer details. **Architecture and Comparison to MLP** MLP: x → W1 (linear) → ReLU (fixed) → W2 (linear) → ReLU → output. KAN: x → [learnable univariate function for each input × output combination] → [learn univariate function for next layer] → output. Layering: KAN(l_1, l_2, ..., l_n) specifies layer widths; each edge is univariate function. Weight count similar to MLP for same layer widths. **Interpretability and Symbolic Regression** Fixed MLPs are black boxes; interpreting learned functions is difficult. KANs: learnable activation functions can be plotted and visualized, revealing input-output relationships. Interactive visualization: identify important features, discover underlying equations via symbolic regression. Example: learning Fourier series, trigonometric identities, physical laws (Newton's laws) automatically from data—with human-interpretable symbolic expressions. **Accuracy and Efficiency** KAN-2 (Zheng et al., 2024): achieves competitive accuracy on standard benchmarks (MNIST, CIFAR-10, ImageNet) compared to ResNets, sometimes with fewer parameters. Vision KAN extends to images via spatial decomposition. Advantage: interpretable architectures without accuracy sacrifice. Disadvantage: training slower than MLPs (spline evaluation more expensive than ReLU). Scaling to very large models (billions of parameters) remains unclear. **Limitations and Extensions** KANs fit smaller networks well (< 100M parameters); scaling properties unexplored for foundation models. Spline choice (cubic, B-spline degree) impacts expressivity-complexity tradeoff. Initialization and hyperparameter sensitivity differs from MLPs. Recent work: KAN-MLP hybrids (learnable activations in some layers only), applications to physics-informed learning, potential for symbolic AI integration.