Kolmogorov-Arnold Networks (KANs): Learnable Activation Functions — replacing fixed activations with spline-based univariate functions

Kolmogorov-Arnold Networks (KANs, Liu et al., 2024) challenge the MLP paradigm by making activation functions learnable rather than fixed. KAN networks achieve competitive or superior accuracy while offering interpretability advantages.

Kolmogorov-Arnold Theorem and KAN Formulation

Kolmogorov-Arnold theorem: multivariate continuous functions decompose as: f(x_1,...,x_n) = Σ_{q=0}^{2n} Φ_q(Σ_{p=1}^n φ_{q,p}(x_p)) (superposition of univariate functions). KAN leverages this: each weight is learnable univariate function (not scalar). Spline basis (B-splines): φ_{q,p}(x) = Σ_j c_j B_j(x) (linear combination of B-spline bases). Grid size (number of spline nodes) controls expressivity; larger grids fit finer details.

Architecture and Comparison to MLP

MLP: x → W1 (linear) → ReLU (fixed) → W2 (linear) → ReLU → output. KAN: x → [learnable univariate function for each input × output combination] → [learn univariate function for next layer] → output. Layering: KAN(l_1, l_2, ..., l_n) specifies layer widths; each edge is univariate function. Weight count similar to MLP for same layer widths.

Interpretability and Symbolic Regression

Fixed MLPs are black boxes; interpreting learned functions is difficult. KANs: learnable activation functions can be plotted and visualized, revealing input-output relationships. Interactive visualization: identify important features, discover underlying equations via symbolic regression. Example: learning Fourier series, trigonometric identities, physical laws (Newton's laws) automatically from data—with human-interpretable symbolic expressions.

Accuracy and Efficiency

KAN-2 (Zheng et al., 2024): achieves competitive accuracy on standard benchmarks (MNIST, CIFAR-10, ImageNet) compared to ResNets, sometimes with fewer parameters. Vision KAN extends to images via spatial decomposition. Advantage: interpretable architectures without accuracy sacrifice. Disadvantage: training slower than MLPs (spline evaluation more expensive than ReLU). Scaling to very large models (billions of parameters) remains unclear.

Limitations and Extensions

KANs fit smaller networks well (< 100M parameters); scaling properties unexplored for foundation models. Spline choice (cubic, B-spline degree) impacts expressivity-complexity tradeoff. Initialization and hyperparameter sensitivity differs from MLPs. Recent work: KAN-MLP hybrids (learnable activations in some layers only), applications to physics-informed learning, potential for symbolic AI integration.