elastic weight consolidation (ewc)

Elastic Weight Consolidation (EWC) prevents catastrophic forgetting in continual learning by adding regularization that protects weights important to previous tasks, estimated through Fisher information. Problem: neural networks trained sequentially on tasks forget earlier tasks as weights are overwritten—catastrophic interference. Key insight: not all weights are equally important for each task; protect important weights while allowing unimportant ones to adapt. Fisher information: F_i = E[(∂logP(D|θ)/∂θ_i)²] measures parameter importance—high Fisher means small weight change causes large output change. EWC loss: L = L_new(θ) + λ × Σ_i F_i × (θ_i - θ_old_i)², penalizing deviation from old weights proportionally to importance. Implementation: after training task A, compute Fisher matrix for each parameter, then add EWC regularization when training task B. Online EWC: accumulate Fisher estimates across tasks rather than storing per-task—more scalable. Comparison: rehearsal (replay old data—memory cost), EWC (regularization—no data storage), and progressive networks (add new modules—architecture growth). Limitations: Fisher diagonal approximation ignores parameter interactions, plastic weights for all tasks become scarce over many tasks. Extensions: Synaptic Intelligence (online importance), PackNet (prune and freeze), and Memory Aware Synapses. Foundational approach for continual learning enabling sequential task learning while preserving earlier knowledge.