Continual/Incremental Learning is the ability of a neural network to sequentially learn new tasks or data distributions without forgetting previously acquired knowledge — addressing the catastrophic forgetting phenomenon where training on new data overwrites the weights responsible for earlier task performance, a fundamental challenge for deploying lifelong learning systems that must adapt to evolving environments.

Catastrophic Forgetting Mechanisms:

• Weight Overwriting: Gradient updates for the new task modify weights critical for previous tasks, degrading stored representations
• Representation Drift: Internal feature representations shift to accommodate new data distributions, invalidating the learned decision boundaries for earlier tasks
• Activation Overlap: When neurons shared across tasks are repurposed, the network loses the capacity to generate task-specific activation patterns
• Loss Landscape Perspective: The optimal weights for the new task lie in a different basin of the loss landscape than the previous task's optimum, and standard SGD navigates directly to the new basin

Regularization-Based Methods:

• Elastic Weight Consolidation (EWC): Add a quadratic penalty preventing important weights (measured by the diagonal of the Fisher information matrix) from deviating far from their values after previous tasks; importance weights are computed per-task and accumulated
• Synaptic Intelligence (SI): Track the contribution of each parameter to the loss decrease during training, using this online importance measure as the regularization strength — avoids the need for separate Fisher computation
• Memory Aware Synapses (MAS): Estimate weight importance based on the sensitivity of the learned function's output to weight perturbations, computed in an unsupervised manner
• PackNet: Iteratively prune and freeze weights for each task, allocating dedicated subsets of the network to each task without interference
• Progressive Neural Networks: Add new columns of network capacity for each task while freezing previous columns and allowing lateral connections — eliminates forgetting at the cost of linear parameter growth

Replay-Based Methods:

• Experience Replay: Store a small buffer of examples from previous tasks and interleave them with current task data during training to maintain performance on old distributions
• Generative Replay: Train a generative model (VAE or GAN) that synthesizes pseudo-examples from previous tasks, replacing the need for a stored memory buffer
• Dark Experience Replay (DER): Store and replay not just input-output pairs but also the model's logits (soft predictions), providing richer supervision for knowledge retention
• Gradient Episodic Memory (GEM): Constrain gradient updates to not increase the loss on stored episodic memories from previous tasks, formulated as a constrained optimization problem
• A-GEM (Averaged GEM): Efficient approximation of GEM that projects gradients onto the average gradient direction from episodic memory rather than solving a quadratic program per step

Architecture-Based Methods:

• Dynamic Expandable Networks (DEN): Automatically expand network capacity when new tasks cannot be adequately learned within existing parameters
• Expert Gate: Route inputs to task-specific expert networks using a learned gating mechanism, isolating task-specific parameters
• Modular Networks: Compose task-specific solutions from a shared pool of reusable modules, with task-specific routing or selection mechanisms
• Hypernetworks for CL: Use a hypernetwork to generate task-specific weight matrices conditioned on a task embedding, enabling distinct parameterizations without storing separate networks

Evaluation Protocols:

• Task-Incremental Learning (Task-IL): Task identity is provided at test time; the model only needs to discriminate within the current task's classes
• Class-Incremental Learning (Class-IL): Task identity is unknown at test time; the model must discriminate among all classes seen so far — significantly harder than Task-IL
• Domain-Incremental Learning (Domain-IL): The task structure is the same but input distribution shifts (e.g., different visual domains), requiring adaptation without forgetting
• Metrics: Average accuracy across all tasks after learning the final task, forward transfer (benefit to new tasks from prior knowledge), backward transfer (impact on old tasks after learning new ones), and forgetting measure (maximum accuracy minus final accuracy per task)

Practical Considerations:

• Memory Budget: Replay methods require choosing buffer size (typically 200–5,000 examples) and selection strategy (reservoir sampling, herding, or loss-based selection)
• Computational Overhead: EWC and SI add modest overhead for importance computation; replay methods add proportional cost for buffer rehearsal
• Scalability: Most continual learning methods are evaluated on relatively small benchmarks (Split CIFAR, Split ImageNet); scaling to production environments with hundreds of tasks remains challenging
• Pretrained Models: Starting from a strong pretrained foundation model substantially reduces forgetting, as the representations are more generalizable and require less modification for new tasks

Continual learning remains a critical frontier in making deep learning systems truly adaptive — where the tension between plasticity (ability to learn new information) and stability (retention of old knowledge) must be carefully balanced through complementary regularization, replay, and architectural strategies to enable lifelong deployment in dynamic real-world environments.