progressive neural networks

**Progressive neural networks** are a continual learning architecture that handles new tasks by **adding new neural network columns** (lateral connections included) while **freezing all previously learned columns**. This completely eliminates catastrophic forgetting because old weights are never modified. **How Progressive Networks Work** - **Task 1**: Train a standard neural network on the first task. Freeze all its weights. - **Task 2**: Add a new network column for task 2. This new column receives **lateral connections** from the frozen task 1 column, allowing it to reuse task 1 features without modifying them. - **Task N**: Add another column with lateral connections from all previous columns. The new column can leverage features from all prior tasks. **Architecture** - Each task has its own **dedicated column** (set of layers) with independent weights. - **Lateral connections** allow new columns to receive intermediate features from all previous columns as additional inputs. - Previous columns are **completely frozen** — their weights never change after initial training. **Advantages** - **Zero Forgetting**: Previous task performance is perfectly preserved because old weights are never updated. - **Forward Transfer**: New tasks can leverage features learned from previous tasks through lateral connections. - **No Replay Needed**: No memory buffer or replay mechanism required. **Disadvantages** - **Linear Growth**: Model size grows linearly with the number of tasks — each new task adds an entire network column. After 100 tasks, the model is 100× its original size. - **No Backward Transfer**: Old columns don't improve when new tasks provide useful information — only forward transfer is possible. - **Compute Cost**: Inference requires running all columns (for determining the task) or knowing which task is active. - **Scalability**: Impractical for scenarios with many tasks or when the number of tasks is unknown in advance. **Where It Works Best** - Few-task scenarios (2–10 tasks) where model growth is manageable. - Applications where **zero forgetting** is an absolute requirement. - Transfer learning experiments studying how features transfer between tasks. Progressive neural networks provided a **foundational proof of concept** for architectural approaches to continual learning, though their growth problem limits practical adoption.