Progressive neural networks are a continual learning architecture that handles new tasks by adding new neural network columns (lateral connections included) while freezing all previously learned columns. This completely eliminates catastrophic forgetting because old weights are never modified.

How Progressive Networks Work

• Task 1: Train a standard neural network on the first task. Freeze all its weights.
• Task 2: Add a new network column for task 2. This new column receives lateral connections from the frozen task 1 column, allowing it to reuse task 1 features without modifying them.
• Task N: Add another column with lateral connections from all previous columns. The new column can leverage features from all prior tasks.

Architecture

• Each task has its own dedicated column (set of layers) with independent weights.
• Lateral connections allow new columns to receive intermediate features from all previous columns as additional inputs.
• Previous columns are completely frozen — their weights never change after initial training.

Advantages

• Zero Forgetting: Previous task performance is perfectly preserved because old weights are never updated.
• Forward Transfer: New tasks can leverage features learned from previous tasks through lateral connections.
• No Replay Needed: No memory buffer or replay mechanism required.

Disadvantages

• Linear Growth: Model size grows linearly with the number of tasks — each new task adds an entire network column. After 100 tasks, the model is 100× its original size.
• No Backward Transfer: Old columns don't improve when new tasks provide useful information — only forward transfer is possible.
• Compute Cost: Inference requires running all columns (for determining the task) or knowing which task is active.
• Scalability: Impractical for scenarios with many tasks or when the number of tasks is unknown in advance.

Where It Works Best

• Few-task scenarios (2–10 tasks) where model growth is manageable.
• Applications where zero forgetting is an absolute requirement.
• Transfer learning experiments studying how features transfer between tasks.

Progressive neural networks provided a foundational proof of concept for architectural approaches to continual learning, though their growth problem limits practical adoption.