PackNet is a continual learning method that uses iterative pruning to allocate separate subnetworks within a single neural network for each task. Instead of growing the network (like progressive networks), PackNet reuses freed capacity from pruning to learn new tasks while protecting important weights for old tasks.
How PackNet Works
- Task 1: Train the full network on task 1. Then prune the network — identify and remove the least important weights (e.g., those with smallest magnitude). This frees up a significant portion of the network capacity.
- Task 1 Freeze: Mark the remaining (unpruned) task 1 weights as frozen — they will never be modified again.
- Task 2: Train only the freed (pruned) weights on task 2. The frozen task 1 weights participate in forward passes but don't receive gradient updates. After training, prune task 2 weights similarly.
- Repeat: Each new task uses the remaining free capacity. The network accumulates binary task masks indicating which weights belong to which task.
Key Properties
- Fixed Network Size: Unlike progressive networks, the model does not grow. All tasks share the same network, just using different subsets of weights.
- Zero Forgetting: Previous task weights are frozen, guaranteeing no catastrophic forgetting.
- Task Masks: Each task has a binary mask indicating its active weights. At inference time, the appropriate mask is applied.
- Capacity Limit: Eventually the network runs out of free weights. The number of tasks is limited by the pruning ratio and network size.
Typical Pruning Ratios
- 50–75% pruning per task is common — meaning each task uses only 25–50% of available weights.
- A network pruned at 75% can theoretically support ~4 tasks (though later tasks have less capacity).
Advantages Over Progressive Networks
- Constant model size — no linear growth.
- Efficient parameter usage — leverages the well-known observation that neural networks are over-parameterized and can achieve good performance with far fewer weights.
Limitations
- Finite Capacity: Cannot support unlimited tasks — the network eventually runs out of free parameters.
- No Forward Transfer: Tasks don't share weights (beyond the architectural structure), limiting knowledge transfer between tasks.
- Task ID Required: Must know which task mask to apply at inference time.
PackNet demonstrated that the over-parameterization of modern neural networks could be directly exploited for continual learning — a key insight for the field.