hat (hard attention to task)

**HAT (Hard Attention to the Task)** is a continual learning method that uses **learnable binary masks** to protect task-specific weights in a neural network, preventing catastrophic forgetting while allowing parameter sharing between tasks when beneficial. **How HAT Works** - **Attention Masks**: For each task, HAT learns a set of **attention embeddings** that produce near-binary gate values (0 or 1) for each unit in each layer. - **Mask Training**: During forward passes, each unit's output is multiplied by its task-specific gate. Gates near 0 mean the unit is **not used** for this task; gates near 1 mean it is **actively used**. - **Gradient Masking**: During backpropagation for task t, gradients are **blocked** for units that have high attention values for any previous task. This prevents updating weights that are important for old tasks. - **Annealing**: The attention values are initially soft (sigmoid-like) during training, then progressively sharpened toward binary values through temperature annealing. **Key Properties** - **Selective Protection**: Only the units that are actually important for a previous task are protected — units unused by old tasks are fully available for new learning. - **Potential Sharing**: If a unit is useful for both an old and new task, it can be shared (gated on for both tasks). - **No Buffer Required**: HAT doesn't store any examples from previous tasks — protection is entirely through gradient masking. - **Task-Conditioned**: At inference time, the model applies the mask for the relevant task, activating the appropriate subnetwork. **Advantages** - **Near-Zero Forgetting**: Very low forgetting due to hard gradient masking on important units. - **Better Capacity Utilization**: More flexible than PackNet — units can be shared between tasks rather than exclusively allocated. - **No Replay**: No memory buffer or generative model needed. **Limitations** - **Task ID Required**: Must know which task is active to select the correct mask. - **Capacity Saturation**: Eventually most units are important for some task, limiting room for new learning. - **Optimization Complexity**: The attention annealing process adds hyperparameters (temperature schedule) that need tuning. HAT represents a **sophisticated middle ground** between rigid weight allocation (PackNet) and soft regularization (EWC) — offering strong forgetting prevention with more efficient parameter sharing.