experience replay

**Experience replay** is a technique from reinforcement learning — adopted for continual learning — where the model **randomly samples and replays stored examples** from previous experiences during training on new data. It prevents catastrophic forgetting by continuously refreshing the model on old knowledge. **How Experience Replay Works** - **Store**: As the model processes data from each task or time period, save a subset of examples to a **replay buffer** (also called experience buffer or memory bank). - **Sample**: When training on new data, randomly sample a mini-batch from the replay buffer. - **Combine**: Mix the replayed sample with the current training batch. The model updates on both old and new data simultaneously. - **Update Buffer**: Optionally add new examples to the buffer and evict old ones using a replacement strategy. **Origins in Reinforcement Learning** - Originally proposed for **DQN (Deep Q-Networks)** by DeepMind to stabilize RL training. The agent stores (state, action, reward, next_state) transitions and samples from them during learning. - In RL, replay breaks the correlation between consecutive experiences, improving training stability and sample efficiency. **Experience Replay for Continual Learning** - In continual learning, replay serves a different purpose — it **prevents forgetting** by ensuring old task data remains in the training distribution. - **Balanced Sampling**: Sample equal numbers of examples from each previous task to maintain balanced performance. - **Prioritized Replay**: Prioritize replaying examples where the model's performance has degraded most — focusing rehearsal where it's most needed. - **Dark Experience Replay (DER)**: Store not just the input and label but also the model's **logits** (soft predictions) at storage time. During replay, use these logits as an additional knowledge distillation target. **Practical Considerations** - **Buffer Size**: Typically 500–5,000 examples total. Even small buffers are surprisingly effective. - **Replay Frequency**: Common approach is to replay one buffer batch for every new data batch (1:1 ratio). - **Storage**: For text, storing examples is cheap. For images or embeddings, storage costs are higher. Experience replay is the **simplest and most robust** approach to continual learning — it's the baseline that every more sophisticated method must beat.