Experience replay is a technique from reinforcement learning — adopted for continual learning — where the model randomly samples and replays stored examples from previous experiences during training on new data. It prevents catastrophic forgetting by continuously refreshing the model on old knowledge.

How Experience Replay Works

• Store: As the model processes data from each task or time period, save a subset of examples to a replay buffer (also called experience buffer or memory bank).
• Sample: When training on new data, randomly sample a mini-batch from the replay buffer.
• Combine: Mix the replayed sample with the current training batch. The model updates on both old and new data simultaneously.
• Update Buffer: Optionally add new examples to the buffer and evict old ones using a replacement strategy.

Origins in Reinforcement Learning

• Originally proposed for DQN (Deep Q-Networks) by DeepMind to stabilize RL training. The agent stores (state, action, reward, next_state) transitions and samples from them during learning.
• In RL, replay breaks the correlation between consecutive experiences, improving training stability and sample efficiency.

Experience Replay for Continual Learning

• In continual learning, replay serves a different purpose — it prevents forgetting by ensuring old task data remains in the training distribution.
• Balanced Sampling: Sample equal numbers of examples from each previous task to maintain balanced performance.
• Prioritized Replay: Prioritize replaying examples where the model's performance has degraded most — focusing rehearsal where it's most needed.
• Dark Experience Replay (DER): Store not just the input and label but also the model's logits (soft predictions) at storage time. During replay, use these logits as an additional knowledge distillation target.

Practical Considerations

• Buffer Size: Typically 500–5,000 examples total. Even small buffers are surprisingly effective.
• Replay Frequency: Common approach is to replay one buffer batch for every new data batch (1:1 ratio).
• Storage: For text, storing examples is cheap. For images or embeddings, storage costs are higher.

Experience replay is the simplest and most robust approach to continual learning — it's the baseline that every more sophisticated method must beat.