her replay

**HER** (Hindsight Experience Replay) is a **technique for learning from failure in goal-conditioned RL** — when the agent fails to reach the intended goal, HER relabels the experience with the actually achieved state as the goal, creating a successful learning signal from every trajectory. **How HER Works** - **Original**: Agent tries to reach goal $g$, ends up at state $s'$ ≠ $g$ — failed trajectory, negative reward. - **Relabeling**: Create a new experience with goal $g' = s'$ — the same trajectory now "succeeded" at reaching $s'$. - **Learning**: The agent learns to reach many states, even though it failed at the original goal. - **Strategies**: Relabel with final state, random future state, or closest achieved state. **Why It Matters** - **Sparse Rewards**: In goal-conditioned tasks with sparse rewards (only at goal), standard RL gets almost no learning signal — HER solves this. - **Sample Efficiency**: Every failed trajectory becomes useful — dramatically improves sample efficiency. - **Robotics**: HER was crucial for robotic manipulation — reaching, pushing, and grasping with sparse rewards. **HER** is **learning from every failure** — relabeling failed goals with achieved states to extract learning from every trajectory.