offline rl

**Offline RL** (Batch RL) is **reinforcement learning from a fixed dataset of previously collected interactions** — learning a policy entirely from logged data without any additional environment interaction, enabling RL in domains where online exploration is costly, dangerous, or impossible. **Offline RL Challenges** - **Distribution Shift**: The learned policy may visit state-action pairs not in the dataset — Q-values for unseen actions are unreliable. - **Overestimation**: Standard Q-learning maximizes over poorly estimated out-of-distribution actions — catastrophic overestimation. - **Conservative Methods**: CQL, IQL, TD3+BC constrain the policy to stay near the data — pessimistic value estimation. - **Dataset Quality**: Performance is bounded by the quality and coverage of the offline dataset. **Why It Matters** - **Safety**: No online exploration needed — critical for autonomous driving, healthcare, semiconductor process control. - **Data Reuse**: Leverage existing logged data (process logs, historical experiments) — no new experiments needed. - **Semiconductor**: Train control policies from historical process data without risking production equipment. **Offline RL** is **learning from logs, not from life** — training RL policies entirely from fixed datasets without environment interaction.