batch rl

**Batch RL** is the **original term for offline reinforcement learning** — learning a policy from a fixed batch of previously collected transition data $(s, a, r, s')$ without any further interaction with the environment. **Batch RL Methods** - **Fitted Q-Iteration (FQI)**: Iteratively fit the Q-function on the batch using supervised regression. - **LSPI**: Least-Squares Policy Iteration — combine least-squares temporal difference with policy improvement. - **BCQ**: Batch-Constrained Q-learning — only consider actions similar to those in the batch. - **BEAR**: Bootstrapping Error Accumulation Reduction — constrain the policy's action distribution to match the data. **Why It Matters** - **No Simulation Needed**: Learn from real logged data — no simulator required. - **Extrapolation Error**: The key challenge — the policy must not exploit Q-value errors for unseen state-action pairs. - **History**: Batch RL predates the "offline RL" terminology — foundational work by Ernst et al., Lange et al. **Batch RL** is **the original offline RL** — learning optimal policies from fixed datasets of previously collected transitions.