maximum entropy rl

**Maximum Entropy RL** is a **reinforcement learning framework that augments the standard reward with an entropy bonus** — the agent maximizes the expected reward PLUS the entropy of its policy, encouraging exploration and leading to more robust, multi-modal policies. **MaxEnt RL Objective** - **Objective**: $pi^* = argmax_pi sum_t mathbb{E}[r_t + alpha H(pi(cdot|s_t))]$ — reward + entropy. - **Temperature ($alpha$)**: Controls the trade-off between reward maximization and entropy maximization. - **Optimal Policy**: $pi^*(a|s) propto exp(Q^*(s,a) / alpha)$ — the Boltzmann (softmax) policy. - **Soft Bellman**: $V(s) = alpha log sum_a exp(Q(s,a)/alpha)$ — the soft value function. **Why It Matters** - **Exploration**: High entropy prevents the policy from collapsing to a single action — maintains exploration. - **Robustness**: MaxEnt policies are more robust to perturbations — they maintain multiple viable strategies. - **Foundation**: The theoretical foundation for SAC (Soft Actor-Critic), one of the most successful continuous control algorithms. **MaxEnt RL** is **rewarding uncertainty** — encouraging the agent to maintain diverse, exploratory behavior while maximizing reward.