Maximum Entropy RL is a reinforcement learning framework that augments the standard reward with an entropy bonus — the agent maximizes the expected reward PLUS the entropy of its policy, encouraging exploration and leading to more robust, multi-modal policies.

MaxEnt RL Objective

• Objective: $pi^* = argmax_pi sum_t mathbb{E}[r_t + alpha H(pi(cdot|s_t))]$ — reward + entropy.
• Temperature ($alpha$): Controls the trade-off between reward maximization and entropy maximization.
• Optimal Policy: $pi^(a|s) propto exp(Q^(s,a) / alpha)$ — the Boltzmann (softmax) policy.
• Soft Bellman: $V(s) = alpha log sum_a exp(Q(s,a)/alpha)$ — the soft value function.

Why It Matters

• Exploration: High entropy prevents the policy from collapsing to a single action — maintains exploration.
• Robustness: MaxEnt policies are more robust to perturbations — they maintain multiple viable strategies.
• Foundation: The theoretical foundation for SAC (Soft Actor-Critic), one of the most successful continuous control algorithms.

MaxEnt RL is rewarding uncertainty — encouraging the agent to maintain diverse, exploratory behavior while maximizing reward.