Reinforcement learning (RL) is a machine learning paradigm in which an agent learns to make decisions by interacting with an environment, receiving rewards or penalties for its actions, and adjusting its behavior to maximize cumulative reward over time. Unlike supervised learning, which requires labeled input-output pairs, RL discovers optimal strategies through trial and error — the agent explores actions, observes their consequences, and gradually develops a policy that maps states to actions. This framework has produced some of AI's most dramatic achievements: AlphaGo defeating the world Go champion, robotic hands solving Rubik's cubes, autonomous drones navigating obstacle courses, and — most consequentially for modern AI — reinforcement learning from human feedback (RLHF) aligning large language models like ChatGPT, Claude, and Gemini to follow human instructions safely and helpfully.

The core RL framework consists of an agent, an environment, states, actions, rewards, and a policy. At each time step, the agent observes the current state of the environment, selects an action according to its policy, transitions to a new state, and receives a reward signal. The agent's goal is to learn a policy that maximizes the expected cumulative discounted reward, called the return. The discount factor (gamma, typically 0.95-0.99) balances immediate versus future rewards — a gamma near 1 values long-term consequences heavily, while a lower gamma makes the agent myopic. The value function V(s) estimates the expected return from a given state under the current policy, while the action-value function Q(s,a) estimates the expected return from taking a specific action in a specific state. The Bellman equation relates these values recursively:

This equation is the foundation of value-based methods: the optimal Q-function satisfies this recursion, and once known, the optimal policy simply selects the action with the highest Q-value in each state.

Value-based methods learn to estimate the value of states or state-action pairs, then derive a policy from those estimates. Q-learning (Watkins, 1989) maintains a table of Q-values for every state-action pair and updates them using the Bellman equation after each transition. Deep Q-Networks (DQN) replaced the Q-table with a neural network, enabling RL to handle high-dimensional state spaces like raw pixel inputs from Atari games. DQN introduced experience replay (storing transitions in a buffer and sampling mini-batches for training) and target networks (a slowly-updated copy of the Q-network used to compute stable targets), both of which stabilize training. Double DQN reduced overestimation bias by using two networks to decouple action selection from value estimation. Dueling DQN separated the network into value and advantage streams, improving learning efficiency for states where the choice of action matters less than being in the right state. Rainbow combined six extensions to DQN — double Q-learning, prioritized replay, dueling architecture, multi-step returns, distributional RL, and noisy exploration — achieving superhuman performance on many Atari games.

Policy gradient methods directly optimize the policy by estimating the gradient of expected reward with respect to policy parameters. Instead of learning value functions and deriving a policy, these methods parameterize the policy as a neural network and adjust its weights to increase the probability of actions that led to high rewards. The REINFORCE algorithm computes the gradient using sampled trajectories, but suffers from high variance. Actor-critic methods reduce this variance by combining a policy network (actor) with a value network (critic) — the critic estimates how good the current state is, and the actor uses this baseline to compute lower-variance gradient estimates. Advantage Actor-Critic (A2C) and its asynchronous variant (A3C) parallelize data collection across multiple environment instances. Proximal Policy Optimization (PPO) constrains policy updates to a trust region using a clipped surrogate objective, preventing destructively large updates that can destabilize training. PPO has become the most widely used RL algorithm due to its simplicity, stability, and strong empirical performance — it is the algorithm used in RLHF for ChatGPT and many other aligned language models.

Reinforcement learning from human feedback (RLHF) is the technique that transformed large language models from next-token predictors into helpful, harmless assistants. The process has three stages. First, a supervised fine-tuning (SFT) stage trains the model on human-written demonstrations of desired behavior. Second, a reward model is trained on human preference data: annotators compare pairs of model outputs and indicate which they prefer, and a neural network learns to predict these preferences. Third, the language model is fine-tuned using PPO to maximize the reward model's score, with a KL-divergence penalty that prevents the model from diverging too far from the SFT baseline. This penalty is critical — without it, the model would exploit weaknesses in the reward model rather than genuinely improving. Constitutional AI (CAI) and direct preference optimization (DPO) are alternatives that modify or eliminate the explicit reward model: DPO reformulates the RLHF objective so the policy can be optimized directly from preference data without training a separate reward model, simplifying the pipeline while achieving comparable alignment quality.

<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,sans-serif">
  <rect x="0" y="0" width="760" height="470" fill="#0d1117"/>
  <text x="380" y="28" fill="#e6edf3" font-size="21" font-weight="700" text-anchor="middle">Reinforcement Learning — Learn by Interaction</text>
  <text x="380" y="48" fill="#8b98a5" font-size="12" text-anchor="middle">agent observes state, takes action, receives reward — maximize cumulative reward through trial and error</text>

  <!-- === TOP: RL loop === -->
  <rect x="25" y="62" width="710" height="135" rx="6" fill="#080d14" stroke="#233043" stroke-width="1.2"/>
  <text x="380" y="82" fill="#e6edf3" font-size="11" text-anchor="middle" font-weight="600">The RL Interaction Loop</text>

  <!-- Agent -->
  <rect x="120" y="110" width="130" height="55" rx="8" fill="#1a1520" stroke="#a78bfa" stroke-width="1.2"/>
  <text x="185" y="135" fill="#c4b5fd" font-size="11" text-anchor="middle" font-weight="600">Agent</text>
  <text x="185" y="150" fill="#8b98a5" font-size="8" text-anchor="middle">policy π(a|s)</text>

  <!-- Environment -->
  <rect x="450" y="110" width="130" height="55" rx="8" fill="#14261f" stroke="#34d399" stroke-width="1.2"/>
  <text x="515" y="135" fill="#6ee7b7" font-size="11" text-anchor="middle" font-weight="600">Environment</text>
  <text x="515" y="150" fill="#8b98a5" font-size="8" text-anchor="middle">dynamics P(s'|s,a)</text>

  <!-- Action arrow (agent → env) -->
  <path d="M253,125 L447,125" fill="none" stroke="#60a5fa" stroke-width="1.2"/>
  <polygon points="445,122 451,125 445,128" fill="#60a5fa"/>
  <text x="350" y="118" fill="#93c5fd" font-size="9" text-anchor="middle">action a_t</text>

  <!-- State + Reward arrow (env → agent) -->
  <path d="M447,152 L253,152" fill="none" stroke="#34d399" stroke-width="1.2"/>
  <polygon points="255,149 249,152 255,155" fill="#34d399"/>
  <text x="350" y="172" fill="#6ee7b7" font-size="9" text-anchor="middle">state s_{t+1}, reward r_t</text>

  <!-- Goal annotation -->
  <text x="350" y="190" fill="#fbbf24" font-size="9" text-anchor="middle">Goal: maximize E[Σ γ^t · r_t] (discounted cumulative reward)</text>

  <!-- === MIDDLE LEFT: Algorithms === -->
  <rect x="25" y="205" width="350" height="128" rx="6" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="200" y="223" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Algorithm Families</text>

  <text x="45" y="245" fill="#60a5fa" font-size="8.5" font-weight="600">Value-based (Q-learning)</text>
  <text x="45" y="261" fill="#8b98a5" font-size="8.5">learn Q(s,a), act greedily. DQN, Rainbow</text>

  <text x="45" y="281" fill="#34d399" font-size="8.5" font-weight="600">Policy gradient (REINFORCE)</text>
  <text x="45" y="297" fill="#8b98a5" font-size="8.5">directly optimize π. PPO, A2C, TRPO</text>

  <text x="45" y="317" fill="#fbbf24" font-size="8.5" font-weight="600">Actor-Critic (hybrid)</text>
  <text x="45" y="333" fill="#8b98a5" font-size="8.5">policy + value function. SAC, TD3, PPO</text>

  <!-- === MIDDLE RIGHT: Key concepts === -->
  <rect x="390" y="205" width="345" height="128" rx="6" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="562" y="223" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Key Concepts</text>

  <text x="410" y="245" fill="#8b98a5" font-size="8.5" font-weight="600">Exploration vs exploitation:</text>
  <text x="410" y="261" fill="#8b98a5" font-size="8.5">try new things vs use what works (ε-greedy)</text>
  <text x="410" y="281" fill="#8b98a5" font-size="8.5" font-weight="600">Discount factor γ:</text>
  <text x="410" y="297" fill="#8b98a5" font-size="8.5">0.99 = long-term, 0.9 = short-term focus</text>
  <text x="410" y="317" fill="#8b98a5" font-size="8.5" font-weight="600">Reward shaping:</text>
  <text x="410" y="333" fill="#8b98a5" font-size="8.5">design reward to guide learning (critical!)</text>

  <!-- === BOTTOM: Applications === -->
  <rect x="25" y="343" width="710" height="60" rx="5" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="380" y="361" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">RL Applications</text>

  <text x="90" y="383" fill="#c4b5fd" font-size="9" text-anchor="middle" font-weight="600">RLHF</text>
  <text x="90" y="397" fill="#8b98a5" font-size="8" text-anchor="middle">align LLMs to prefs</text>

  <text x="215" y="383" fill="#fbbf24" font-size="9" text-anchor="middle" font-weight="600">Games</text>
  <text x="215" y="397" fill="#8b98a5" font-size="8" text-anchor="middle">AlphaGo, Atari, DOTA</text>

  <text x="340" y="383" fill="#34d399" font-size="9" text-anchor="middle" font-weight="600">Robotics</text>
  <text x="340" y="397" fill="#8b98a5" font-size="8" text-anchor="middle">manipulation, locomotion</text>

  <text x="465" y="383" fill="#60a5fa" font-size="9" text-anchor="middle" font-weight="600">Chip design</text>
  <text x="465" y="397" fill="#8b98a5" font-size="8" text-anchor="middle">placement (AlphaChip)</text>

  <text x="590" y="383" fill="#f87171" font-size="9" text-anchor="middle" font-weight="600">Trading</text>
  <text x="590" y="397" fill="#8b98a5" font-size="8" text-anchor="middle">portfolio optimization</text>

  <text x="700" y="383" fill="#8b98a5" font-size="9" text-anchor="middle" font-weight="600">Search</text>
  <text x="700" y="397" fill="#8b98a5" font-size="8" text-anchor="middle">o1 reasoning (MCTS)</text>

  <!-- Key insight -->
  <rect x="25" y="411" width="710" height="22" rx="3" fill="#0b1220" stroke="#233043" stroke-width="0.8"/>
  <text x="380" y="426" fill="#fbbf24" font-size="9" text-anchor="middle">RL learns what supervised learning cannot: optimal behavior from interaction, without labeled examples of the right answer.</text>

  <text x="380" y="460" fill="#6b7684" font-size="11" text-anchor="middle">RL is how AI systems learn to act: from playing Go to aligning LLMs, reward drives behavior.</text>
</svg>

RL has achieved landmark results across games, robotics, and industrial applications. AlphaGo (2016) combined Monte Carlo tree search with deep RL, defeating world champion Lee Sedol at Go — a game with 10^170 possible board positions that was considered decades away from AI mastery. AlphaZero (2017) learned Go, chess, and shogi entirely through self-play without human expert data, starting from random play and surpassing all previous engines within hours. In robotics, RL trains controllers for locomotion, manipulation, and dexterous tasks that are difficult to program manually — OpenAI's robotic hand solved a Rubik's cube using sim-to-real transfer, where a policy trained in simulation was deployed on physical hardware. Google used RL for chip placement in their TPU design pipeline, demonstrating that RL agents could produce floor plans competitive with weeks of expert human effort in hours. Autonomous driving, recommendation systems, resource scheduling, and energy grid management all use RL variants in production.

The exploration-exploitation dilemma is the fundamental challenge in RL. The agent must balance exploiting known high-reward actions with exploring unknown actions that might yield even higher rewards. Pure exploitation converges to suboptimal policies because the agent never discovers better strategies; pure exploration wastes time on bad actions. Epsilon-greedy exploration takes a random action with probability epsilon and the best-known action otherwise, but this is inefficient in large state spaces. Entropy regularization (used in SAC) adds a bonus for stochastic policies, encouraging the agent to maintain diverse behavior. Curiosity-driven exploration rewards the agent for visiting novel states, measured by prediction error of a learned world model. Upper confidence bound (UCB) methods balance exploration and exploitation mathematically by choosing actions with high estimated value or high uncertainty. In practice, the choice of exploration strategy often determines whether RL succeeds or fails on a given problem.