multi-agent reinforcement learning game-theoretic learning
# Multi-Agent Reinforcement Learning & Game-Theoretic Learning
## Introduction & Motivation
Multi-agent reinforcement learning (MARL) extends the single-agent RL framework to settings where multiple learners act simultaneously in a shared environment, each influencing the outcomes and rewards of the others. This shift breaks the fundamental stationarity assumption that underlies most single-agent RL theory: from the perspective of any one agent, the environment's transition dynamics appear to shift over time as the other agents update their policies. MARL sits at the intersection of reinforcement learning, game theory, and multi-agent systems, and it underlies a wide range of practical applications including autonomous vehicle fleets negotiating traffic, algorithmic trading agents competing in markets, multiplayer game AI (e.g., OpenAI Five, AlphaStar), robotic swarms coordinating on warehouse logistics, and negotiation or auction mechanisms. The core research questions in MARL revolve around three regimes: fully cooperative settings where agents share a common reward and must learn to coordinate; fully competitive (zero-sum) settings where one agent's gain is another's loss; and mixed or general-sum settings where agents have partially aligned and partially conflicting incentives, requiring reasoning about equilibria rather than a single optimal policy. Motivation for studying MARL formally, rather than simply running independent single-agent learners side by side, comes from empirical and theoretical evidence that naive approaches suffer from non-stationarity, credit assignment ambiguity in cooperative teams, and convergence failures when equilibrium concepts other than a single optimum are the correct solution target.
## Core Concepts & Theory
The foundational formalism for MARL is the Markov game (also called a stochastic game), which generalizes the Markov Decision Process (MDP) to N agents. Each agent i observes a (possibly partial) state, selects an action from its own action space, and receives an individual reward that depends on the joint action of all agents, not just its own. Because the transition function and reward functions depend on the joint action vector, an agent's environment appears non-stationary whenever other agents are also learning and changing their policies. Central solution concepts borrowed from game theory include the Nash equilibrium, a joint policy where no agent can improve its expected return by unilaterally deviating; the correlated equilibrium, which allows for shared randomization signals that can achieve better social outcomes than independent randomization; and Pareto optimality, describing joint policies where no agent can be made better off without making another worse off. Key algorithmic paradigms include independent learning (each agent runs a single-agent algorithm, ignoring the presence of others, often called Independent Q-Learning or IQL), centralized training with decentralized execution (CTDE), where agents have access to global information during training but must act on local observations at deployment, and fully centralized control, where a single controller selects the joint action but which is often intractable at scale due to the exponential growth of the joint action space. Self-play, in which an agent trains against copies or past versions of itself, is a critical technique for competitive and cooperative-competitive settings, having driven landmark results in Go, poker, Dota 2, and StarCraft II.
## Mathematical Formulation
A Markov game is formally defined as a tuple (N, S, \{A_i\}_{i=1}^N, T, \{R_i\}_{i=1}^N, \gamma), where N is the number of agents, S is the state space, A_i is the action space of agent i, T: S imes A_1 imes \cdots imes A_N imes S o [0,1] is the joint transition function, R_i: S imes A_1 imes \cdots imes A_N o \mathbb{R} is agent i's reward function, and \gamma is the discount factor. Each agent seeks to maximize its own expected discounted return:
$$ J_i(\pi_1, \ldots, \pi_N) = \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^t R_i(s_t, a_1^t, \ldots, a_N^t) ight] $$
A joint policy (\pi_1^*, \ldots, \pi_N^*) is a Nash equilibrium if, for every agent i and every alternative policy \pi_i':
$$ J_i(\pi_1^*, \ldots, \pi_i^*, \ldots, \pi_N^*) \geq J_i(\pi_1^*, \ldots, \pi_i', \ldots, \pi_N^*) $$
For independent Q-learning, each agent maintains its own action-value function Q_i(s, a_i) and applies the standard Q-learning update while treating all other agents as part of the environment:
$$ Q_i(s, a_i) \leftarrow Q_i(s, a_i) + \alpha \left[r_i + \gamma \max_{a_i'} Q_i(s', a_i') - Q_i(s, a_i) ight] $$
In the centralized-critic actor-critic framework (as used in MADDPG), each agent's critic is conditioned on the joint observation-action space, while its actor conditions only on local information:
$$ abla_{ heta_i} J(\pi_i) = \mathbb{E}\left[ abla_{ heta_i} \pi_i(a_i \mid o_i) abla_{a_i} Q_i^{\pi}(x, a_1, \ldots, a_N) \big|_{a_i = \pi_i(o_i)} ight] $$
where x = (o_1, \ldots, o_N) denotes the concatenation of all agents' observations, allowing the critic Q_i^\pi to account for the effects of every agent's action during training even though execution remains decentralized.
## Advanced Theory & Extensions
Beyond the basic Markov game formalism, several extensions address specific structural challenges. Partially observable Markov games (POMGs, sometimes called Dec-POMDPs in the fully cooperative case) restrict each agent to a local observation function rather than full state access, which is the realistic setting for most physical multi-agent systems and necessitates recurrent policies or explicit belief-state tracking. Mean-field MARL approximates interactions in very-large-population settings by replacing pairwise interactions between all agents with an interaction between each agent and the average (mean-field) behavior of the population, reducing computational complexity from exponential in N to linear. Opponent modeling techniques equip agents with an explicit or implicit model of other agents' policies or intentions, enabling more sophisticated strategic reasoning than treating opponents as a black-box part of the environment; LOLA (Learning with Opponent-Learning Awareness) explicitly differentiates through one step of the opponent's anticipated policy update. Communication-augmented MARL (e.g., CommNet, TarMAC, Differentiable Inter-Agent Learning) allows agents to learn discrete or continuous communication protocols end-to-end via backpropagation through a differentiable communication channel, which is particularly valuable in cooperative settings with partial observability. Population-based training and league play, as used in AlphaStar, maintain a diverse population of agents (including exploiters and past checkpoints) to avoid the strategy collapse and cyclic non-convergence that can occur with naive self-play against only the most recent policy.
## Computational Considerations
MARL training is substantially more computationally demanding than single-agent RL due to several compounding factors. The joint action space grows exponentially with the number of agents when using centralized control, making tabular or exhaustive-search methods infeasible beyond a handful of agents. Centralized-critic methods such as MADDPG require the critic network's input dimensionality to scale with N, increasing memory and compute costs for the critic while keeping actor networks lightweight. Simulation throughput often becomes the binding constraint: environments must be stepped forward with all agents' actions applied simultaneously, and many MARL research platforms (e.g., PettingZoo, SMAC, Melting Pot, MuJoCo multi-agent variants) support vectorized or parallel environment execution to amortize simulator overhead across thousands of concurrent episodes. Self-play and league-based training additionally require maintaining a pool of historical policy checkpoints for opponents, which multiplies both storage and rollout compute since inference must be run for each opponent snapshot deployed in parallel matches. Replay buffer design also becomes agent-aware in off-policy MARL, since transitions sampled from old policy versions may no longer accurately represent the current joint policy's induced dynamics, requiring either fresh on-policy rollouts or importance-weighting corrections.
## Practical Implementation Strategies
A pragmatic entry point into MARL is to first validate an independent-learning baseline (each agent runs PPO or DQN treating others as environment noise) before introducing coordination machinery, since independent learning is simple to implement and surprisingly competitive in many cooperative tasks with well-shaped individual rewards. For genuinely cooperative tasks, adopting the CTDE paradigm via frameworks like MAPPO (Multi-Agent PPO with a centralized value function) or QMIX (which factors a joint action-value function into a monotonic combination of per-agent utilities) typically yields more stable and higher-performing policies than fully independent or fully centralized alternatives. Reward shaping requires particular care in MARL: sparse team rewards create severe credit-assignment problems, so difference rewards (comparing an agent's actual contribution against a counterfactual baseline where it takes a default action) or potential-based shaping is often used to disentangle individual contributions from collective outcomes. Curriculum design matters more in MARL than in single-agent RL because agents co-adapt; starting with a smaller number of agents or simplified opponent policies and gradually scaling up complexity and population diversity tends to produce more robust final policies than training against maximum difficulty from the start. Parameter sharing across homogeneous agents (using a single shared policy network conditioned on agent identity) is a standard technique that dramatically reduces the number of trainable parameters and sample complexity when agents are functionally interchangeable.
## Benchmark Datasets & Evaluation
The StarCraft Multi-Agent Challenge (SMAC) is a widely used cooperative micromanagement benchmark where a team of allied units must jointly defeat a scripted or symmetric enemy army, testing coordination under partial observability. The Multi-Agent Particle Environment (MPE) suite, including tasks like Cooperative Navigation, Predator-Prey, and Physical Deception, provides lightweight 2D testbeds that were used to validate MADDPG and remain popular for rapid prototyping. PettingZoo offers a standardized, Gym-like API spanning cooperative, competitive, and mixed environments including Atari multiplayer games, classic board games, and continuous-control multi-agent tasks, facilitating fair cross-algorithm comparison. Google DeepMind's Melting Pot benchmark specifically targets generalization and social dilemma dynamics, evaluating whether agents trained in one social configuration behave robustly (e.g., cooperatively, fairly) when paired with unfamiliar co-players at test time. Google Research Football and the Hanabi Learning Environment test respectively long-horizon cooperative strategy with sparse team rewards and communication under severe informational asymmetry, since Hanabi restricts agents to see only their teammates' cards, not their own. Evaluation in MARL typically reports win rate or team reward against a fixed evaluation opponent pool rather than a single scalar return, since performance against one opponent policy can be a poor proxy for performance against the full space of possible strategies, and cross-play matrices (evaluating every trained policy against every other) are used to detect strategy collapse or overfitting to training partners.
## Key Challenges & Limitations
Non-stationarity remains the central theoretical challenge in MARL: from any single agent's perspective, the environment's effective dynamics shift continuously as co-learners update their policies, which violates the Markov assumption that convergence guarantees for single-agent Q-learning and policy gradient methods depend on. Credit assignment in cooperative teams with a shared team reward is difficult because an agent cannot easily determine how much of the team's success or failure is attributable to its own actions versus its teammates', motivating counterfactual and difference-reward methods that remain imperfect. Equilibrium selection is a further complication unique to multi-agent settings with multiple valid Nash equilibria: independent learners can converge to different, mutually incompatible equilibria, or oscillate without converging at all, particularly in general-sum games without a unique equilibrium. The curse of dimensionality in the joint action and observation space limits the scalability of fully centralized approaches to a modest number of agents, forcing a trade-off between the coordination benefits of centralization and the tractability of decentralized execution. Non-transitivity in competitive strategy spaces (analogous to rock-paper-scissors, where policy A beats B, B beats C, but C beats A) means self-play against only the latest policy checkpoint can cycle indefinitely without converging to a strong, general strategy, which is why league-based training with diverse historical opponents was necessary for AlphaStar and OpenAI Five. Finally, MARL evaluation and reproducibility are hampered by high variance across random seeds, sensitivity to reward-shaping choices, and a lack of standardized reporting protocols for opponent pools, all of which make published results harder to compare than in single-agent RL.
## Hyperparameter Tuning
Reward scaling and normalization are unusually consequential in MARL, since imbalanced reward magnitudes between agents can cause the training dynamics to be dominated by whichever agent's reward signal happens to have larger variance; per-agent reward normalization (e.g., running mean/std normalization) is a common corrective. The mixing network architecture and hypernetwork capacity in value-decomposition methods like QMIX and VDN require tuning independently of the per-agent utility network size, since an overly restrictive monotonic mixing function can prevent representation of the true joint value function in tasks with strong negative interactions between agents. Learning rate and update-frequency mismatches between actor and centralized critic networks in MADDPG-style methods need careful balancing, since a critic that lags too far behind the current joint policy produces stale, misleading gradient signals for the actor. The size and refresh rate of the opponent/self-play pool (how many past checkpoints to retain and how often to add new ones) directly trades off training stability against strategic diversity; too small a pool risks overfitting to a narrow strategy distribution, while too large a pool dilutes gradient signal on the current relevant matchups. Entropy regularization coefficients for exploration often need to be higher and decayed more slowly in MARL than in single-agent RL, since premature convergence to a deterministic joint policy in a non-stationary multi-agent setting can lock agents into a suboptimal equilibrium before adequate exploration of the joint strategy space has occurred.
## Real-World Applications & Case Studies
OpenAI Five demonstrated that large-scale self-play combined with PPO and massive parallel rollout could train a team of five cooperating agents to defeat professional human teams at Dota 2, a game requiring long-horizon planning, resource management, and implicit team communication over an average match length of 45 minutes. DeepMind's AlphaStar combined league-based self-play (main agents, main exploiters, and league exploiters) with imitation learning from human replays to reach Grandmaster level in StarCraft II, a real-time strategy game with a combinatorial action space and imperfect information due to fog of war. In autonomous driving research, MARL formulations model interactions among multiple vehicles at intersections or merging scenarios as a general-sum Markov game, where each vehicle agent must balance its own progress against safety constraints shared with nearby vehicles. Multi-agent formulations are used in algorithmic market-making and trading research, where competing trading agents are modeled as a Markov game to study emergent price dynamics, collusion risks, and market stability under learning-based participants. Warehouse and logistics robotics (e.g., Amazon's robotic fulfillment systems) apply decentralized MARL-inspired coordination policies to route large fleets of robots while avoiding collisions and balancing task allocation, often using CTDE-style training in simulation before decentralized deployment on physical hardware where centralized control would be latency-prohibitive.
## Integration with Other Methods
MARL is frequently combined with hierarchical reinforcement learning, where a high-level coordinator policy assigns sub-goals or roles to individual agents (role-based MARL), which then execute low-level policies independently, reducing the effective search space compared to flat joint-action learning. Graph neural networks are increasingly used as the backbone for agent communication and coordination, representing agents as nodes and modeling their interactions as message-passing over a learned or given graph structure, which naturally handles variable numbers of agents and local interaction neighborhoods. Model-based RL techniques, including learned dynamics models of the joint environment transition function, are being integrated into MARL to improve sample efficiency, particularly in physical multi-robot settings where real-world rollout is expensive. Meta-learning approaches (as covered in prior treatments of few-shot adaptation) are applied to MARL to enable rapid adaptation to new teammates or opponents at test time without full retraining, an important capability for ad-hoc teamwork scenarios where the composition of the team is not known during training. Offline RL techniques are also being extended to the multi-agent setting to enable training from logged multi-agent interaction datasets (e.g., historical traffic or trading data) without the need for costly or risky online exploration in the true multi-agent environment.
## Future Research Directions
A major open direction is improving sample efficiency for MARL, since the combinatorial growth of joint state-action spaces makes current methods highly data-hungry relative to single-agent RL, motivating continued work on model-based MARL, transfer learning across tasks with varying numbers of agents, and more effective use of offline interaction logs. Scalable coordination for very large agent populations (hundreds to thousands of agents) remains challenging beyond the approximations offered by mean-field methods, and research into more expressive yet tractable population-level abstractions is ongoing. Robust ad-hoc teamwork, in which an agent must cooperate effectively with previously unseen teammates or adapt to unknown opponent strategies without retraining, is an active area bridging MARL with meta-learning and opponent modeling. Emergent communication research continues to investigate whether agents can develop compositional, human-interpretable communication protocols purely through task-driven pressure, with implications for both explainability and human-agent teaming. Finally, safety and alignment concerns specific to multi-agent systems, such as preventing emergent collusion in competitive markets, ensuring fairness across agents in shared-resource settings, and providing formal guarantees against catastrophic miscoordination in safety-critical deployments like autonomous vehicle fleets, are increasingly recognized as prerequisites for real-world MARL deployment at scale.
## Summary & Key Takeaways
Multi-agent reinforcement learning generalizes single-agent RL to settings with multiple simultaneously-learning agents, formalized through Markov games and game-theoretic solution concepts such as Nash equilibria rather than a single optimal policy. The central technical challenge is non-stationarity: each agent's environment appears to shift as co-learners update, which breaks standard convergence guarantees and motivates architectural solutions like centralized training with decentralized execution (CTDE), value decomposition (QMIX, VDN), and centralized critics (MADDPG). Self-play and league-based training have driven landmark results in competitive games (Dota 2, StarCraft II) by exposing agents to a diverse, evolving population of opponents rather than a single fixed adversary, mitigating non-transitive strategy cycles. Cooperative settings introduce distinct challenges around credit assignment and communication, addressed through difference rewards, differentiable communication channels, and graph-based coordination architectures. Looking forward, sample efficiency, scalability to large populations, ad-hoc teamwork with unfamiliar partners, and multi-agent safety and alignment represent the field's most pressing open problems as MARL techniques move from research benchmarks toward real-world deployment in robotics, autonomous vehicles, and economic simulation.
---
## Appendix: Practical Labs
### Lab 1: Independent Q-Learning in a Grid-World Markov Game
import numpy as np
class IndependentQLearner:
"""A single agent running tabular Q-learning, oblivious to other agents."""
def __init__(self, n_states, n_actions, alpha=0.1, gamma=0.95, epsilon=0.2):
self.q_table = np.zeros((n_states, n_actions))
self.alpha = alpha
self.gamma = gamma
self.epsilon = epsilon
self.n_actions = n_actions
def select_action(self, state):
if np.random.rand() < self.epsilon:
return np.random.randint(self.n_actions)
return int(np.argmax(self.q_table[state]))
def update(self, state, action, reward, next_state):
best_next = np.max(self.q_table[next_state])
td_target = reward + self.gamma * best_next
td_error = td_target - self.q_table[state, action]
self.q_table[state, action] += self.alpha * td_error
def run_two_agent_grid_game(n_states=5, n_actions=2, n_episodes=500):
"""Two independent learners share a small grid; each treats the other as
part of the environment. Reward depends on whether both agents pick the
same action (a simple coordination game)."""
agent_a = IndependentQLearner(n_states, n_actions)
agent_b = IndependentQLearner(n_states, n_actions)
coordination_history = []
for episode in range(n_episodes):
state_a = np.random.randint(n_states)
state_b = np.random.randint(n_states)
action_a = agent_a.select_action(state_a)
action_b = agent_b.select_action(state_b)
# Coordination game: both agents rewarded only if actions match.
coordinated = int(action_a == action_b)
reward_a = 1.0 if coordinated else -0.1
reward_b = 1.0 if coordinated else -0.1
next_state_a = np.random.randint(n_states)
next_state_b = np.random.randint(n_states)
agent_a.update(state_a, action_a, reward_a, next_state_a)
agent_b.update(state_b, action_b, reward_b, next_state_b)
coordination_history.append(coordinated)
return agent_a, agent_b, coordination_history
def test_independent_q_learning():
agent_a, agent_b, history = run_two_agent_grid_game(n_episodes=2000)
early_rate = np.mean(history[:200])
late_rate = np.mean(history[-200:])
print(f"Early coordination rate: {early_rate:.3f}")
print(f"Late coordination rate: {late_rate:.3f}")
assert late_rate >= early_rate, "Coordination should improve or stay stable with learning"
print("Independent Q-learning coordination test passed.")
if __name__ == "__main__":
test_independent_q_learning()### Lab 2: Centralized Critic with Decentralized Actors (MADDPG-style)
import torch
import torch.nn as nn
import torch.nn.functional as F
class Actor(nn.Module):
"""Decentralized actor: maps local observation to action (continuous)."""
def __init__(self, obs_dim, action_dim, hidden_dim=64):
super().__init__()
self.net = nn.Sequential(
nn.Linear(obs_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, action_dim),
nn.Tanh(),
)
def forward(self, obs):
return self.net(obs)
class CentralizedCritic(nn.Module):
"""Centralized critic: sees the joint observation and joint action of
all agents during training, even though actors act on local info only."""
def __init__(self, joint_obs_dim, joint_action_dim, hidden_dim=128):
super().__init__()
self.net = nn.Sequential(
nn.Linear(joint_obs_dim + joint_action_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, 1),
)
def forward(self, joint_obs, joint_action):
x = torch.cat([joint_obs, joint_action], dim=-1)
return self.net(x)
def maddpg_critic_loss(critic, target_critic, actors, target_actors,
joint_obs, joint_action, rewards_i, joint_next_obs, gamma=0.95):
"""Compute the TD loss for agent i's centralized critic."""
with torch.no_grad():
next_actions = [target_actors[j](joint_next_obs[:, j, :]) for j in range(len(target_actors))]
joint_next_action = torch.cat(next_actions, dim=-1)
flat_next_obs = joint_next_obs.reshape(joint_next_obs.shape[0], -1)
target_q = target_critic(flat_next_obs, joint_next_action)
td_target = rewards_i + gamma * target_q
flat_obs = joint_obs.reshape(joint_obs.shape[0], -1)
current_q = critic(flat_obs, joint_action)
return F.mse_loss(current_q, td_target)
def test_maddpg_components():
n_agents = 3
obs_dim = 4
action_dim = 2
batch_size = 16
actors = [Actor(obs_dim, action_dim) for _ in range(n_agents)]
target_actors = [Actor(obs_dim, action_dim) for _ in range(n_agents)]
critic = CentralizedCritic(joint_obs_dim=obs_dim * n_agents, joint_action_dim=action_dim * n_agents)
target_critic = CentralizedCritic(joint_obs_dim=obs_dim * n_agents, joint_action_dim=action_dim * n_agents)
joint_obs = torch.randn(batch_size, n_agents, obs_dim)
joint_next_obs = torch.randn(batch_size, n_agents, obs_dim)
joint_action = torch.randn(batch_size, action_dim * n_agents)
rewards_i = torch.randn(batch_size, 1)
loss = maddpg_critic_loss(critic, target_critic, actors, target_actors,
joint_obs, joint_action, rewards_i, joint_next_obs)
assert loss.item() >= 0, "MSE loss should be non-negative"
print(f"Centralized critic TD loss: {loss.item():.4f}")
print("MADDPG component test passed.")
if __name__ == "__main__":
test_maddpg_components()### Lab 3: Value Decomposition Network (VDN-style Joint Q Factorization)
import torch
import torch.nn as nn
class AgentUtilityNetwork(nn.Module):
"""Per-agent utility network producing Q(o_i, a_i) for each local action."""
def __init__(self, obs_dim, n_actions, hidden_dim=32):
super().__init__()
self.net = nn.Sequential(
nn.Linear(obs_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, n_actions),
)
def forward(self, obs):
return self.net(obs)
class VDNMixer(nn.Module):
"""Value Decomposition Network: joint Q is simply the sum of per-agent
utilities. This additive factorization guarantees that the argmax over
the joint action decomposes into independent per-agent argmax choices."""
def forward(self, agent_q_values):
# agent_q_values: list of (batch,) tensors, one selected Q-value per agent
stacked = torch.stack(agent_q_values, dim=1) # (batch, n_agents)
return stacked.sum(dim=1) # (batch,)
def compute_joint_q(agent_nets, mixer, observations, actions):
"""observations: (batch, n_agents, obs_dim), actions: (batch, n_agents) long tensor."""
n_agents = len(agent_nets)
selected_qs = []
for i in range(n_agents):
q_values = agent_nets[i](observations[:, i, :]) # (batch, n_actions)
q_selected = q_values.gather(1, actions[:, i].unsqueeze(1)).squeeze(1)
selected_qs.append(q_selected)
return mixer(selected_qs)
def test_vdn_factorization():
n_agents = 4
obs_dim = 6
n_actions = 3
batch_size = 8
agent_nets = [AgentUtilityNetwork(obs_dim, n_actions) for _ in range(n_agents)]
mixer = VDNMixer()
observations = torch.randn(batch_size, n_agents, obs_dim)
actions = torch.randint(0, n_actions, (batch_size, n_agents))
joint_q = compute_joint_q(agent_nets, mixer, observations, actions)
assert joint_q.shape == (batch_size,), f"Expected shape ({batch_size},), got {joint_q.shape}"
# Verify additive factorization: manually sum the selected per-agent Qs.
manual_sum = torch.zeros(batch_size)
for i in range(n_agents):
q_values = agent_nets[i](observations[:, i, :])
manual_sum += q_values.gather(1, actions[:, i].unsqueeze(1)).squeeze(1)
assert torch.allclose(joint_q, manual_sum, atol=1e-5), "VDN mixer must equal sum of per-agent Q-values"
print(f"Joint Q-values (batch of {batch_size}): {joint_q.detach().numpy()}")
print("VDN factorization test passed.")
if __name__ == "__main__":
test_vdn_factorization()### Lab 4: Self-Play with an Opponent Pool
import numpy as np
class SimplePolicy:
"""A minimal parametric policy for a matrix game (e.g., rock-paper-scissors
style), represented as a probability distribution over actions that is
nudged via a simple policy-gradient-like update."""
def __init__(self, n_actions, learning_rate=0.05):
self.logits = np.zeros(n_actions)
self.lr = learning_rate
self.n_actions = n_actions
def probs(self):
exp_logits = np.exp(self.logits - np.max(self.logits))
return exp_logits / exp_logits.sum()
def sample_action(self):
return np.random.choice(self.n_actions, p=self.probs())
def update(self, action, reward):
probs = self.probs()
grad = -probs
grad[action] += 1.0
self.logits += self.lr * reward * grad
def snapshot(self):
clone = SimplePolicy(self.n_actions, self.lr)
clone.logits = self.logits.copy()
return clone
def payoff(action_a, action_b):
"""Rock-paper-scissors payoff for agent A: 0=rock,1=paper,2=scissors."""
if action_a == action_b:
return 0.0
wins = {(0, 2), (1, 0), (2, 1)} # rock beats scissors, paper beats rock, scissors beats paper
return 1.0 if (action_a, action_b) in wins else -1.0
def train_with_opponent_pool(n_iterations=3000, pool_size=5, snapshot_every=300):
main_agent = SimplePolicy(n_actions=3)
opponent_pool = [main_agent.snapshot()]
for it in range(n_iterations):
opponent = opponent_pool[np.random.randint(len(opponent_pool))]
action_a = main_agent.sample_action()
action_b = opponent.sample_action()
reward_a = payoff(action_a, action_b)
main_agent.update(action_a, reward_a)
if (it + 1) % snapshot_every == 0:
opponent_pool.append(main_agent.snapshot())
if len(opponent_pool) > pool_size:
opponent_pool.pop(0) # drop oldest to bound pool size
return main_agent, opponent_pool
def test_self_play_pool():
main_agent, pool = train_with_opponent_pool()
final_probs = main_agent.probs()
print(f"Final action distribution (rock, paper, scissors): {final_probs}")
print(f"Final opponent pool size: {len(pool)}")
# In a well-mixed rock-paper-scissors self-play run, the policy should not
# collapse to a single deterministic action against a diverse pool.
assert np.max(final_probs) < 0.95, "Policy should not fully collapse against a diverse opponent pool"
assert len(pool) <= 5, "Opponent pool should respect the maximum size bound"
print("Self-play opponent pool test passed.")
if __name__ == "__main__":
test_self_play_pool()