Marl - Multi-Agent Reinforcement Learning
# MARL - Multi-Agent Reinforcement Learning
## Introduction & Motivation
MARL: multiple agents learning simultaneously in shared environment. Competitive and cooperative settings. Enables scalable systems and emergent behaviors through agent interaction.
Motivation: Enable multiple agents to coordinate or compete effectively.
Applications: Game playing, swarm robotics, traffic control.
---
## Core Concepts & Theory
### Cooperative vs Competitive
Shared vs conflicting objectives.
### Credit Assignment
Attribute reward to individual agents.
### Communication
Information sharing between agents.
### Emergent Behavior
Coordination without explicit rules.
---
## Mathematical Formulation
Joint Policy:
$$\pi = (\pi_1, \pi_2, ..., \pi_n)$$
Agent Reward:
$$R_i = r_i(s, a_1, ..., a_n)$$
Nash Equilibrium:
$$\pi_i^* = \arg\max_{\pi_i} \mathbb{E}[R_i | \pi_i, \pi_{-i}^*]$$
---
## Advanced Theory & Extensions
### CTDE Framework
Centralized training, decentralized execution.
### Value Decomposition
Factor joint value into individual components.
### Communication Protocols
Learn what to communicate.
---
## Computational Considerations
State Space: O(S^n) for n agents.
Action Space: O(A^n) combinations.
Complexity: Exponential in agent count.
---
## Practical Implementation Strategies
### Independent Learners
Each agent learns independently.
### Centralized Value
Shared value function.
### Message Passing
Information exchange between agents.
---
## Benchmark Datasets & Evaluation
StarCraft: Complex MARL benchmark.
Atari Multi-Agent: Competitive games.
MuJoCo Multi-Agent: Cooperative control.
---
## Key Challenges & Limitations
### Non-Stationarity
Other agents' policies changing.
### Credit Assignment
Isolating individual contributions.
### Scalability
Complexity grows exponentially.
---
## Hyperparameter Tuning
Learning rate: 1e-4.
Communication rate: Full (all steps).
Coordination: Implicit vs explicit.
---
## Real-World Applications & Case Studies
Game AI: StarCraft, Dota.
Robotics: Swarm coordination.
Traffic: Flow management.
---
## Integration with Other Methods
MARL + communication learning; + social influence; + curriculum.
---
## Summary & Key Takeaways
MARL enables multi-agent coordination and competition.
Principles:
1. Multiple: Independent agents.
2. Coordination: Implicit or explicit.
3. Competition: Conflicting objectives.
4. Cooperation: Shared goals.
5. Emergence: System-level behaviors.
---
## Appendix: Practical Labs
### Lab 1: Multi-Agent Environment
import numpy as np
class MultiAgentEnv:
def __init__(self, num_agents=3, grid_size=10):
self.num_agents = num_agents
self.grid_size = grid_size
self.positions = np.random.rand(num_agents, 2) * grid_size
self.goal = np.random.rand(2) * grid_size
def step(self, actions):
"""Execute joint action"""
rewards = []
for i, action in enumerate(actions):
# Move agent
move = np.array([0, 0])
if action == 0: # Up
move = [0, 1]
elif action == 1: # Down
move = [0, -1]
elif action == 2: # Right
move = [1, 0]
else: # Left
move = [-1, 0]
self.positions[i] += move
# Compute reward
dist = np.linalg.norm(self.positions[i] - self.goal)
reward = -dist / self.grid_size
rewards.append(reward)
return rewards, self.positions.copy()
env = MultiAgentEnv(num_agents=3)
actions = [0, 1, 2] # Joint action
rewards, positions = env.step(actions)
print(f"✓ MARL environment: {len(rewards)} agent rewards: {rewards}")### Lab 2: Nash Equilibrium Computation
import numpy as np
def compute_payoff_matrix(num_strategies=3):
"""Generate 2-player payoff matrix"""
payoff_p1 = np.random.randn(num_strategies, num_strategies)
payoff_p2 = np.random.randn(num_strategies, num_strategies)
return payoff_p1, payoff_p2
def find_pure_nash_equilibrium(payoff_p1, payoff_p2):
"""Find pure strategy Nash equilibrium"""
nash_equilibria = []
for i in range(len(payoff_p1)):
for j in range(len(payoff_p1[0])):
# Check if (i, j) is Nash
is_nash = True
# P1 prefers not to deviate
if i > 0 and payoff_p1[i-1, j] > payoff_p1[i, j]:
is_nash = False
# P2 prefers not to deviate
if j > 0 and payoff_p2[i, j-1] > payoff_p2[i, j]:
is_nash = False
if is_nash:
nash_equilibria.append((i, j))
return nash_equilibria
payoff1, payoff2 = compute_payoff_matrix(3)
nash = find_pure_nash_equilibrium(payoff1, payoff2)
print(f"✓ Nash equilibria found: {len(nash)}")### Lab 3: Value Decomposition
import numpy as np
def decompose_joint_value(joint_value, num_agents):
"""Decompose joint value into individual components"""
individual_values = joint_value / num_agents
# Ensure components sum to joint value
decomposed = np.ones(num_agents) * individual_values
return decomposed
def reconstruct_joint_value(individual_values):
"""Reconstruct joint value from components"""
joint_value = np.sum(individual_values)
return joint_value
joint_v = 10.0
num_agents = 3
individual_v = decompose_joint_value(joint_v, num_agents)
reconstructed = reconstruct_joint_value(individual_v)
assert abs(reconstructed - joint_v) < 1e-6
print(f"✓ Value decomposition: {joint_v} -> {individual_v} -> {reconstructed}")### Lab 4: Cooperative Agent Learning
import numpy as np
class CooperativeMARL:
def __init__(self, num_agents=3, state_dim=10):
self.num_agents = num_agents
self.state_dim = state_dim
# Shared value network
self.value_network = np.random.randn(state_dim, 1) * 0.01
# Individual policy networks
self.policies = [np.random.randn(state_dim, 4) * 0.01 for _ in range(num_agents)]
def select_actions(self, state):
"""All agents select actions from shared state"""
actions = []
for policy in self.policies:
action_logits = state @ policy
action = np.argmax(action_logits)
actions.append(action)
return actions
def update(self, state, actions, reward, next_state, gamma=0.99):
"""Cooperative update with shared value"""
# Joint reward for all agents
shared_reward = reward / self.num_agents
# Value update
v_current = (state @ self.value_network)[0]
v_next = (next_state @ self.value_network)[0]
td_error = shared_reward + gamma * v_next - v_current
# Policy update for each agent
policy_losses = []
for i, action in enumerate(actions):
policy_loss = -td_error # Maximize value
policy_losses.append(policy_loss)
return sum(policy_losses), td_error
marl = CooperativeMARL(num_agents=3)
state = np.random.randn(10)
actions = marl.select_actions(state)
loss, td_err = marl.update(state, actions, 1.0, state*0.9)
print(f"✓ Cooperative MARL: actions={actions}, total loss={loss:.3f}")---