Reinforcement Learning for Process Control
# Reinforcement Learning for Process Control
## Introduction & Motivation
RL agents learn optimal control policies through trial-and-error interaction. Critical for process optimization, autonomous systems, and complex decision-making where reward structures guide discovery of superior operating strategies without explicit programming.
Motivation: Apply RL to discover optimal process control policies.
Applications: Process optimization, equipment control, autonomous systems, sequential decision-making.
---
## Core Concepts & Theory
### States and Actions
Environment representation.
### Rewards
Learning signal design.
### Value Functions
Expected return estimation.
### Policy Optimization
Improving decision-making.
---
## Mathematical Formulation
Bellman Equation:
$$V(s) = \mathbb{E}[R_t + \gamma V(s')]$$
TD Learning:
$$V(s) \leftarrow V(s) + \alpha[R + \gamma V(s') - V(s)]$$
Policy Gradient:
$$
abla_ heta J = \mathbb{E}[
abla_ heta \ln \pi_ heta(a|s) Q(s,a)]$$
---
## Advanced Theory & Extensions
### Model-Based RL
Learning environment model.
### Actor-Critic
Combined value and policy.
### Exploration-Exploitation
Balancing discovery and exploitation.
---
## Computational Considerations
QL: O(S·A·E) for states, actions, episodes.
Policy Gradient: O(T²) for trajectory length T.
Neural RL: O(N·B) for samples N, batch B.
---
## Practical Implementation Strategies
### Reward Shaping
Designing effective rewards.
### Exploration Strategies
Epsilon-greedy and UCB.
### Function Approximation
Neural network policies.
---
## Benchmark Datasets & Evaluation
OpenAI Gym: Standard environments.
Control Benchmarks: Continuous control.
Process Simulators: Domain-specific environments.
---
## Key Challenges & Limitations
### Sample Efficiency
Data requirement.
### Stability
Training convergence.
### Generalization
Policy transfer.
---
## Hyperparameter Tuning
Learning rate: 1e-4 to 1e-2.
Discount factor: 0.9-0.99.
Exploration: 0.01-0.1.
---
## Real-World Applications & Case Studies
Chemical Processes: Reactor control.
Manufacturing: Equipment optimization.
Robotics: Motion control.
---
## Integration with Other Methods
RL + planning; + neural networks; + transfer learning.
---
## Summary & Key Takeaways
RL enables autonomous policy discovery.
Principles:
1. Environment: Define state-action space.
2. Reward: Design learning signal.
3. Learning: Optimize value or policy.
4. Exploration: Balance discovery.
5. Control: Deploy learned policies.
---
## Appendix: Practical Labs
### Lab 1: Q-Learning
import numpy as np
class QLearningAgent:
def __init__(self, n_states, n_actions, lr=0.1, gamma=0.9):
self.n_states = n_states
self.n_actions = n_actions
self.lr = lr
self.gamma = gamma
self.Q = np.zeros((n_states, n_actions))
def select_action(self, state, epsilon=0.1):
"""Epsilon-greedy action selection"""
if np.random.random() < epsilon:
return np.random.randint(self.n_actions)
return np.argmax(self.Q[state])
def update(self, state, action, reward, next_state):
"""Q-learning update"""
target = reward + self.gamma * np.max(self.Q[next_state])
td_error = target - self.Q[state, action]
self.Q[state, action] += self.lr * td_error
# Test
agent = QLearningAgent(n_states=5, n_actions=2)
# Simulate episodes
for episode in range(100):
state = np.random.randint(5)
for _ in range(10):
action = agent.select_action(state)
reward = 1 if action == 0 else 0
next_state = (state + action) % 5
agent.update(state, action, reward, next_state)
state = next_state
print(f"✓ Q-learning training complete")
print(f" Q-table shape: {agent.Q.shape}")### Lab 2: Policy Gradient
import numpy as np
class PolicyGradientAgent:
def __init__(self, n_states, n_actions, lr=0.01):
self.n_states = n_states
self.n_actions = n_actions
self.lr = lr
self.policy_weights = np.random.randn(n_states, n_actions) * 0.1
def policy(self, state):
"""Softmax policy"""
logits = self.policy_weights[state]
return np.exp(logits) / np.sum(np.exp(logits))
def select_action(self, state):
"""Sample from policy"""
probs = self.policy(state)
return np.random.choice(self.n_actions, p=probs)
def update(self, state, action, advantage):
"""Policy gradient update"""
probs = self.policy(state)
grad = np.zeros(self.n_actions)
grad[action] = 1
grad -= probs
self.policy_weights[state] += self.lr * advantage * grad
# Test
agent = PolicyGradientAgent(n_states=5, n_actions=2)
for _ in range(50):
state = np.random.randint(5)
action = agent.select_action(state)
advantage = np.random.randn()
agent.update(state, action, advantage)
print(f"✓ Policy gradient training complete")### Lab 3: Value Iteration
import numpy as np
def value_iteration(n_states, n_actions, transitions, rewards, gamma=0.9, n_iter=100):
"""Value iteration algorithm"""
V = np.zeros(n_states)
for _ in range(n_iter):
V_old = V.copy()
for s in range(n_states):
values = np.zeros(n_actions)
for a in range(n_actions):
# Expected future reward
for next_s in range(n_states):
prob = transitions[s, a, next_s]
r = rewards[s, a]
values[a] += prob * (r + gamma * V_old[next_s])
V[s] = np.max(values)
if np.linalg.norm(V - V_old) < 1e-6:
break
return V
# Simple gridworld
n_states = 10
n_actions = 2
transitions = np.random.rand(n_states, n_actions, n_states)
transitions /= transitions.sum(axis=2, keepdims=True)
rewards = np.random.randn(n_states, n_actions)
V = value_iteration(n_states, n_actions, transitions, rewards)
print(f"✓ Value iteration convergence")### Lab 4: Process Control
import numpy as np
class ProcessController:
def __init__(self, target_setpoint=100):
self.target = target_setpoint
self.Q = np.zeros((10, 5)) # 10 states, 5 actions
self.lr = 0.1
self.gamma = 0.95
def discretize_state(self, process_value):
"""Discretize continuous state"""
return int(np.clip(process_value / 20, 0, 9))
def compute_reward(self, current, target):
"""Reward based on setpoint error"""
error = abs(current - target)
return 100 * np.exp(-error / 50)
def select_action(self, state, epsilon=0.05):
"""Choose control action"""
if np.random.random() < epsilon:
return np.random.randint(5) - 2 # -2 to 2
return np.argmax(self.Q[state]) - 2
def learn_control(self, process_readings, n_episodes=100):
"""Learn optimal control"""
for _ in range(n_episodes):
for reading in process_readings:
state = self.discretize_state(reading)
action = self.select_action(state)
reward = self.compute_reward(reading + action, self.target)
next_state = self.discretize_state(reading + action)
target_val = reward + self.gamma * np.max(self.Q[next_state])
self.Q[state, action+2] += self.lr * (target_val - self.Q[state, action+2])
controller = ProcessController(target_setpoint=100)
readings = np.random.uniform(80, 120, 50)
controller.learn_control(readings)
print(f"✓ Process control policy learned")---