reinforcement learning
**Reinforcement Learning Fundamentals**
**RL Overview**
Agent learns by interacting with environment, receiving rewards for good actions.
```
[Agent]
|
| action
v
[Environment]
|
| state, reward
v
[Agent] (update policy)
```
**Key Concepts**
| Concept | Description |
|---------|-------------|
| State (s) | Current environment observation |
| Action (a) | Agent choice |
| Reward (r) | Feedback signal |
| Policy (π) | Maps states to actions |
| Value (V) | Expected cumulative reward |
| Q-value (Q) | Expected reward for action in state |
**RL Algorithms**
**Value-Based (Q-Learning, DQN)**
Learn value of state-action pairs:
```python
# Q-learning update
Q[s][a] = Q[s][a] + lr * (reward + gamma * max(Q[s_next]) - Q[s][a])
```
**Policy Gradient**
Directly optimize policy:
```python
# Policy gradient update
loss = -log_prob(action) * advantage
```
**Actor-Critic**
Combine value estimation with policy optimization:
```python
# Critic estimates value
value = critic(state)
# Actor updates policy using advantage
advantage = reward + gamma * critic(next_state) - value
actor_loss = -log_prob(action) * advantage
```
**Common Algorithms**
| Algorithm | Type | Use Case |
|-----------|------|----------|
| DQN | Value-based | Discrete actions |
| PPO | Policy gradient | General purpose |
| SAC | Actor-critic | Continuous control |
| A3C | Distributed | Parallel training |
**RL for LLMs (RLHF)**
Fine-tune LLMs with human preferences:
```
1. Collect human preference data
2. Train reward model
3. Use RL (PPO) to optimize LLM against reward
```
**Libraries**
| Library | Features |
|---------|----------|
| Stable Baselines3 | Ready-to-use algorithms |
| RLlib | Distributed RL |
| Gymnasium | Environments |
| Tianshou | Modular RL |
**Challenges**
| Challenge | Consideration |
|-----------|---------------|
| Sample efficiency | RL often needs many samples |
| Reward design | Reward hacking |
| Exploration | Balancing exploration vs exploitation |
| Stability | Training can be unstable |
**Best Practices**
- Start with well-tested algorithms (PPO)
- Normalize observations and rewards
- Monitor training closely
- Use domain knowledge for reward shaping
- Consider offline RL for data efficiency