sqil
**SQIL (Soft Q Imitation Learning)** combines imitation learning with soft Q-learning by treating expert demonstrations as receiving constant positive reward.
## What Is SQIL?
- **Method**: Assigns reward +1 to expert transitions, 0 to policy transitions
- **Foundation**: Built on Soft Actor-Critic (SAC) framework
- **Simplicity**: No adversarial training or reward learning required
- **Sample Efficiency**: Leverages off-policy replay from both expert and agent data
## Why SQIL Matters
SQIL achieves competitive imitation learning performance with minimal algorithmic complexity—just modify the reward signal in standard RL.
```python
# SQIL reward assignment
def sqil_reward(transition, is_expert):
if is_expert:
return 1.0 # Expert demonstrations
else:
return 0.0 # Agent-generated transitions
# Training combines both buffers
expert_batch = sample(expert_buffer)
agent_batch = sample(agent_buffer)
# SAC update with SQIL rewards
sac_update(expert_batch, rewards=1.0)
sac_update(agent_batch, rewards=0.0)
```
**Key Insight**: The agent learns to stay close to expert states because deviating leads to zero reward, naturally encouraging imitation without explicit behavior cloning loss.