generative adversarial imitation learning

**GAIL** (Generative Adversarial Imitation Learning) is an **imitation learning algorithm that uses a GAN-like framework to match the agent's state-action distribution to the expert's** — a discriminator distinguishes expert from learner trajectories, and the learner's policy is trained to fool the discriminator. **GAIL Framework** - **Discriminator**: $D(s,a)$ — classifies whether $(s,a)$ came from the expert or the learner. - **Generator (Policy)**: $pi_ heta(a|s)$ — trained to produce behavior indistinguishable from the expert's. - **Reward**: $r(s,a) = -log(1 - D(s,a))$ — the discriminator's output serves as the RL reward. - **Training**: Alternate between updating the discriminator (on expert vs. learner data) and the policy (using the discriminator reward). **Why It Matters** - **No Reward Engineering**: GAIL learns directly from demonstrations — no manual reward function design. - **Distribution Matching**: Matches the entire occupancy measure, not just per-state actions — handles distribution shift. - **End-to-End**: Combines IRL and RL into a single adversarial training loop — simpler than two-stage IRL. **GAIL** is **the GAN of imitation** — adversarially matching the learner's behavior distribution to the expert's for robust imitation learning.