a3c

**A3C** (Asynchronous Advantage Actor-Critic) is an **asynchronous RL algorithm that runs multiple parallel agent instances to explore different parts of the environment simultaneously** — each instance independently computes gradients and asynchronously updates a shared global model. **A3C Architecture** - **Parallel Workers**: Multiple CPU workers each run their own copy of the environment. - **Async Updates**: Each worker computes gradients locally and asynchronously updates the shared global model. - **No Replay Buffer**: On-policy — no experience replay needed because parallel workers provide decorrelated data. - **Exploration**: Different workers explore different regions of the state space — diverse experience. **Why It Matters** - **CPU Efficiency**: A3C runs on CPUs — no GPU needed — workers parallelize across CPU cores. - **Decorrelation**: Parallel environments naturally decorrelate the experience stream — stabilizes training. - **Historical**: A3C was a breakthrough (Mnih et al., 2016) — but has been largely superseded by A2C and PPO. **A3C** is **parallel exploration** — running many agents asynchronously for efficient, decorrelated reinforcement learning.