swa-gaussian

**SWAG** (SWA-Gaussian) is an **approximation to Bayesian deep learning that uses the SWA trajectory to fit a Gaussian distribution over weights** — capturing both the mean (SWA solution) and the covariance (spread of the SWA trajectory) for uncertainty estimation. **How Does SWAG Work?** - **Mean**: $ar{ heta}$ from SWA (average of collected checkpoints). - **Covariance**: Estimate a low-rank + diagonal covariance from the deviations of collected checkpoints from the mean. - **Posterior**: $q( heta) = mathcal{N}(ar{ heta}, Sigma_{SWAG})$ (Gaussian approximate posterior). - **Inference**: Sample multiple models from the posterior and average predictions (Bayesian model averaging). - **Paper**: Maddox et al. (2019). **Why It Matters** - **Uncertainty**: Provides calibrated uncertainty estimates without the cost of full Bayesian inference. - **Efficient**: Only requires the SWA trajectory — no special modifications to training. - **Scalable**: Works for modern deep networks (ResNets, etc.) where full Bayesian methods are intractable. **SWAG** is **SWA with uncertainty** — using the natural variation in the SWA trajectory to estimate a Bayesian posterior for calibrated predictions.