Fast Geometric Ensembling (FGE) is an efficient ensemble construction technique that exploits the geometric structure of the loss landscape to collect diverse model checkpoints along a single training trajectory, using a cyclical learning rate schedule with carefully chosen cycle length to traverse low-loss paths connecting different local minima. FGE extends the snapshot ensemble concept by leveraging the observation that good minima in deep neural network loss landscapes are connected by low-loss "tunnels."

Why FGE Matters in AI/ML: FGE provides high-quality ensembles at single-training-run cost by exploiting the connected geometry of the loss landscape, producing models that are diverse yet individually high-performing by traversing the low-loss manifold between minima.

• Loss landscape connectivity — Research shows that independently trained neural networks converge to minima connected by low-loss paths; FGE exploits this by traversing these paths during training, collecting checkpoints at different points along the connected low-loss manifold • High-frequency cyclical schedule — FGE uses shorter learning rate cycles than standard snapshot ensembles, enabling more frequent checkpoint collection; the shorter cycles keep the model in low-loss regions while providing sufficient perturbation for diversity • Geometric averaging — Beyond simple prediction averaging, FGE supports weight-space averaging of checkpoints along the trajectory, producing a single model (SWA-style) that approximates the ensemble at no additional inference cost • Diversity vs. quality tradeoff — FGE carefully balances checkpoint diversity (models should make different predictions) against individual quality (each checkpoint should perform well); the connected loss landscape ensures both conditions hold simultaneously • Relationship to SWA — Stochastic Weight Averaging (SWA) averages the weights collected by FGE into a single model, while FGE keeps them separate for ensemble prediction; FGE provides better uncertainty estimation while SWA provides better single-model performance

Fast Geometric Ensembling leverages the connected geometry of neural network loss landscapes to efficiently collect diverse, high-quality model checkpoints along low-loss paths, providing ensemble-quality predictions and uncertainty estimates at the computational cost of a single training run—making it the optimal choice when training budget constraints preclude independent ensemble training.