lion optimizer

Lion optimizer is a memory-efficient alternative to Adam that uses only the sign of gradients for updates. **Algorithm**: Track momentum (m), update weights using sign(m) instead of scaled gradients. w -= lr * sign(m). **Memory savings**: Only stores momentum (1 state per parameter) vs Adams 2 states. 2x memory reduction for optimizer states. **Discovery**: Found via AutoML/neural architecture search at Google. Searched over update rules. **Performance**: Matches or exceeds AdamW on vision and language tasks while using less memory. **Hyperparameters**: lr (typically higher than Adam, ~3e-4 to 1e-3), beta1 (0.9), beta2 (0.99). **Sign-based updates**: Uniform step size regardless of gradient magnitude. Can be more stable for some tasks. **Use cases**: Memory-constrained training, large batch training, when AdamW works. **Limitations**: May be sensitive to batch size, less established than Adam, fewer tuning guidelines. **Implementation**: Available in optax (JAX), community PyTorch implementations. **Current status**: Gaining adoption but AdamW remains default. Worth trying for memory savings.