Adam-mini
Adaptive Moment Estimation - Mini
**Adam-mini** is a memory-efficient [Adam](https://paperswithcode.com/method/adam) variant that achieves on-par or better performance than [AdamW](https://paperswithcode.com/method/adamw) with 45% to 50% less memory footprint. Adam-mini reduces the memory footprint by cutting down the learning rate resources in Adam (i.e., $ {1}/{\sqrt{v}} $). The authors find that ≥ 90% of these learning rates in $ v $ could be harmlessly removed if they (1) carefully partition the parameters into blocks following their proposed principle on Hessian structure; (2) assign a single but good learning rate to each parameter block. They further find that, for each of these parameter blocks, there exists a single high-quality learning rate that can outperform Adam, provided that sufficient resources are available to search it out.
Adam-mini: Use Fewer Learning Rates To Gain More
https://paperswithcode.com/paper/adam-mini-use-fewer-learning-rates-to-gain
Adam-mini: Use Fewer Learning Rates To Gain More