YellowFin
YellowFin
**YellowFin** is a learning rate and momentum tuner motivated by robustness properties and analysis of quadratic objectives. It stems from a known but obscure fact: the momentum operator's spectral radius is constant in a large subset of the hyperparameter space. For quadratic objectives, the optimizer tunes both the learning rate and the momentum to keep the hyperparameters within a region in which the convergence rate is a constant rate equal to the root momentum. This notion is extended empirically to non-convex objectives. On every iteration, YellowFin optimizes the hyperparameters to minimize a local quadratic optimization.
YellowFin and the Art of Momentum Tuning
https://paperswithcode.com/paper/yellowfin-and-the-art-of-momentum-tuning
YellowFin and the Art of Momentum Tuning