Cosine Annealing
Cosine Annealing
**Cosine Annealing** is a type of learning rate schedule that has the effect of starting with a large learning rate that is relatively rapidly decreased to a minimum value before being increased rapidly again. The resetting of the learning rate acts like a simulated restart of the learning process and the re-use of good weights as the starting point of the restart is referred to as a "warm restart" in contrast to a "cold restart" where a new set of small random numbers may be used as a starting point.

$$\eta\_{t} = \eta\_{min}^{i} + \frac{1}{2}\left(\eta\_{max}^{i}-\eta\_{min}^{i}\right)\left(1+\cos\left(\frac{T\_{cur}}{T\_{i}}\pi\right)\right)
$$

Where where $\eta\_{min}^{i}$ and $ \eta\_{max}^{i}$ are ranges for the learning rate, and $T\_{cur}$ account for how many epochs have been performed since the last restart.

Text Source: [Jason Brownlee](https://machinelearningmastery.com/snapshot-ensemble-deep-learning-neural-network/)

Image Source: [Gao Huang](https://www.researchgate.net/figure/Training-loss-of-100-layer-DenseNet-on-CIFAR10-using-standard-learning-rate-blue-and-M_fig2_315765130)
SGDR: Stochastic Gradient Descent with Warm Restarts
https://paperswithcode.com/paper/sgdr-stochastic-gradient-descent-with-warm
SGDR: Stochastic Gradient Descent with Warm Restarts