Demon CM
Demon CM
**Demon CM**, or **SGD with Momentum and Demon**,  is the [Demon](https://paperswithcode.com/method/demon) momentum rule applied to [SGD with momentum](https://paperswithcode.com/method/sgd-with-momentum).

$$ \beta\_{t} = \beta\_{init}\cdot\frac{\left(1-\frac{t}{T}\right)}{\left(1-\beta\_{init}\right) + \beta\_{init}\left(1-\frac{t}{T}\right)} $$

$$ \theta\_{t+1} = \theta\_{t} - \eta{g}\_{t} + \beta\_{t}v\_{t} $$

$$ v\_{t+1} = \beta\_{t}{v\_{t}} - \eta{g\_{t}} $$
Demon: Improved Neural Network Training with Momentum Decay
https://paperswithcode.com/paper/decaying-momentum-helps-neural-network
Demon: Improved Neural Network Training with Momentum Decay