RMSProp
RMSProp
**RMSProp** is an unpublished adaptive learning rate optimizer [proposed by Geoff Hinton](http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf). The motivation is that the magnitude of gradients can differ for different weights, and can change during learning, making it hard to choose a single global learning rate. RMSProp tackles this by keeping a moving average of the squared gradient and adjusting the weight updates by this magnitude. The gradient updates are performed as:

$$E\left[g^{2}\right]\_{t} = \gamma E\left[g^{2}\right]\_{t-1} + \left(1 - \gamma\right) g^{2}\_{t}$$

$$\theta\_{t+1} = \theta\_{t} - \frac{\eta}{\sqrt{E\left[g^{2}\right]\_{t} + \epsilon}}g\_{t}$$

Hinton suggests $\gamma=0.9$, with a good default for $\eta$ as $0.001$.

Image: [Alec Radford](https://twitter.com/alecrad)