AdamW
AdamW
**AdamW** is a stochastic optimization method that modifies the typical implementation of weight decay in [Adam](https://paperswithcode.com/method/adam), by decoupling [weight decay](https://paperswithcode.com/method/weight-decay) from the gradient update. To see this, $L\_{2}$ regularization in Adam is usually implemented with the below modification where $w\_{t}$ is the rate of the weight decay at time $t$:

$$ g\_{t} = \nabla{f\left(\theta\_{t}\right)} + w\_{t}\theta\_{t}$$

while AdamW adjusts the weight decay term to appear in the gradient update:

$$ \theta\_{t+1, i} = \theta\_{t, i} - \eta\left(\frac{1}{\sqrt{\hat{v}\_{t} + \epsilon}}\cdot{\hat{m}\_{t}} + w\_{t, i}\theta\_{t, i}\right), \forall{t}$$
Decoupled Weight Decay Regularization
https://paperswithcode.com/paper/decoupled-weight-decay-regularization
Decoupled Weight Decay Regularization