Activation Regularization
Activation Regularization
**Activation Regularization (AR)**, or $L\_{2}$ activation regularization, is regularization performed on activations as opposed to weights. It is usually used in conjunction with [RNNs](https://paperswithcode.com/methods/category/recurrent-neural-networks). It is defined as:

$$\alpha{L}\_{2}\left(m\circ{h\_{t}}\right) $$

where $m$ is a [dropout](https://paperswithcode.com/method/dropout) mask used by later parts of the model, $L\_{2}$ is the $L\_{2}$ norm, and $h_{t}$ is the output of an RNN at timestep $t$, and $\alpha$ is a scaling coefficient. 

When applied to the output of a dense layer, AR penalizes activations that are substantially away from 0, encouraging activations to remain small.
Revisiting Activation Regularization for Language RNNs
https://paperswithcode.com/paper/revisiting-activation-regularization-for
Revisiting Activation Regularization for Language RNNs