NormFormer
NormFormer
**NormFormer** is a type of [Pre-LN](https://paperswithcode.com/method/layer-normalization) transformer that adds three normalization operations to each layer: a Layer Norm after self attention, head-wise scaling of self-attention outputs, and a Layer Norm after the first [fully connected layer](https://paperswithcode.com/method/position-wise-feed-forward-layer). The modifications introduce a small number of additional learnable parameters, which provide a cost-effective way for each layer to change the magnitude of its features, and therefore the magnitude of the gradients to subsequent components.
NormFormer: Improved Transformer Pretraining with Extra Normalization
https://paperswithcode.com/paper/normformer-improved-transformer-pretraining-1
NormFormer: Improved Transformer Pretraining with Extra Normalization