T-Fixup
T-Fixup
**T-Fixup** is an [initialization](https://paperswithcode.com/methods/category/initialization) method for [Transformers](https://paperswithcode.com/methods/category/transformers) that aims to remove the need for [layer normalization](https://paperswithcode.com/method/layer-normalization) and [warmup](https://paperswithcode.com/method/linear-warmup). The initialization procedure is as follows:

- Apply [Xavier initialization](https://paperswithcode.com/method/xavier-initialization) for all parameters excluding input embeddings. Use Gaussian initialization $\mathcal{N}\left(0, d^{-\frac{1}{2}}\right)$ for input embeddings where $d$ is the embedding dimension.
- Scale $\mathbf{v}\_{d}$ and $\mathbf{w}\_{d}$ matrices in each decoder [attention block](https://paperswithcode.com/method/multi-head-attention), weight matrices in each decoder [MLP block](https://paperswithcode.com/method/position-wise-feed-forward-layer) and input embeddings $\mathbf{x}$ and $\mathbf{y}$ in encoder and decoder by $(9 N)^{-\frac{1}{4}}$
- Scale $\mathbf{v}\_{e}$ and $\mathbf{w}\_{e}$ matrices in each encoder [attention block](https://paperswithcode.com/method/multi-head-attention) and weight matrices in each encoder [MLP block](https://paperswithcode.com/method/position-wise-feed-forward-layer) by $0.67 N^{-\frac{1}{4}}$
Improving Transformer Optimization Through Better Initialization
https://paperswithcode.com/paper/improving-transformer-optimization-through-1
Improving Transformer Optimization Through Better Initialization