Fixup Initialization
Fixup Initialization
**FixUp Initialization**, or **Fixed-Update Initialization**, is an initialization method that rescales the standard initialization of [residual branches](https://paperswithcode.com/method/residual-block) by adjusting for the network architecture. Fixup aims to enables training very deep [residual networks](https://paperswithcode.com/method/resnet) stably at a maximal learning rate without [normalization](https://paperswithcode.com/methods/category/normalization).

The steps are as follows:

1. Initialize the classification layer and the last layer of each residual branch to 0.

2. Initialize every other layer using a standard method, e.g. [Kaiming Initialization](https://paperswithcode.com/method/he-initialization), and scale only the weight layers inside residual branches by $L^{\frac{1}{2m-2}}$.

3. Add a scalar multiplier (initialized at 1) in every branch and a scalar bias (initialized at 0) before each [convolution](https://paperswithcode.com/method/convolution), [linear](https://paperswithcode.com/method/linear-layer), and element-wise activation layer.
Fixup Initialization: Residual Learning Without Normalization
https://paperswithcode.com/paper/fixup-initialization-residual-learning
Fixup Initialization: Residual Learning Without Normalization