LayerDrop
LayerDrop
**LayerDrop** is a form of structured [dropout](https://paperswithcode.com/method/dropout) for [Transformer](https://paperswithcode.com/method/transformer) models which has a regularization effect during training and allows for efficient pruning at inference time. It randomly drops layers from the Transformer according to an "every other" strategy where pruning with a rate $p$ means dropping the layers at depth $d$ such that $d = 0\left\(\text{mod}\left(\text{floor}\left(\frac{1}{p}\right)\right)\right)$.
Reducing Transformer Depth on Demand with Structured Dropout
https://paperswithcode.com/paper/reducing-transformer-depth-on-demand-with-1
Reducing Transformer Depth on Demand with Structured Dropout