DeLighT
DeLighT
**DeLiGHT** is a [transformer](https://paperswithcode.com/method/transformer) architecture that delivers parameter efficiency improvements by (1) within each Transformer block using [DExTra](https://paperswithcode.com/method/dextra), a deep and light-weight transformation, allowing for the use of [single-headed attention](https://paperswithcode.com/method/single-headed-attention) and bottleneck FFN layers and (2) across blocks using block-wise scaling, that allows for shallower and narrower [DeLighT blocks](https://paperswithcode.com/method/delight-block) near the input and wider and deeper DeLighT blocks near the output.
DeLighT: Deep and Light-weight Transformer
https://paperswithcode.com/paper/delight-very-deep-and-light-weight
DeLighT: Deep and Light-weight Transformer