Subformer
Subformer
**Subformer** is a [Transformer](https://paperswithcode.com/method/transformer) that combines sandwich-style parameter sharing, which overcomes naive cross-layer parameter sharing in generative models, and self-attentive embedding factorization (SAFE). In SAFE, a small self-attention layer is used to reduce embedding parameter count.
Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformers
https://paperswithcode.com/paper/subformer-exploring-weight-sharing-for
Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformers