Linformer
Linformer
**Linformer** is a linear [Transformer](https://paperswithcode.com/method/transformer) that utilises a linear self-attention mechanism to tackle the self-attention bottleneck with [Transformer models](https://paperswithcode.com/methods/category/transformers). The original [scaled dot-product attention](https://paperswithcode.com/method/scaled) is decomposed into multiple smaller attentions through linear projections, such that the combination of these operations forms a low-rank factorization of the original attention.
Linformer: Self-Attention with Linear Complexity
https://paperswithcode.com/paper/linformer-self-attention-with-linear
Linformer: Self-Attention with Linear Complexity