GPSA
Gated Positional Self-Attention
**Gated Positional Self-Attention (GPSA)** is a self-attention module for vision transformers, used in the [ConViT](https://paperswithcode.com/method/convit) architecture, that can be initialized as a convolutional layer -- helping a ViT learn inductive biases about locality.
ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases
https://paperswithcode.com/paper/convit-improving-vision-transformers-with
ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases