ConViT
ConViT
**ConViT** is a type of [vision transformer](https://paperswithcode.com/method/vision-transformer) that uses a gated positional self-attention module ([GPSA](https://paperswithcode.com/method/gpsa)), a form of positional self-attention which can be equipped with a “soft” convolutional inductive bias. The GPSA layers are initialized to mimic the locality of convolutional layers, then each attention head is given the freedom to escape locality by adjusting a gating parameter regulating the attention paid to position versus content information.
ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases
https://paperswithcode.com/paper/convit-improving-vision-transformers-with
ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases