HaloNet
HaloNet
A **HaloNet** is a self-attention based model for efficient image classification. It relies on a local self-attention architecture that efficiently maps to existing hardware with haloing. The formulation breaks translational equivariance, but the authors observe that it improves  throughput and accuracies over the centered local self-attention used in regular self-attention. The approach also utilises a strided self-attentive downsampling operation for multi-scale feature extraction.
Scaling Local Self-Attention for Parameter Efficient Visual Backbones
https://paperswithcode.com/paper/scaling-local-self-attention-for-parameter
Scaling Local Self-Attention for Parameter Efficient Visual Backbones