RealFormer
RealFormer
**RealFormer** is a type of [Transformer](https://paperswithcode.com/methods/category/transformers) based on the idea of [residual](https://paperswithcode.com/method/residual-connection) attention. It adds skip edges to the backbone [Transformer](https://paperswithcode.com/method/transformer) to create multiple direct paths, one for each type of attention module. It adds no parameters or hyper-parameters. Specifically, RealFormer uses a Post-[LN](https://paperswithcode.com/method/layer-normalization) style Transformer as backbone and adds skip edges to connect [Multi-Head Attention](https://paperswithcode.com/method/multi-head-attention) modules in adjacent layers.
RealFormer: Transformer Likes Residual Attention
https://paperswithcode.com/paper/informer-transformer-likes-informed-attention
RealFormer: Transformer Likes Residual Attention