Primer
Primer
**Primer** is a [Transformer](https://paperswithcode.com/methods/category/transformers)-based architecture that improves upon the [Transformer](https://paperswithcode.com/method/transformer) architecture with two improvements found through [neural architecture search](https://paperswithcode.com/methods/category/neural-architecture-search): [squared RELU activations](https://paperswithcode.com/method/squared-relu) in the feedforward block, and [depthwise convolutions]() added to the attention multi-head projections: resulting in a new module called [Multi-DConv-Head-Attention](https://paperswithcode.com/method/multi-dconv-head-attention).
Primer: Searching for Efficient Transformers for Language Modeling
https://paperswithcode.com/paper/primer-searching-for-efficient-transformers
Primer: Searching for Efficient Transformers for Language Modeling