Single-Headed Attention
Single-Headed Attention
**Single-Headed Attention** is a single-headed attention module used in the [SHA-RNN](https://paperswithcode.com/method/sha-rnn) language model. The principle design reasons for single-headedness were simplicity (avoiding running out of memory) and scepticism about the benefits of using multiple heads.
Single Headed Attention RNN: Stop Thinking With Your Head
https://paperswithcode.com/paper/single-headed-attention-rnn-stop-thinking
Single Headed Attention RNN: Stop Thinking With Your Head