Content-based Attention
Content-based Attention
**Content-based attention** is an attention mechanism based on cosine similarity:

$$f_{att}\left(\textbf{h}_{i}, \textbf{s}\_{j}\right) = \cos\left[\textbf{h}\_{i};\textbf{s}\_{j}\right] $$

It was utilised in [Neural Turing Machines](https://paperswithcode.com/method/neural-turing-machine) as part of the Addressing Mechanism.

We produce a normalized attention weighting by taking a [softmax](https://paperswithcode.com/method/softmax) over these attention alignment scores.
Neural Turing Machines
https://paperswithcode.com/paper/neural-turing-machines
Neural Turing Machines