Feedback Transformer
Feedback Transformer
A **Feedback Transformer** is a type of sequential transformer that exposes all previous representations to all future representations, meaning the lowest representation of the current timestep is formed from the highest-level abstract representation of the past. This feedback nature allows this architecture to perform recursive computation, building stronger representations iteratively upon previous states. To achieve this, the self-attention mechanism of the standard [Transformer](https://paperswithcode.com/method/transformer) is modified so it attends to higher level representations rather than lower ones.
Addressing Some Limitations of Transformers with Feedback Memory
https://paperswithcode.com/paper/accessing-higher-level-representations-in
Addressing Some Limitations of Transformers with Feedback Memory