Transformer-XL
Transformer-XL
**Transformer-XL** (meaning extra long) is a [Transformer](https://paperswithcode.com/method/transformer) architecture that introduces the notion of recurrence to the deep self-attention network. Instead of computing the hidden states from scratch for each new segment, Transformer-XL reuses the hidden states obtained in previous segments. The reused hidden states serve as memory for the current segment, which builds up a recurrent connection between the segments. As a result, modeling very long-term dependency becomes possible because information can be propagated through the recurrent connections. As an additional contribution, the Transformer-XL uses a new relative positional encoding formulation that generalizes to attention lengths longer than the one observed during training.
Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
https://paperswithcode.com/paper/transformer-xl-attentive-language-models
Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context