Routing Transformer
Routing Transformer
The **Routing Transformer** is a [Transformer](https://paperswithcode.com/method/transformer) that endows self-attention with a sparse routing module based on online k-means. Each attention module considers a clustering of the space: the current timestep only attends to context belonging to the same cluster. In other word, the current time-step query is routed to a limited number of context through its cluster assignment.
Efficient Content-Based Sparse Attention with Routing Transformers
https://paperswithcode.com/paper/efficient-content-based-sparse-attention-with-1
Efficient Content-Based Sparse Attention with Routing Transformers