Rotary Embeddings
Rotary Position Embedding
**Rotary Position Embedding**, or **RoPE**, is a type of position embedding which encodes absolute positional information with rotation matrix and naturally incorporates explicit relative position dependency in self-attention formulation. Notably, RoPE comes with valuable properties such as flexibility of being expand to any sequence lengths, decaying inter-token dependency with increasing relative distances, and capability of equipping the linear self-attention with relative position encoding.
RoFormer: Enhanced Transformer with Rotary Position Embedding
https://paperswithcode.com/paper/roformer-enhanced-transformer-with-rotary
RoFormer: Enhanced Transformer with Rotary Position Embedding