RoFormer: Enhanced Transformer with Rotary Position Embedding

The paper proposes Rotary Position Embedding (RoPE), a method for integrating positional information into transformer-based language models. RoPE encodes the absolute position of a token using a rotation matrix and, at the same time, incorporates explicit relative-position dependency directly into the self-attention formulation. This design gives the method several valuable properties, including flexibility with respect to sequence length, inter-token dependency that decays with increasing relative distance, and the ability to equip linear self-attention with relative position encoding.

The authors evaluate the enhanced transformer, called RoFormer, on various long-text classification benchmark datasets, where it consistently overcomes its alternatives. They also provide a theoretical analysis to explain some of the experimental results. RoFormer has since been integrated into Hugging Face, reflecting its practical adoption as a position-encoding technique for transformers.