RoFormer: Enhanced Transformer with Rotary Position Embedding
J. Su, Y. Lu, S. Pan, A. Murtadha, B. Wen, Y. Liu
- Date
- 20 April 2021
- What the date marks
- arXiv submission
- Kind
- Architectural idea
- Organisation
- Zhuiyi Technology
- Access
- Apache 2.0
- Other dates
- Neurocomputing 2024
- Learning phases
- 1. Transformer architecture
A turning point: 4 later milestones descend directly from it, and its descendants reach 9 of the 10 threads. How levels are decided
Drafted with Claude from the paper on 15 October 2026; reviewed by Zeeshan Arif. Read from arxiv.org (arXiv version 5 (November 2023); the paper was later published in the journal Neurocomputing (2024)).
What it changed
It gave Transformers a new way to know the order of tokens: rotate each query and key vector by an angle that depends on the token’s position. When two tokens are compared, only the difference between their positions is left. This method, rotary position embedding (RoPE), was taken up by many later large language models, such as LLaMA.
The problem before it
Attention on its own has no sense of word order. So a Transformer must add position information in some way. There were two main families:
- Absolute positions. Add a vector for each position to each token’s vector. The vector is either a fixed pattern of sine waves (the original Transformer) or learned (BERT and GPT).
- Relative positions. Change the attention score so that it depends on the distance between two tokens (Transformer-XL, T5 and others).
The authors point out that almost all of these methods add the position information to the token representation or to the attention score. They argue that this does not work with “linear attention”, a cheaper form of attention that never computes the full table of scores.
The idea
The mechanism. Instead of adding a position vector, multiply by a rotation.
- In two dimensions. Think of a query or key vector as an arrow on a flat plane. For a token at position
m, turn the arrow by the angle m × θ. When you compare a query at position m with a key at position n, their dot product depends only on the angle between them. That angle is (m − n) × θ, so the score depends only on the relative position.
- In many dimensions. Split each vector into pairs of numbers, and rotate each pair by its own angle. The first pairs turn fast, the last pairs turn slowly. The speeds follow the same pattern of frequencies as the original Transformer’s sine waves.
- Only queries and keys are rotated, not the values. A rotation does not change the length of a vector.
The authors list three useful properties:
- Flexible length. Positions are computed from a formula, not looked up in a learned table of fixed size.
- Long-range decay. They show that an upper bound on the attention score falls as the distance between two tokens grows, which fits the idea that distant words are usually less related.
- Linear attention. Because the rotation keeps vector lengths the same, it can be used with linear attention.
The notation. The position-encoded query is f_q(x_m, m) = R_(Θ,m) W_q x_m, and the key is the same with W_k and position n. R_(Θ,m) is a block-diagonal matrix of 2 × 2 rotations by angles m θ_i, with θ_i = 10000^(−2(i−1)/d). Then q_m^T k_n = x_m^T W_q^T R_(Θ,n−m) W_k x_n, which depends on position only through n − m. (Section 3.3 writes the frequencies as 10000^(−2i/d), a slightly different indexing.)
What it showed
Reported in the paper. We read version 5 of the arXiv paper. The experiments are small, and all were run on two cloud servers with 4 GPUs each.
- Translation (WMT 2014 English to German). 27.5 BLEU, against 27.3 for a base Transformer trained the same way.
- Pretraining like BERT. With RoPE in place of BERT’s position embedding, the training loss fell faster. The paper reports this as loss curves, not test scores.
- GLUE fine-tuning: mixed results. RoPE was better on 3 of the 6 tasks (MRPC, STS-B and QQP) and worse on the other 3 (SST-2, QNLI and MNLI). On MNLI it was about 4 points lower. The paper’s text reports only the 3 gains.
- Linear attention. Adding RoPE to a Performer (a linear-attention model) on Enwik8 gave faster training and lower training loss.
- Long Chinese texts. On a legal case-matching task with long documents, a Chinese RoFormer reached 69.79% test accuracy with inputs of 1,024 characters. Limited to 512 characters, it scored 68.29%, and the WoBERT model scored 68.10%.
Limits, and what came later
The limits come from the paper, and from its own results:
- No explanation for the speed. The authors say they cannot explain why RoPE trains faster than other position methods.
- No explanation for long texts. Other methods also have long-range decay. The authors cannot say why RoPE does better on long texts.
- Small experiments. The tests use base-sized models and short training runs, and most results are training-loss curves or single runs.
- Mixed fine-tuning results. RoPE did not beat BERT on half the GLUE tasks tested.
- An inconsistency. The paper says it replaces BERT’s “sinusoidal” position encoding, but its own section 2.2 says BERT uses learned position vectors.
Even so, RoPE spread widely. LLaMA and Qwen2 use it. Position interpolation rescales its positions to extend the context length. DeepSeek-V2 adapted it for its own attention design. ALiBi, a different position method from the same year, compared against it on longer inputs.
What to know first
- Self-attention in the Transformer, and its sine-wave position encoding.
- Dot products, and how rotating two vectors by the same angle leaves their dot product unchanged.
- Basic complex numbers help with section 3.4, but are not needed for the idea.
How to read it
The paper is 14 pages, with 12 pages of main text.
- Read section 2 for the existing position methods. Equations 3 to 10 can be skimmed.
- Read section 3.2 with Figure 1. The two-dimensional case in 3.2.1 is the whole idea.
- Read section 3.3 for the three properties.
- Look at Tables 1, 2 and 5 for the results, and read section 4.5.5 for the authors’ limits.
Section 3.4 has the full derivation and is optional. Expect about 1.5 hours for the steps above (an estimate).
Links