Rotary position embedding (RoPE)

A way to give attention a sense of word order by rotating each query and key vector by an angle that depends on its position. When two tokens are compared, only the distance between them matters.