Attention with linear biases (ALiBi)

A way to give attention a sense of word order without position vectors: each attention score is lowered by an amount that grows with the distance between the two tokens.