NLP Lineage

27 August 2021 · Language modelling

ALiBi

Train Short, Test Long: Attention with Linear Biases

O. Press, N. Smith, M. Lewis

Date
27 August 2021
What the date marks
arXiv submission
Kind
Architectural idea
Organisation
U. Washington / FAIR / AI2
Access
MIT
Other dates
ICLR 2022
Threads
In the guided tour
Chapter 10. Reading more than you remember
Learning phases
1. Transformer architecture

A milestone: 1 later milestone descends directly from it, and its descendants reach 3 of the 10 threads. How levels are decided

Read the primary source (arxiv.org)

Full summary in progress. Until then, this page shows the milestone's record from the research data.

What it changed

Before it

Models trained at one context length degraded sharply when run on longer inputs.

The problem

Position embeddings do not generalise past the lengths they were trained on.

What it introduced

Drop position embeddings and instead subtract a penalty from attention scores proportional to distance.

Why it matters

It framed length extrapolation as a goal in itself and offered the main alternative to RoPE.

What modern models inherit

Length extrapolation as an explicit design criterion.

What came next

Used in BLOOM; RoPE with interpolation eventually won more adopters.

Before and after

The milestones it descends from, and the milestones that descend from it. Work developed alongside it is not counted. Weak links are left out unless you include them.

Before: where it came from

ALiBi has 10 documented ancestors across 3 chapters, reaching back to Recurrent networks in 1990. Showing its direct parents and the 4 turning points among them.

ALiBi has 10 documented ancestors across 3 chapters, reaching back to Recurrent networks in 1990. Showing all of them.

ALiBi has 11 recorded ancestors, including indirect ones, across 3 chapters, reaching back to Recurrent networks in 1990. Showing its direct parents and the 4 turning points among them.

ALiBi has 11 recorded ancestors, including indirect ones, across 3 chapters, reaching back to Recurrent networks in 1990. Showing all of them.

  1. 2021ALiBiThis milestone.
  2. 2017Transformer Turning pointALiBi built directly on thisChapter 3. Attention, then only attention
  3. 2016Layer normTransformer built directly on thisChapter 3. Attention, then only attention
  4. 2015Residual connectionsTransformer was influenced by thisChapter 3. Attention, then only attention
  5. 2015Luong attentionTransformer was influenced by thisChapter 3. Attention, then only attention
  6. 2014AdamTransformer was influenced by thisChapter 2. Words become vectors
  7. 2014Seq2seq Turning pointTransformer built directly on thisChapter 2. Words become vectors
  8. 2014Attention Turning pointTransformer built directly on thisChapter 3. Attention, then only attention
  9. 2014GRU / encoder-decoderAttention built directly on thisChapter 2. Words become vectors
  10. 1997LSTM Turning pointSeq2seq built directly on thisChapter 1. Language as probability
  11. 1990Recurrent networksLSTM was influenced by thisChapter 1. Language as probability
  1. 2021ALiBiThis milestone.
  2. 2017Transformer Turning pointALiBi built directly on thisChapter 3. Attention, then only attention
  3. 2017ConvS2STransformer was influenced by this (indirect influence)Chapter 3. Attention, then only attention
  4. 2016Layer normTransformer built directly on thisChapter 3. Attention, then only attention
  5. 2015Residual connectionsTransformer was influenced by thisChapter 3. Attention, then only attention
  6. 2015Luong attentionTransformer was influenced by thisChapter 3. Attention, then only attention
  7. 2014AdamTransformer was influenced by thisChapter 2. Words become vectors
  8. 2014Seq2seq Turning pointTransformer built directly on thisChapter 2. Words become vectors
  9. 2014Attention Turning pointTransformer built directly on thisChapter 3. Attention, then only attention
  10. 2014GRU / encoder-decoderAttention built directly on thisChapter 2. Words become vectors
  11. 1997LSTM Turning pointSeq2seq built directly on thisChapter 1. Language as probability
  12. 1990Recurrent networksLSTM was influenced by thisChapter 1. Language as probability

After: what it led to

ALiBi has 1 documented descendants across 1 chapter, reaching forward to BLOOM in 2022. Showing all of them.

ALiBi has 1 documented descendants across 1 chapter, reaching forward to BLOOM in 2022. Showing all of them.

ALiBi has 1 recorded descendants, including indirect ones, across 1 chapter, reaching forward to BLOOM in 2022. Showing all of them.

ALiBi has 1 recorded descendants, including indirect ones, across 1 chapter, reaching forward to BLOOM in 2022. Showing all of them.

  1. 2021ALiBiThis milestone.
  2. 2022BLOOMbuilt directly on ALiBiChapter 8. The weights get out
  1. 2021ALiBiThis milestone.
  2. 2022BLOOMbuilt directly on ALiBiChapter 8. The weights get out

Direct links and their sources

Where it came from

Built directly on

Developed alongside

What it led to

Built on directly by

Terms