NLP Lineage

10 April 2020 · Language modelling

Longformer

Longformer: The Long-Document Transformer

I. Beltagy, M. Peters, A. Cohan

Date
10 April 2020
What the date marks
arXiv submission
Kind
Architectural idea
Organisation
Allen Institute for AI
Access
Apache 2.0
Threads
In the guided tour
Chapter 10. Reading more than you remember
Learning phases
1. Transformer architecture

A milestone: 1 later milestone descends directly from it, and its descendants reach 4 of the 10 threads. How levels are decided

Read the primary source (arxiv.org)

Full summary in progress. Until then, this page shows the milestone's record from the research data.

What it changed

Before it

Transformers were capped around 512 tokens by quadratic attention.

The problem

Documents are longer than 512 tokens, so they had to be chunked and processed separately.

What it introduced

A sliding local window plus a few global positions, giving attention that scales linearly with length.

Why it matters

One of the first practical long-context designs, and the sliding-window pattern Mistral later revived.

What modern models inherit

Windowed attention as a standard long-context tool.

What came next

BigBird added random attention; Mistral used sliding windows in 2023.

Before and after

The milestones it descends from, and the milestones that descend from it. Work developed alongside it is not counted. Weak links are left out unless you include them.

Before: where it came from

Longformer has 10 documented ancestors across 3 chapters, reaching back to Recurrent networks in 1990. Showing its direct parents and the 4 turning points among them.

Longformer has 10 documented ancestors across 3 chapters, reaching back to Recurrent networks in 1990. Showing all of them.

Longformer has 11 recorded ancestors, including indirect ones, across 3 chapters, reaching back to Recurrent networks in 1990. Showing its direct parents and the 4 turning points among them.

Longformer has 11 recorded ancestors, including indirect ones, across 3 chapters, reaching back to Recurrent networks in 1990. Showing all of them.

  1. 2020LongformerThis milestone.
  2. 2017Transformer Turning pointLongformer built directly on thisChapter 3. Attention, then only attention
  3. 2016Layer normTransformer built directly on thisChapter 3. Attention, then only attention
  4. 2015Residual connectionsTransformer was influenced by thisChapter 3. Attention, then only attention
  5. 2015Luong attentionTransformer was influenced by thisChapter 3. Attention, then only attention
  6. 2014AdamTransformer was influenced by thisChapter 2. Words become vectors
  7. 2014Seq2seq Turning pointTransformer built directly on thisChapter 2. Words become vectors
  8. 2014Attention Turning pointTransformer built directly on thisChapter 3. Attention, then only attention
  9. 2014GRU / encoder-decoderAttention built directly on thisChapter 2. Words become vectors
  10. 1997LSTM Turning pointSeq2seq built directly on thisChapter 1. Language as probability
  11. 1990Recurrent networksLSTM was influenced by thisChapter 1. Language as probability
  1. 2020LongformerThis milestone.
  2. 2017Transformer Turning pointLongformer built directly on thisChapter 3. Attention, then only attention
  3. 2017ConvS2STransformer was influenced by this (indirect influence)Chapter 3. Attention, then only attention
  4. 2016Layer normTransformer built directly on thisChapter 3. Attention, then only attention
  5. 2015Residual connectionsTransformer was influenced by thisChapter 3. Attention, then only attention
  6. 2015Luong attentionTransformer was influenced by thisChapter 3. Attention, then only attention
  7. 2014AdamTransformer was influenced by thisChapter 2. Words become vectors
  8. 2014Seq2seq Turning pointTransformer built directly on thisChapter 2. Words become vectors
  9. 2014Attention Turning pointTransformer built directly on thisChapter 3. Attention, then only attention
  10. 2014GRU / encoder-decoderAttention built directly on thisChapter 2. Words become vectors
  11. 1997LSTM Turning pointSeq2seq built directly on thisChapter 1. Language as probability
  12. 1990Recurrent networksLSTM was influenced by thisChapter 1. Language as probability

After: what it led to

Longformer has 3 documented descendants across 1 chapter, reaching forward to Gemma in 2024. Showing its direct descendants.

Longformer has 3 documented descendants across 1 chapter, reaching forward to Gemma in 2024. Showing all of them.

Longformer has 5 recorded descendants, including indirect ones, across 1 chapter, reaching forward to Muse Spark in 2026. Showing its direct descendants.

Longformer has 5 recorded descendants, including indirect ones, across 1 chapter, reaching forward to Muse Spark in 2026. Showing all of them.

  1. 2020LongformerThis milestone.
  2. 2023Mistral 7Bwas influenced by LongformerChapter 8. The weights get out
  3. 2023Mixtral 8x7Bbuilt directly on Mistral 7BChapter 8. The weights get out
  4. 2024Gemmawas influenced by Mistral 7BChapter 8. The weights get out
  1. 2020LongformerThis milestone.
  2. 2023Mistral 7Bwas influenced by LongformerChapter 8. The weights get out
  3. 2023Mixtral 8x7Bbuilt directly on Mistral 7BChapter 8. The weights get out
  4. 2024Gemmawas influenced by Mistral 7BChapter 8. The weights get out
  5. 2025Llama 4was influenced by Mixtral 8x7B (indirect influence)Chapter 8. The weights get out
  6. 2026Muse Sparksucceeded Llama 4Chapter 8. The weights get out

Direct links and their sources

Where it came from

Built directly on

What it led to

Influenced

Terms