NLP Lineage

28 July 2020 · Language modelling

BigBird

Big Bird: Transformers for Longer Sequences

M. Zaheer, G. Guruganesh, A. Dubey, et al. (11 authors)

Date
28 July 2020
What the date marks
arXiv submission
Kind
Architectural idea
Organisation
Google Research
Access
Apache 2.0
Other dates
NeurIPS 2020
Threads
In the guided tour
Chapter 10. Reading more than you remember
Learning phases
1. Transformer architecture

A milestone: 1 later milestone descends directly from it, and its descendants reach 4 of the 10 threads. How levels are decided

Read the primary source (arxiv.org)

Full summary in progress. Until then, this page shows the milestone's record from the research data.

What it changed

Before it

Sparse attention patterns were designed by hand with no guarantee they preserved expressiveness.

The problem

Does a sparse pattern still let information flow between any two positions?

What it introduced

Global, window and random attention combined, with a proof that the result is a universal approximator of full attention.

Why it matters

It gave the sparse-attention family a theoretical footing as well as a practical one.

What modern models inherit

The vocabulary of global, local and random attention patterns.

What came next

Superseded in practice by exact methods once FlashAttention made full attention cheap.

Before and after

The milestones it descends from, and the milestones that descend from it. Work developed alongside it is not counted. Weak links are left out unless you include them.

Before: where it came from

BigBird has 11 documented ancestors across 4 chapters, reaching back to Recurrent networks in 1990. Showing its direct parents and the 5 turning points among them.

BigBird has 11 documented ancestors across 4 chapters, reaching back to Recurrent networks in 1990. Showing all of them.

BigBird has 13 recorded ancestors, including indirect ones, across 4 chapters, reaching back to Recurrent networks in 1990. Showing its direct parents and the 5 turning points among them.

BigBird has 13 recorded ancestors, including indirect ones, across 4 chapters, reaching back to Recurrent networks in 1990. Showing all of them.

  1. 2020BigBirdThis milestone.
  2. 2019Sparse Transformer Turning pointBigBird was influenced by thisChapter 4. Pretrain once, use everywhere
  3. 2017Transformer Turning pointBigBird built directly on thisChapter 3. Attention, then only attention
  4. 2016Layer normTransformer built directly on thisChapter 3. Attention, then only attention
  5. 2015Residual connectionsTransformer was influenced by thisChapter 3. Attention, then only attention
  6. 2015Luong attentionTransformer was influenced by thisChapter 3. Attention, then only attention
  7. 2014AdamTransformer was influenced by thisChapter 2. Words become vectors
  8. 2014Seq2seq Turning pointTransformer built directly on thisChapter 2. Words become vectors
  9. 2014Attention Turning pointTransformer built directly on thisChapter 3. Attention, then only attention
  10. 2014GRU / encoder-decoderAttention built directly on thisChapter 2. Words become vectors
  11. 1997LSTM Turning pointSeq2seq built directly on thisChapter 1. Language as probability
  12. 1990Recurrent networksLSTM was influenced by thisChapter 1. Language as probability
  1. 2020BigBirdThis milestone.
  2. 2019Sparse Transformer Turning pointBigBird was influenced by thisChapter 4. Pretrain once, use everywhere
  3. 2019Transformer-XLSparse Transformer was influenced by this (indirect influence)Chapter 4. Pretrain once, use everywhere
  4. 2017Transformer Turning pointBigBird built directly on thisChapter 3. Attention, then only attention
  5. 2017ConvS2STransformer was influenced by this (indirect influence)Chapter 3. Attention, then only attention
  6. 2016Layer normTransformer built directly on thisChapter 3. Attention, then only attention
  7. 2015Residual connectionsTransformer was influenced by thisChapter 3. Attention, then only attention
  8. 2015Luong attentionTransformer was influenced by thisChapter 3. Attention, then only attention
  9. 2014AdamTransformer was influenced by thisChapter 2. Words become vectors
  10. 2014Seq2seq Turning pointTransformer built directly on thisChapter 2. Words become vectors
  11. 2014Attention Turning pointTransformer built directly on thisChapter 3. Attention, then only attention
  12. 2014GRU / encoder-decoderAttention built directly on thisChapter 2. Words become vectors
  13. 1997LSTM Turning pointSeq2seq built directly on thisChapter 1. Language as probability
  14. 1990Recurrent networksLSTM was influenced by thisChapter 1. Language as probability

After: what it led to

BigBird has 6 documented descendants across 2 chapters, reaching forward to FlashAttention-3 in 2024. Showing its direct descendants.

BigBird has 6 documented descendants across 2 chapters, reaching forward to FlashAttention-3 in 2024. Showing all of them.

BigBird has 8 recorded descendants, including indirect ones, across 2 chapters, reaching forward to Muse Spark in 2026. Showing its direct descendants.

BigBird has 8 recorded descendants, including indirect ones, across 2 chapters, reaching forward to Muse Spark in 2026. Showing all of them.

  1. 2020BigBirdThis milestone.
  2. 2022FlashAttentionsucceeded BigBirdChapter 9. Making it run anywhere
  3. 2023Mistral 7Bwas influenced by FlashAttentionChapter 8. The weights get out
  4. 2023Mambawas influenced by FlashAttentionChapter 9. Making it run anywhere
  5. 2023Mixtral 8x7Bbuilt directly on Mistral 7BChapter 8. The weights get out
  6. 2024Gemmawas influenced by Mistral 7BChapter 8. The weights get out
  7. 2024FlashAttention-3built directly on FlashAttentionChapter 9. Making it run anywhere
  1. 2020BigBirdThis milestone.
  2. 2022FlashAttentionsucceeded BigBirdChapter 9. Making it run anywhere
  3. 2023Mistral 7Bwas influenced by FlashAttentionChapter 8. The weights get out
  4. 2023Mambawas influenced by FlashAttentionChapter 9. Making it run anywhere
  5. 2023Mixtral 8x7Bbuilt directly on Mistral 7BChapter 8. The weights get out
  6. 2024Gemmawas influenced by Mistral 7BChapter 8. The weights get out
  7. 2024FlashAttention-3built directly on FlashAttentionChapter 9. Making it run anywhere
  8. 2025Llama 4was influenced by Mixtral 8x7B (indirect influence)Chapter 8. The weights get out
  9. 2026Muse Sparksucceeded Llama 4Chapter 8. The weights get out

Direct links and their sources

Where it came from

Built directly on

Influenced by

What it led to

Replaced by

Terms