NLP Lineage

15 June 2022 · Scale and data

Emergent abilities

Emergent Abilities of Large Language Models

J. Wei, Y. Tay, R. Bommasani, et al. (16 authors)

Date
15 June 2022
What the date marks
arXiv submission
Kind
Research paper
Organisation
Google / Stanford / DeepMind
Access
Not applicable
Other dates
TMLR 2022
Threads
In the guided tour
Chapter 5. Scale becomes a science
Learning phases
2. How LLMs are trained

A milestone: 1 later milestone descends directly from it, and its descendants reach 2 of the 10 threads. How levels are decided

Read the primary source (arxiv.org)

Full summary in progress. Until then, this page shows the milestone's record from the research data.

What it changed

Before it

Scaling laws described smooth, predictable improvement in loss.

The problem

Some abilities appeared absent at small scale and present at large scale, which smooth curves do not explain.

What it introduced

A catalogue of tasks where performance jumps sharply past a scale threshold, framed as emergence.

Why it matters

It shaped how the field talked about capability and risk for two years, and provoked a serious rebuttal in 2023.

What modern models inherit

An unresolved argument about whether capability arrives smoothly or in steps.

What came next

Schaeffer's mirage paper argued the jumps come from discontinuous metrics, not the models.

Before and after

The milestones it descends from, and the milestones that descend from it. Work developed alongside it is not counted. Weak links are left out unless you include them.

Before: where it came from

Emergent abilities has 24 documented ancestors across 6 chapters, reaching back to Recurrent networks in 1990. Showing its direct parents and the 12 turning points among them.

Emergent abilities has 24 documented ancestors across 6 chapters, reaching back to Recurrent networks in 1990. Showing all of them.

Emergent abilities has 33 recorded ancestors, including indirect ones, across 6 chapters, reaching back to Recurrent networks in 1990. Showing its direct parents and the 12 turning points among them.

Emergent abilities has 33 recorded ancestors, including indirect ones, across 6 chapters, reaching back to Recurrent networks in 1990. Showing all of them.

  1. 2022Emergent abilitiesThis milestone.
  2. 2022PaLMEmergent abilities was influenced by thisChapter 5. Scale becomes a science
  3. 2022Chain of thought Turning pointEmergent abilities built directly on thisChapter 12. Thinking longer
  4. 2020MMLUEmergent abilities was influenced by thisChapter 5. Scale becomes a science
  5. 2020GShard Turning pointPaLM was influenced by thisChapter 5. Scale becomes a science
  6. 2020GPT-3 Turning pointChain of thought was influenced by thisChapter 5. Scale becomes a science
  7. 2020Scaling laws Turning pointEmergent abilities built directly on thisChapter 5. Scale becomes a science
  8. 2019Sparse Transformer Turning pointGPT-3 built directly on thisChapter 4. Pretrain once, use everywhere
  9. 2019GPT-2 Turning pointScaling laws was influenced by thisChapter 4. Pretrain once, use everywhere
  10. 2019WebTextGPT-2 was influenced by thisChapter 4. Pretrain once, use everywhere
  11. 2018GPT-1GPT-2 built directly on thisChapter 4. Pretrain once, use everywhere
  12. 2017Transformer Turning pointScaling laws built directly on thisChapter 3. Attention, then only attention
  13. 2017Sparse MoE Turning pointGShard built directly on thisChapter 3. Attention, then only attention
  14. 2016Layer normTransformer built directly on this
    GPT-2 built directly on this    Chapter 3. Attention, then only attention
  15. 2015Residual connectionsTransformer was influenced by thisChapter 3. Attention, then only attention
  16. 2015LM pretraining Turning pointGPT-1 was influenced by thisChapter 2. Words become vectors
  17. 2015BPE tokenizationGPT-2 built directly on thisChapter 2. Words become vectors
  18. 2015Luong attentionTransformer was influenced by thisChapter 3. Attention, then only attention
  19. 2014AdamTransformer was influenced by thisChapter 2. Words become vectors
  20. 2014Seq2seq Turning pointTransformer built directly on thisChapter 2. Words become vectors
  21. 2014Attention Turning pointTransformer built directly on thisChapter 3. Attention, then only attention
  22. 2014GRU / encoder-decoderAttention built directly on thisChapter 2. Words become vectors
  23. 1997LSTM Turning pointSeq2seq built directly on this
    Sparse MoE built directly on this    Chapter 1. Language as probability
  24. 1991Mixture of expertsSparse MoE built directly on thisChapter 1. Language as probability
  25. 1990Recurrent networksLSTM was influenced by thisChapter 1. Language as probability
  1. 2022Emergent abilitiesThis milestone.
  2. 2022PaLMEmergent abilities was influenced by thisChapter 5. Scale becomes a science
  3. 2022Chain of thought Turning pointEmergent abilities built directly on thisChapter 12. Thinking longer
  4. 2020MMLUEmergent abilities was influenced by thisChapter 5. Scale becomes a science
  5. 2020GShard Turning pointPaLM was influenced by thisChapter 5. Scale becomes a science
  6. 2020GPT-3 Turning pointChain of thought was influenced by thisChapter 5. Scale becomes a science
  7. 2020Scaling laws Turning pointEmergent abilities built directly on thisChapter 5. Scale becomes a science
  8. 2019Sparse Transformer Turning pointGPT-3 built directly on thisChapter 4. Pretrain once, use everywhere
  9. 2019GPT-2 Turning pointScaling laws was influenced by thisChapter 4. Pretrain once, use everywhere
  10. 2019WebTextGPT-2 was influenced by thisChapter 4. Pretrain once, use everywhere
  11. 2019Transformer-XLSparse Transformer was influenced by this (indirect influence)Chapter 4. Pretrain once, use everywhere
  12. 2018GPT-1GPT-2 built directly on thisChapter 4. Pretrain once, use everywhere
  13. 2018ELMoGPT-1 was influenced by this (indirect influence)Chapter 4. Pretrain once, use everywhere
  14. 2018ULMFiTGPT-1 was influenced by this (indirect influence)Chapter 4. Pretrain once, use everywhere
  15. 2017CoVeELMo was influenced by thisChapter 3. Attention, then only attention
  16. 2017Transformer Turning pointScaling laws built directly on thisChapter 3. Attention, then only attention
  17. 2017ConvS2STransformer was influenced by this (indirect influence)Chapter 3. Attention, then only attention
  18. 2017Sparse MoE Turning pointGShard built directly on thisChapter 3. Attention, then only attention
  19. 2016Layer normTransformer built directly on this
    GPT-2 built directly on this    Chapter 3. Attention, then only attention
  20. 2015Residual connectionsTransformer was influenced by thisChapter 3. Attention, then only attention
  21. 2015LM pretraining Turning pointGPT-1 was influenced by thisChapter 2. Words become vectors
  22. 2015BPE tokenizationGPT-2 built directly on thisChapter 2. Words become vectors
  23. 2015Luong attentionTransformer was influenced by thisChapter 3. Attention, then only attention
  24. 2014AdamTransformer was influenced by thisChapter 2. Words become vectors
  25. 2014GloVeELMo built directly on thisChapter 2. Words become vectors
  26. 2014Seq2seq Turning pointTransformer built directly on thisChapter 2. Words become vectors
  27. 2014Attention Turning pointTransformer built directly on thisChapter 3. Attention, then only attention
  28. 2014GRU / encoder-decoderAttention built directly on thisChapter 2. Words become vectors
  29. 2013word2vecELMo was influenced by thisChapter 2. Words become vectors
  30. 2011NLP from scratchELMo was influenced by thisChapter 1. Language as probability
  31. 2003Neural LMNLP from scratch was influenced by this
    word2vec was influenced by this    Chapter 1. Language as probability
  32. 1997LSTM Turning pointSeq2seq built directly on this
    Sparse MoE built directly on this    Chapter 1. Language as probability
  33. 1991Mixture of expertsSparse MoE built directly on thisChapter 1. Language as probability
  34. 1990Recurrent networksLSTM was influenced by thisChapter 1. Language as probability

After: what it led to

Emergent abilities has 1 documented descendants across 1 chapter, reaching forward to Emergence mirage in 2023. Showing all of them.

Emergent abilities has 1 documented descendants across 1 chapter, reaching forward to Emergence mirage in 2023. Showing all of them.

Emergent abilities has 1 recorded descendants, including indirect ones, across 1 chapter, reaching forward to Emergence mirage in 2023. Showing all of them.

Emergent abilities has 1 recorded descendants, including indirect ones, across 1 chapter, reaching forward to Emergence mirage in 2023. Showing all of them.

  1. 2022Emergent abilitiesThis milestone.
  2. 2023Emergence miragesucceeded Emergent abilitiesChapter 5. Scale becomes a science
  1. 2022Emergent abilitiesThis milestone.
  2. 2023Emergence miragesucceeded Emergent abilitiesChapter 5. Scale becomes a science

Direct links and their sources

Where it came from

Built directly on

Influenced by

What it led to

Replaced by

Terms