NLP Lineage

8 December 2021 · Language modelling

Gopher

Scaling Language Models: Methods, Analysis and Insights from Training Gopher

J. Rae, S. Borgeaud, T. Cai, et al. (80 authors)

Date
8 December 2021
What the date marks
arXiv submission
Kind
Model
Organisation
DeepMind
Access
Not applicable
Threads
In the guided tour
Chapter 5. Scale becomes a science
Learning phases
2. How LLMs are trained

A milestone: 1 later milestone descends directly from it, and its descendants reach 9 of the 10 threads. How levels are decided

Read the primary source (arxiv.org)

Full summary in progress. Until then, this page shows the milestone's record from the research data.

What it changed

Before it

GPT-3 was the reference point, with limited public analysis of what scale does.

The problem

Which capabilities improve with scale, and which do not?

What it introduced

A 280B model evaluated across 152 tasks, showing gains concentrated in knowledge tasks and much weaker in reasoning.

Why it matters

It supplied the careful evaluation that scaling claims lacked, and the compute budget Chinchilla reanalysed.

What modern models inherit

The habit of broad, task-by-task evaluation of scale.

What came next

Chinchilla used the same budget to show Gopher was badly undertrained.

Before and after

The milestones it descends from, and the milestones that descend from it. Work developed alongside it is not counted. Weak links are left out unless you include them.

Before: where it came from

No documented ancestors are recorded for Gopher without weak links.

No documented ancestors are recorded for Gopher without weak links.

No recorded ancestors are recorded for Gopher.

No recorded ancestors are recorded for Gopher.

After: what it led to

Gopher has 25 documented descendants across 6 chapters, reaching forward to Qwen3.8 in 2026. Showing its direct descendants and the 3 turning points among them.

Gopher has 25 documented descendants across 6 chapters, reaching forward to Qwen3.8 in 2026. Showing all of them.

Gopher has 28 recorded descendants, including indirect ones, across 6 chapters, reaching forward to Qwen3.8 in 2026. Showing its direct descendants and the 3 turning points among them.

Gopher has 28 recorded descendants, including indirect ones, across 6 chapters, reaching forward to Qwen3.8 in 2026. Showing all of them.

  1. 2021GopherThis milestone.
  2. 2022Chinchilla Turning pointbuilt directly on GopherChapter 5. Scale becomes a science
  3. 2023LLaMA Turning pointbuilt directly on ChinchillaChapter 8. The weights get out
  4. 2023llama.cppwas influenced by LLaMAChapter 9. Making it run anywhere
  5. 2023Alpacabuilt directly on LLaMAChapter 8. The weights get out
  6. 2023LLaVAbuilt directly on LLaMAChapter 11. Seeing and hearing
  7. 2023QLoRAbuilt directly on LLaMAChapter 9. Making it run anywhere
  8. 2023Position interpolationbuilt directly on LLaMAChapter 10. Reading more than you remember
  9. 2023Llama 2 Turning pointbuilt directly on LLaMAChapter 8. The weights get out
  10. 2024DeepSeek LLMwas influenced by ChinchillaChapter 8. The weights get out
  11. 2024OLMowas influenced by Llama 2Chapter 8. The weights get out
  12. 2024GRPObuilt directly on DeepSeek LLMChapter 12. Thinking longer
  13. 2024Gemmawas influenced by Llama 2Chapter 8. The weights get out
  14. 2024Llama 3was influenced by ChinchillaChapter 8. The weights get out
  15. 2024Llama 3.1 405Bwas influenced by ChinchillaChapter 8. The weights get out
  16. 2024DeepSeek-V3was influenced by Llama 3.1 405BChapter 8. The weights get out
  17. 2025DeepSeek-R1built directly on GRPO
    built directly on DeepSeek-V3    Chapter 12. Thinking longer
  18. 2025Llama 4succeeded Llama 3.1 405BChapter 8. The weights get out
  19. 2025Qwen3was influenced by DeepSeek-R1Chapter 8. The weights get out
  20. 2025Kimi K2was influenced by GRPOChapter 8. The weights get out
  21. 2025DeepSeek-V3.2built directly on DeepSeek-V3Chapter 10. Reading more than you remember
  22. 2026Qwen3.5built directly on Qwen3Chapter 8. The weights get out
  23. 2026Muse Sparksucceeded Llama 4Chapter 8. The weights get out
  24. 2026DeepSeek-V4was influenced by DeepSeek-V3Chapter 8. The weights get out
  25. 2026Kimi K3built directly on Kimi K2Chapter 8. The weights get out
  26. 2026Qwen3.8built directly on Qwen3.5Chapter 8. The weights get out
  1. 2021GopherThis milestone.
  2. 2022Chinchilla Turning pointbuilt directly on GopherChapter 5. Scale becomes a science
  3. 2023LLaMA Turning pointbuilt directly on ChinchillaChapter 8. The weights get out
  4. 2023llama.cppwas influenced by LLaMAChapter 9. Making it run anywhere
  5. 2023Alpacabuilt directly on LLaMAChapter 8. The weights get out
  6. 2023LLaVAbuilt directly on LLaMAChapter 11. Seeing and hearing
  7. 2023QLoRAbuilt directly on LLaMAChapter 9. Making it run anywhere
  8. 2023Position interpolationbuilt directly on LLaMAChapter 10. Reading more than you remember
  9. 2023Llama 2 Turning pointbuilt directly on LLaMAChapter 8. The weights get out
  10. 2024DeepSeek LLMwas influenced by ChinchillaChapter 8. The weights get out
  11. 2024OLMowas influenced by Llama 2Chapter 8. The weights get out
  12. 2024GRPObuilt directly on DeepSeek LLMChapter 12. Thinking longer
  13. 2024Gemmawas influenced by Llama 2Chapter 8. The weights get out
  14. 2024Llama 3was influenced by ChinchillaChapter 8. The weights get out
  15. 2024Llama 3.1 405Bwas influenced by ChinchillaChapter 8. The weights get out
  16. 2024DeepSeek-V3was influenced by Llama 3.1 405BChapter 8. The weights get out
  17. 2025DeepSeek-R1built directly on GRPO
    built directly on DeepSeek-V3    Chapter 12. Thinking longer
  18. 2025Llama 4succeeded Llama 3.1 405BChapter 8. The weights get out
  19. 2025Qwen3was influenced by DeepSeek-R1Chapter 8. The weights get out
  20. 2025Kimi K2was influenced by DeepSeek-V3 (indirect influence)
    was influenced by GRPO    Chapter 8. The weights get out
  21. 2025gpt-osswas influenced by DeepSeek-R1 (indirect influence)Chapter 8. The weights get out
  22. 2025GLM-4.5was influenced by DeepSeek-R1 (indirect influence)Chapter 8. The weights get out
  23. 2025DeepSeek-V3.2built directly on DeepSeek-V3Chapter 10. Reading more than you remember
  24. 2026Qwen3.5built directly on Qwen3Chapter 8. The weights get out
  25. 2026Muse Sparksucceeded Llama 4Chapter 8. The weights get out
  26. 2026DeepSeek-V4was influenced by DeepSeek-V3Chapter 8. The weights get out
  27. 2026GLM-5.2built directly on GLM-4.5Chapter 8. The weights get out
  28. 2026Kimi K3built directly on Kimi K2Chapter 8. The weights get out
  29. 2026Qwen3.8built directly on Qwen3.5Chapter 8. The weights get out

Direct links and their sources

Where it came from

Developed alongside

What it led to

Built on directly by