NLP Lineage

14 February 2019 · Language modelling

WebText

WebText

A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever

Date
14 February 2019
What the date marks
Technical report
Kind
Dataset
Organisation
OpenAI
Access
not released
Other dates
OpenWebText community replication
Threads
In the guided tour
Chapter 4. Pretrain once, use everywhere
Learning phases
2. How LLMs are trained; 8. Training from scratch

A milestone: 1 later milestone descends directly from it, and its descendants reach 10 of the 10 threads. How levels are decided

Read the primary source (cdn.openai.com)

Full summary in progress. Until then, this page shows the milestone's record from the research data.

What it changed

Before it

Language models trained on curated corpora like Wikipedia and books.

The problem

Curated text is small; raw web crawl is enormous but mostly junk.

What it introduced

Use human curation as a quality signal: scrape only pages linked from Reddit posts above a karma threshold.

Why it matters

It introduced proxy-based quality filtering, the ancestor of every data pipeline from C4 to FineWeb.

What modern models inherit

The idea that data quality filtering is a first-class part of training.

What came next

C4, the Pile, RefinedWeb and FineWeb all refined the filtering problem.

Before and after

The milestones it descends from, and the milestones that descend from it. Work developed alongside it is not counted. Weak links are left out unless you include them.

Before: where it came from

WebText has 12 documented ancestors across 4 chapters, reaching back to Recurrent networks in 1990. Showing its direct parents and the 5 turning points among them.

WebText has 12 documented ancestors across 4 chapters, reaching back to Recurrent networks in 1990. Showing all of them.

WebText has 20 recorded ancestors, including indirect ones, across 4 chapters, reaching back to Recurrent networks in 1990. Showing its direct parents and the 5 turning points among them.

WebText has 20 recorded ancestors, including indirect ones, across 4 chapters, reaching back to Recurrent networks in 1990. Showing all of them.

  1. 2019WebTextThis milestone.
  2. 2018GPT-1WebText was influenced by thisChapter 4. Pretrain once, use everywhere
  3. 2017Transformer Turning pointGPT-1 built directly on thisChapter 3. Attention, then only attention
  4. 2016Layer normTransformer built directly on thisChapter 3. Attention, then only attention
  5. 2015Residual connectionsTransformer was influenced by thisChapter 3. Attention, then only attention
  6. 2015LM pretraining Turning pointGPT-1 was influenced by thisChapter 2. Words become vectors
  7. 2015Luong attentionTransformer was influenced by thisChapter 3. Attention, then only attention
  8. 2014AdamTransformer was influenced by thisChapter 2. Words become vectors
  9. 2014Seq2seq Turning pointTransformer built directly on thisChapter 2. Words become vectors
  10. 2014Attention Turning pointTransformer built directly on thisChapter 3. Attention, then only attention
  11. 2014GRU / encoder-decoderAttention built directly on thisChapter 2. Words become vectors
  12. 1997LSTM Turning pointSeq2seq built directly on thisChapter 1. Language as probability
  13. 1990Recurrent networksLSTM was influenced by thisChapter 1. Language as probability
  1. 2019WebTextThis milestone.
  2. 2018GPT-1WebText was influenced by thisChapter 4. Pretrain once, use everywhere
  3. 2018ELMoGPT-1 was influenced by this (indirect influence)Chapter 4. Pretrain once, use everywhere
  4. 2018ULMFiTGPT-1 was influenced by this (indirect influence)Chapter 4. Pretrain once, use everywhere
  5. 2017CoVeELMo was influenced by thisChapter 3. Attention, then only attention
  6. 2017Transformer Turning pointGPT-1 built directly on thisChapter 3. Attention, then only attention
  7. 2017ConvS2STransformer was influenced by this (indirect influence)Chapter 3. Attention, then only attention
  8. 2016Layer normTransformer built directly on thisChapter 3. Attention, then only attention
  9. 2015Residual connectionsTransformer was influenced by thisChapter 3. Attention, then only attention
  10. 2015LM pretraining Turning pointGPT-1 was influenced by thisChapter 2. Words become vectors
  11. 2015Luong attentionTransformer was influenced by thisChapter 3. Attention, then only attention
  12. 2014AdamTransformer was influenced by thisChapter 2. Words become vectors
  13. 2014GloVeELMo built directly on thisChapter 2. Words become vectors
  14. 2014Seq2seq Turning pointTransformer built directly on thisChapter 2. Words become vectors
  15. 2014Attention Turning pointTransformer built directly on thisChapter 3. Attention, then only attention
  16. 2014GRU / encoder-decoderAttention built directly on thisChapter 2. Words become vectors
  17. 2013word2vecELMo was influenced by thisChapter 2. Words become vectors
  18. 2011NLP from scratchELMo was influenced by thisChapter 1. Language as probability
  19. 2003Neural LMNLP from scratch was influenced by this
    word2vec was influenced by this    Chapter 1. Language as probability
  20. 1997LSTM Turning pointELMo built directly on thisChapter 1. Language as probability
  21. 1990Recurrent networksLSTM was influenced by thisChapter 1. Language as probability

After: what it led to

WebText has 123 documented descendants across 11 chapters, reaching forward to Claude Opus 5.5 in 2026. Showing its direct descendants and the 13 turning points among them.

WebText has 123 documented descendants across 11 chapters, reaching forward to Claude Opus 5.5 in 2026. Showing all of them.

WebText has 133 recorded descendants, including indirect ones, across 11 chapters, reaching forward to Claude Opus 5.5 in 2026. Showing its direct descendants and the 14 turning points among them.

WebText has 133 recorded descendants, including indirect ones, across 11 chapters, reaching forward to Claude Opus 5.5 in 2026. Showing all of them.

  1. 2019WebTextThis milestone.
  2. 2019GPT-2 Turning pointwas influenced by WebTextChapter 4. Pretrain once, use everywhere
  3. 2019Megatron-LMwas influenced by GPT-2Chapter 4. Pretrain once, use everywhere
  4. 2019RLHF for LMsbuilt directly on GPT-2Chapter 6. Teaching models what people want
  5. 2019T5 Turning pointwas influenced by GPT-2Chapter 4. Pretrain once, use everywhere
  6. 2020Scaling laws Turning pointwas influenced by GPT-2Chapter 5. Scale becomes a science
  7. 2020GPT-3 Turning pointbuilt directly on GPT-2Chapter 5. Scale becomes a science
  8. 2020RLHF for summarisation Turning pointbuilt directly on RLHF for LMsChapter 6. Teaching models what people want
  9. 2020The Pilewas influenced by GPT-3Chapter 5. Scale becomes a science
  10. 2021Switch Transformerbuilt directly on T5Chapter 5. Scale becomes a science
  11. 2021CLIPbuilt directly on GPT-2Chapter 11. Seeing and hearing
  12. 2021LoRAbuilt directly on GPT-3Chapter 9. Making it run anywhere
  13. 2021Copilotwas influenced by Codex / HumanEvalChapter 7. The assistant goes public
  14. 2021Codex / HumanEvalbuilt directly on GPT-3Chapter 13. Models that act
  15. 2021FLANwas influenced by GPT-3
    was influenced by T5    Chapter 6. Teaching models what people want
  16. 2021T0built directly on T5Chapter 6. Teaching models what people want
  17. 2021HHH assistantwas influenced by RLHF for summarisationChapter 6. Teaching models what people want
  18. 2021WebGPTbuilt directly on GPT-3Chapter 13. Models that act
  19. 2022MT-NLGwas influenced by Scaling laws
    built directly on Megatron-LM    Chapter 5. Scale becomes a science
  20. 2022Chain of thought Turning pointwas influenced by GPT-3Chapter 12. Thinking longer
  21. 2022InstructGPT Turning pointbuilt directly on GPT-3Chapter 6. Teaching models what people want
  22. 2022Self-consistencybuilt directly on Chain of thoughtChapter 12. Thinking longer
  23. 2022STaRwas influenced by Chain of thoughtChapter 12. Thinking longer
  24. 2022Chinchilla Turning pointbuilt directly on Scaling lawsChapter 5. Scale becomes a science
  25. 2022PaLMwas influenced by Chain of thoughtChapter 5. Scale becomes a science
  26. 2022HH-RLHFbuilt directly on RLHF for summarisationChapter 6. Teaching models what people want
  27. 2022Flamingowas influenced by GPT-3
    built directly on CLIP    Chapter 11. Seeing and hearing
  28. 2022OPT-175Bbuilt directly on GPT-3
    was influenced by Megatron-LM    Chapter 8. The weights get out
  29. 2022Zero-shot CoTwas influenced by GPT-3Chapter 12. Thinking longer
  30. 2022Emergent abilitiesbuilt directly on Scaling lawsChapter 5. Scale becomes a science
  31. 2022BLOOMwas influenced by The PileChapter 8. The weights get out
  32. 2022LLM.int8()was influenced by OPT-175BChapter 9. Making it run anywhere
  33. 2022ReActbuilt directly on Chain of thought
    was influenced by WebGPT    Chapter 13. Models that act
  34. 2022Flan-T5 / Flan-PaLMbuilt directly on T5Chapter 6. Teaching models what people want
  35. 2022GPTQwas influenced by LLM.int8()Chapter 9. Making it run anywhere
  36. 2022GPT-3.5 Turning pointbuilt directly on GPT-3Chapter 7. The assistant goes public
  37. 2022ChatGPTbuilt directly on InstructGPT
    built directly on GPT-3.5    Chapter 7. The assistant goes public
  38. 2022Constitutional AIwas influenced by Chain of thoughtChapter 6. Teaching models what people want
  39. 2022Self-Instructbuilt directly on GPT-3Chapter 6. Teaching models what people want
  40. 2023Toolformerbuilt directly on GPT-3Chapter 13. Models that act
  41. 2023LLaMA Turning pointbuilt directly on ChinchillaChapter 8. The weights get out
  42. 2023llama.cppwas influenced by LLaMAChapter 9. Making it run anywhere
  43. 2023Alpacawas influenced by GPT-3.5
    built directly on Self-Instruct    Chapter 8. The weights get out
  44. 2023Claudebuilt directly on Constitutional AI
    was influenced by HH-RLHF    Chapter 7. The assistant goes public
  45. 2023GPT-4 Turning pointbuilt directly on Scaling lawsChapter 7. The assistant goes public
  46. 2023Reflexionbuilt directly on ReActChapter 13. Models that act
  47. 2023AutoGPTwas influenced by GPT-4Chapter 13. Models that act
  48. 2023Pythiawas influenced by Scaling lawsChapter 8. The weights get out
  49. 2023LLaVAbuilt directly on CLIPChapter 11. Seeing and hearing
  50. 2023Emergence miragesucceeded Emergent abilitiesChapter 5. Scale becomes a science
  51. 2023100K contextbuilt directly on ClaudeChapter 10. Reading more than you remember
  52. 2023Tree of thoughtsbuilt directly on Chain of thoughtChapter 12. Thinking longer
  53. 2023QLoRAbuilt directly on LoRAChapter 9. Making it run anywhere
  54. 2023DPObuilt directly on RLHF for summarisation
    built directly on InstructGPT    Chapter 6. Teaching models what people want
  55. 2023Process supervisionbuilt directly on Chain of thoughtChapter 12. Thinking longer
  56. 2023phi-1was influenced by GPT-3.5Chapter 8. The weights get out
  57. 2023Position interpolationbuilt directly on LLaMAChapter 10. Reading more than you remember
  58. 2023Llama 2 Turning pointwas influenced by InstructGPTChapter 8. The weights get out
  59. 2023GPT-4Vbuilt directly on GPT-4Chapter 11. Seeing and hearing
  60. 2023SWE-benchwas influenced by GPT-4
    built directly on Codex / HumanEval    Chapter 13. Models that act
  61. 2023Gemini 1.0succeeded PaLMChapter 11. Seeing and hearing
  62. 2023Mixtral 8x7Bbuilt directly on Switch TransformerChapter 8. The weights get out
  63. 2024DeepSeek LLMbuilt directly on Scaling lawsChapter 8. The weights get out
  64. 2024DeepSeekMoEwas influenced by Switch TransformerChapter 8. The weights get out
  65. 2024Dolmawas influenced by The PileChapter 8. The weights get out
  66. 2024OLMowas influenced by PythiaChapter 8. The weights get out
  67. 2024GRPOwas influenced by InstructGPT
    built directly on DeepSeek LLM    Chapter 12. Thinking longer
  68. 2024Gemini 1.5built directly on Gemini 1.0Chapter 10. Reading more than you remember
  69. 2024Gemmawas influenced by Llama 2Chapter 8. The weights get out
  70. 2024Claude 3was influenced by GPT-4Chapter 14. Where the threads meet now
  71. 2024Chatbot Arenawas influenced by ChatGPTChapter 14. Where the threads meet now
  72. 2024Devinwas influenced by GPT-4Chapter 13. Models that act
  73. 2024Llama 3was influenced by ChinchillaChapter 8. The weights get out
  74. 2024Phi-3built directly on phi-1Chapter 8. The weights get out
  75. 2024DeepSeek-V2 / MLAbuilt directly on DeepSeekMoEChapter 9. Making it run anywhere
  76. 2024SWE-agentbuilt directly on SWE-bench
    was influenced by ReAct    Chapter 13. Models that act
  77. 2024GPT-4obuilt directly on GPT-4Chapter 11. Seeing and hearing
  78. 2024Claude 3.5 Sonnetbuilt directly on Claude 3Chapter 14. Where the threads meet now
  79. 2024FineWebwas influenced by The PileChapter 8. The weights get out
  80. 2024Llama 3.1 405Bwas influenced by ChinchillaChapter 8. The weights get out
  81. 2024Test-time computewas influenced by Process supervision
    built directly on Self-consistency    Chapter 12. Thinking longer
  82. 2024o1 Turning pointwas influenced by Chain of thoughtChapter 12. Thinking longer
  83. 2024Computer usewas influenced by GPT-4VChapter 13. Models that act
  84. 2024Gemini 2.0built directly on Gemini 1.5Chapter 14. Where the threads meet now
  85. 2024o3 announcedbuilt directly on o1Chapter 12. Thinking longer
  86. 2024DeepSeek-V3built directly on DeepSeekMoE
    built directly on GRPO
    was influenced by Llama 3.1 405B    Chapter 8. The weights get out
  87. 2025DeepSeek-R1was influenced by Chain of thoughtChapter 12. Thinking longer
  88. 2025Deep researchwas influenced by o3 releasedChapter 13. Models that act
  89. 2025Claude 3.7was influenced by o1Chapter 12. Thinking longer
  90. 2025Claude Codewas influenced by Computer use
    was influenced by Claude 3.7    Chapter 13. Models that act
  91. 2025GPT-4.5was influenced by Scaling lawsChapter 14. Where the threads meet now
  92. 2025Gemini 2.5was influenced by o1Chapter 12. Thinking longer
  93. 2025Llama 4was influenced by Switch TransformerChapter 8. The weights get out
  94. 2025o3 releasedbuilt directly on o1Chapter 12. Thinking longer
  95. 2025Qwen3was influenced by DeepSeek-R1Chapter 8. The weights get out
  96. 2025Claude 4succeeded Claude 3
    was influenced by SWE-bench    Chapter 13. Models that act
  97. 2025Kimi K2was influenced by DeepSeekMoE
    was influenced by GRPO    Chapter 8. The weights get out
  98. 2025IMO goldbuilt directly on Gemini 2.5
    was influenced by Test-time compute    Chapter 12. Thinking longer
  99. 2025gpt-ossbuilt directly on o3 releasedChapter 8. The weights get out
  100. 2025GPT-5succeeded GPT-4oChapter 14. Where the threads meet now
  101. 2025DeepSeek-V3.2was influenced by DeepSeek-V2 / MLA
    built directly on DeepSeek-V3    Chapter 10. Reading more than you remember
  102. 2025Claude Sonnet 4.5built directly on Claude 4Chapter 13. Models that act
  103. 2025IMO-ProofBenchwas influenced by IMO goldChapter 12. Thinking longer
  104. 2025GPT-5.1built directly on GPT-5Chapter 14. Where the threads meet now
  105. 2025Gemini 3built directly on Gemini 2.5Chapter 14. Where the threads meet now
  106. 2025Claude Opus 4.5was influenced by SWE-benchChapter 13. Models that act
  107. 2025GPT-5.2built directly on GPT-5.1Chapter 14. Where the threads meet now
  108. 2026Claude Opus 4.6built directly on Claude Opus 4.5Chapter 14. Where the threads meet now
  109. 2026GPT-5.3-Codexbuilt directly on GPT-5.2Chapter 13. Models that act
  110. 2026Gemini 3 Deep Thinkbuilt directly on Gemini 3
    was influenced by IMO gold    Chapter 12. Thinking longer
  111. 2026Qwen3.5built directly on Qwen3Chapter 8. The weights get out
  112. 2026Gemini 3.1built directly on Gemini 3Chapter 14. Where the threads meet now
  113. 2026GPT-5.4built directly on GPT-5.3-CodexChapter 14. Where the threads meet now
  114. 2026Muse Sparksucceeded Llama 4Chapter 8. The weights get out
  115. 2026Claude Opus 4.7built directly on Claude Opus 4.6Chapter 14. Where the threads meet now
  116. 2026GPT-5.5built directly on GPT-5.4Chapter 14. Where the threads meet now
  117. 2026DeepSeek-V4was influenced by DeepSeek-V3Chapter 8. The weights get out
  118. 2026Claude Opus 4.8built directly on Claude Opus 4.7Chapter 14. Where the threads meet now
  119. 2026Claude Sonnet 5succeeded Claude Opus 4.8Chapter 14. Where the threads meet now
  120. 2026GPT-5.6 Solbuilt directly on GPT-5.5Chapter 14. Where the threads meet now
  121. 2026Kimi K3built directly on Kimi K2Chapter 8. The weights get out
  122. 2026Claude Opus 5succeeded Claude Opus 4.8Chapter 14. Where the threads meet now
  123. 2026Qwen3.8built directly on Qwen3.5Chapter 8. The weights get out
  124. 2026Claude Opus 5.5built directly on Claude Opus 5Chapter 14. Where the threads meet now
  1. 2019WebTextThis milestone.
  2. 2019GPT-2 Turning pointwas influenced by WebTextChapter 4. Pretrain once, use everywhere
  3. 2019Megatron-LMwas influenced by GPT-2Chapter 4. Pretrain once, use everywhere
  4. 2019RLHF for LMsbuilt directly on GPT-2Chapter 6. Teaching models what people want
  5. 2019C4 Turning pointwas influenced by WebText (indirect influence)Chapter 4. Pretrain once, use everywhere
  6. 2019T5 Turning pointwas influenced by GPT-2
    was influenced by C4    Chapter 4. Pretrain once, use everywhere
  7. 2020Scaling laws Turning pointwas influenced by GPT-2Chapter 5. Scale becomes a science
  8. 2020GPT-3 Turning pointbuilt directly on GPT-2Chapter 5. Scale becomes a science
  9. 2020RLHF for summarisation Turning pointbuilt directly on RLHF for LMsChapter 6. Teaching models what people want
  10. 2020The Pilewas influenced by C4Chapter 5. Scale becomes a science
  11. 2021Switch Transformerbuilt directly on T5Chapter 5. Scale becomes a science
  12. 2021CLIPbuilt directly on GPT-2Chapter 11. Seeing and hearing
  13. 2021LoRAbuilt directly on GPT-3Chapter 9. Making it run anywhere
  14. 2021Copilotwas influenced by Codex / HumanEvalChapter 7. The assistant goes public
  15. 2021Codex / HumanEvalbuilt directly on GPT-3Chapter 13. Models that act
  16. 2021FLANwas influenced by GPT-3
    was influenced by T5    Chapter 6. Teaching models what people want
  17. 2021T0built directly on T5Chapter 6. Teaching models what people want
  18. 2021HHH assistantwas influenced by RLHF for summarisationChapter 6. Teaching models what people want
  19. 2021WebGPTbuilt directly on GPT-3Chapter 13. Models that act
  20. 2022MT-NLGwas influenced by Scaling laws
    built directly on Megatron-LM    Chapter 5. Scale becomes a science
  21. 2022Chain of thought Turning pointwas influenced by GPT-3Chapter 12. Thinking longer
  22. 2022InstructGPT Turning pointbuilt directly on GPT-3Chapter 6. Teaching models what people want
  23. 2022Self-consistencybuilt directly on Chain of thoughtChapter 12. Thinking longer
  24. 2022STaRwas influenced by Chain of thoughtChapter 12. Thinking longer
  25. 2022Chinchilla Turning pointbuilt directly on Scaling lawsChapter 5. Scale becomes a science
  26. 2022PaLMwas influenced by Chain of thoughtChapter 5. Scale becomes a science
  27. 2022HH-RLHFbuilt directly on RLHF for summarisationChapter 6. Teaching models what people want
  28. 2022Flamingowas influenced by GPT-3
    built directly on CLIP    Chapter 11. Seeing and hearing
  29. 2022OPT-175Bbuilt directly on GPT-3
    built directly on The Pile
    was influenced by Megatron-LM    Chapter 8. The weights get out
  30. 2022Zero-shot CoTwas influenced by GPT-3Chapter 12. Thinking longer
  31. 2022Emergent abilitiesbuilt directly on Scaling lawsChapter 5. Scale becomes a science
  32. 2022BLOOMwas influenced by The PileChapter 8. The weights get out
  33. 2022LLM.int8()was influenced by OPT-175BChapter 9. Making it run anywhere
  34. 2022ReActbuilt directly on Chain of thought
    was influenced by WebGPT    Chapter 13. Models that act
  35. 2022Flan-T5 / Flan-PaLMbuilt directly on T5Chapter 6. Teaching models what people want
  36. 2022GPTQwas influenced by LLM.int8()Chapter 9. Making it run anywhere
  37. 2022GPT-3.5 Turning pointbuilt directly on GPT-3Chapter 7. The assistant goes public
  38. 2022ChatGPTbuilt directly on InstructGPT
    built directly on GPT-3.5    Chapter 7. The assistant goes public
  39. 2022Constitutional AIwas influenced by Chain of thoughtChapter 6. Teaching models what people want
  40. 2022Self-Instructbuilt directly on GPT-3Chapter 6. Teaching models what people want
  41. 2023Toolformerbuilt directly on GPT-3Chapter 13. Models that act
  42. 2023LLaMA Turning pointbuilt directly on ChinchillaChapter 8. The weights get out
  43. 2023llama.cppwas influenced by LLaMAChapter 9. Making it run anywhere
  44. 2023Alpacawas influenced by GPT-3.5
    built directly on Self-Instruct    Chapter 8. The weights get out
  45. 2023Claudebuilt directly on Constitutional AI
    was influenced by HH-RLHF    Chapter 7. The assistant goes public
  46. 2023GPT-4 Turning pointbuilt directly on Scaling lawsChapter 7. The assistant goes public
  47. 2023Reflexionbuilt directly on ReActChapter 13. Models that act
  48. 2023AutoGPTwas influenced by GPT-4Chapter 13. Models that act
  49. 2023Pythiawas influenced by Scaling laws
    built directly on The Pile    Chapter 8. The weights get out
  50. 2023LLaVAbuilt directly on CLIPChapter 11. Seeing and hearing
  51. 2023Emergence miragesucceeded Emergent abilitiesChapter 5. Scale becomes a science
  52. 2023100K contextbuilt directly on ClaudeChapter 10. Reading more than you remember
  53. 2023Tree of thoughtsbuilt directly on Chain of thoughtChapter 12. Thinking longer
  54. 2023QLoRAbuilt directly on LoRAChapter 9. Making it run anywhere
  55. 2023Falconbuilt directly on RefinedWebChapter 8. The weights get out
  56. 2023DPObuilt directly on RLHF for summarisation
    built directly on InstructGPT    Chapter 6. Teaching models what people want
  57. 2023Process supervisionbuilt directly on Chain of thoughtChapter 12. Thinking longer
  58. 2023RefinedWebwas influenced by C4Chapter 8. The weights get out
  59. 2023Function callingwas influenced by Toolformer (indirect influence)Chapter 13. Models that act
  60. 2023phi-1was influenced by GPT-3.5
    was influenced by Self-Instruct (indirect influence)    Chapter 8. The weights get out
  61. 2023Position interpolationbuilt directly on LLaMAChapter 10. Reading more than you remember
  62. 2023Lost in the middlewas influenced by 100K context (indirect influence)Chapter 10. Reading more than you remember
  63. 2023Llama 2 Turning pointwas influenced by InstructGPTChapter 8. The weights get out
  64. 2023GPT-4Vwas influenced by CLIP (indirect influence)Chapter 11. Seeing and hearing
  65. 2023SWE-benchwas influenced by GPT-4
    built directly on Codex / HumanEval    Chapter 13. Models that act
  66. 2023Gemini 1.0succeeded PaLMChapter 11. Seeing and hearing
  67. 2023Mixtral 8x7Bbuilt directly on Switch TransformerChapter 8. The weights get out
  68. 2024DeepSeek LLMbuilt directly on Scaling lawsChapter 8. The weights get out
  69. 2024DeepSeekMoEwas influenced by Switch TransformerChapter 8. The weights get out
  70. 2024Dolmawas influenced by C4Chapter 8. The weights get out
  71. 2024OLMobuilt directly on DolmaChapter 8. The weights get out
  72. 2024GRPOwas influenced by InstructGPT
    built directly on DeepSeek LLM    Chapter 12. Thinking longer
  73. 2024Gemini 1.5built directly on Gemini 1.0Chapter 10. Reading more than you remember
  74. 2024Gemmawas influenced by Llama 2Chapter 8. The weights get out
  75. 2024Claude 3was influenced by GPT-4Chapter 14. Where the threads meet now
  76. 2024Chatbot Arenawas influenced by ChatGPTChapter 14. Where the threads meet now
  77. 2024Devinwas influenced by GPT-4Chapter 13. Models that act
  78. 2024Llama 3was influenced by ChinchillaChapter 8. The weights get out
  79. 2024Phi-3built directly on phi-1Chapter 8. The weights get out
  80. 2024DeepSeek-V2 / MLAbuilt directly on DeepSeekMoEChapter 9. Making it run anywhere
  81. 2024SWE-agentbuilt directly on SWE-bench
    was influenced by ReAct    Chapter 13. Models that act
  82. 2024GPT-4obuilt directly on GPT-4
    built directly on GPT-4V    Chapter 11. Seeing and hearing
  83. 2024Claude 3.5 Sonnetbuilt directly on Claude 3Chapter 14. Where the threads meet now
  84. 2024FineWebbuilt directly on RefinedWeb
    was influenced by The Pile    Chapter 8. The weights get out
  85. 2024Llama 3.1 405Bwas influenced by ChinchillaChapter 8. The weights get out
  86. 2024Test-time computewas influenced by Process supervision
    was influenced by Tree of thoughts (indirect influence)
    was influenced by STaR (indirect influence)    Chapter 12. Thinking longer
  87. 2024o1 Turning pointwas influenced by Chain of thoughtChapter 12. Thinking longer
  88. 2024Computer usewas influenced by GPT-4VChapter 13. Models that act
  89. 2024MCPwas influenced by Toolformer (indirect influence)Chapter 13. Models that act
  90. 2024Gemini 2.0was influenced by Computer use (indirect influence)Chapter 14. Where the threads meet now
  91. 2024o3 announcedbuilt directly on o1Chapter 12. Thinking longer
  92. 2024DeepSeek-V3built directly on DeepSeekMoE
    built directly on GRPO
    was influenced by Llama 3.1 405B    Chapter 8. The weights get out
  93. 2025DeepSeek-R1was influenced by Chain of thoughtChapter 12. Thinking longer
  94. 2025Deep researchwas influenced by WebGPT (indirect influence)Chapter 13. Models that act
  95. 2025Claude 3.7was influenced by o1Chapter 12. Thinking longer
  96. 2025Claude Codewas influenced by Computer use
    was influenced by MCP    Chapter 13. Models that act
  97. 2025GPT-4.5was influenced by Scaling lawsChapter 14. Where the threads meet now
  98. 2025Gemini 2.5was influenced by o1Chapter 12. Thinking longer
  99. 2025Llama 4was influenced by Switch TransformerChapter 8. The weights get out
  100. 2025o3 releasedbuilt directly on o1
    was influenced by Function calling    Chapter 12. Thinking longer
  101. 2025Qwen3was influenced by DeepSeek-R1Chapter 8. The weights get out
  102. 2025Claude 4succeeded Claude 3
    was influenced by SWE-bench    Chapter 13. Models that act
  103. 2025Kimi K2was influenced by DeepSeekMoE
    was influenced by GRPO    Chapter 8. The weights get out
  104. 2025IMO goldbuilt directly on Gemini 2.5
    was influenced by Test-time compute    Chapter 12. Thinking longer
  105. 2025gpt-osswas influenced by DeepSeek-R1 (indirect influence)Chapter 8. The weights get out
  106. 2025GPT-5succeeded GPT-4oChapter 14. Where the threads meet now
  107. 2025GLM-4.5was influenced by DeepSeek-R1 (indirect influence)Chapter 8. The weights get out
  108. 2025DeepSeek-V3.2was influenced by DeepSeek-V2 / MLA
    built directly on DeepSeek-V3    Chapter 10. Reading more than you remember
  109. 2025Claude Sonnet 4.5built directly on Claude 4Chapter 13. Models that act
  110. 2025IMO-ProofBenchwas influenced by IMO goldChapter 12. Thinking longer
  111. 2025GPT-5.1built directly on GPT-5Chapter 14. Where the threads meet now
  112. 2025Gemini 3built directly on Gemini 2.5Chapter 14. Where the threads meet now
  113. 2025Claude Opus 4.5was influenced by SWE-benchChapter 13. Models that act
  114. 2025GPT-5.2built directly on GPT-5.1Chapter 14. Where the threads meet now
  115. 2026GPT-5.3-Codexwas influenced by Computer use (indirect influence)Chapter 13. Models that act
  116. 2026Claude Opus 4.6built directly on Claude Opus 4.5Chapter 14. Where the threads meet now
  117. 2026Gemini 3 Deep Thinkbuilt directly on Gemini 3
    was influenced by IMO gold    Chapter 12. Thinking longer
  118. 2026Qwen3.5built directly on Qwen3Chapter 8. The weights get out
  119. 2026Gemini 3.1built directly on Gemini 3Chapter 14. Where the threads meet now
  120. 2026GPT-5.4built directly on GPT-5.3-CodexChapter 14. Where the threads meet now
  121. 2026Project Glasswingwas influenced by Computer use (indirect influence)Chapter 14. Where the threads meet now
  122. 2026Muse Sparksucceeded Llama 4Chapter 8. The weights get out
  123. 2026Claude Opus 4.7built directly on Claude Opus 4.6Chapter 14. Where the threads meet now
  124. 2026GPT-5.5built directly on GPT-5.4Chapter 14. Where the threads meet now
  125. 2026DeepSeek-V4was influenced by DeepSeek-V3Chapter 8. The weights get out
  126. 2026Claude Opus 4.8built directly on Claude Opus 4.7Chapter 14. Where the threads meet now
  127. 2026Claude Sonnet 5succeeded Claude Opus 4.8Chapter 14. Where the threads meet now
  128. 2026Fable 5 / Mythos 5was influenced by Project GlasswingChapter 14. Where the threads meet now
  129. 2026GLM-5.2built directly on GLM-4.5Chapter 8. The weights get out
  130. 2026GPT-5.6 Solbuilt directly on GPT-5.5Chapter 14. Where the threads meet now
  131. 2026Kimi K3built directly on Kimi K2Chapter 8. The weights get out
  132. 2026Claude Opus 5succeeded Claude Opus 4.8Chapter 14. Where the threads meet now
  133. 2026Qwen3.8built directly on Qwen3.5Chapter 8. The weights get out
  134. 2026Claude Opus 5.5built directly on Claude Opus 5Chapter 14. Where the threads meet now

Direct links and their sources

Where it came from

Influenced by

What it led to

Influenced