NLP Lineage

1990 · Language modelling

Recurrent networks

Finding Structure in Time

Jeffrey L. Elman

Date
1990
What the date marks
Journal
Kind
Research paper
Organisation
UC San Diego
Access
Not applicable
Threads
In the guided tour
Chapter 1. Language as probability

A milestone: 1 later milestone descends directly from it, and its descendants reach 10 of the 10 threads. How levels are decided

Read the primary source (onlinelibrary.wiley.com)

Full summary in progress. Until then, this page shows the milestone's record from the research data.

What it changed

Before it

Neural nets processed fixed-size inputs with no notion of order.

The problem

Language arrives as a sequence, and context from earlier words has to persist.

What it introduced

A recurrent hidden layer that feeds its own previous state back in, giving the network a memory of what came before.

Why it matters

It established that sequence can live in a network's internal state rather than in its input window.

What modern models inherit

The idea of a running hidden state summarising everything read so far.

What came next

LSTM fixed the gradient problems this design exposed.

Before and after

The milestones it descends from, and the milestones that descend from it. Work developed alongside it is not counted. Weak links are left out unless you include them.

Before: where it came from

No documented ancestors are recorded for Recurrent networks without weak links.

No documented ancestors are recorded for Recurrent networks without weak links.

No recorded ancestors are recorded for Recurrent networks.

No recorded ancestors are recorded for Recurrent networks.

After: what it led to

Recurrent networks has 162 documented descendants across 14 chapters, reaching forward to Claude Opus 5.5 in 2026. Showing its direct descendants and the 23 turning points among them.

Recurrent networks has 162 documented descendants across 14 chapters, reaching forward to Claude Opus 5.5 in 2026. Showing all of them.

Recurrent networks has 171 recorded descendants, including indirect ones, across 14 chapters, reaching forward to Claude Opus 5.5 in 2026. Showing its direct descendants and the 24 turning points among them.

Recurrent networks has 171 recorded descendants, including indirect ones, across 14 chapters, reaching forward to Claude Opus 5.5 in 2026. Showing all of them.

  1. 1990Recurrent networksThis milestone.
  2. 1997LSTM Turning pointwas influenced by Recurrent networksChapter 1. Language as probability
  3. 2010RNN LMwas influenced by LSTMChapter 1. Language as probability
  4. 2014GRU / encoder-decoderbuilt directly on LSTMChapter 2. Words become vectors
  5. 2014Attention Turning pointbuilt directly on Seq2seq
    built directly on GRU / encoder-decoder    Chapter 3. Attention, then only attention
  6. 2014Seq2seq Turning pointbuilt directly on LSTMChapter 2. Words become vectors
  7. 2015Luong attentionbuilt directly on AttentionChapter 3. Attention, then only attention
  8. 2016GNMTbuilt directly on Seq2seqChapter 2. Words become vectors
  9. 2017Sparse MoE Turning pointbuilt directly on LSTMChapter 3. Attention, then only attention
  10. 2017ConvS2Sbuilt directly on AttentionChapter 3. Attention, then only attention
  11. 2017Transformer Turning pointbuilt directly on Seq2seqChapter 3. Attention, then only attention
  12. 2017CoVebuilt directly on LSTMChapter 3. Attention, then only attention
  13. 2018ELMobuilt directly on LSTMChapter 4. Pretrain once, use everywhere
  14. 2018GPT-1built directly on TransformerChapter 4. Pretrain once, use everywhere
  15. 2018BERT Turning pointwas influenced by ELMoChapter 4. Pretrain once, use everywhere
  16. 2019Transformer-XLbuilt directly on TransformerChapter 4. Pretrain once, use everywhere
  17. 2019GPT-2 Turning pointbuilt directly on TransformerChapter 4. Pretrain once, use everywhere
  18. 2019WebTextwas influenced by GPT-1Chapter 4. Pretrain once, use everywhere
  19. 2019Sparse Transformer Turning pointbuilt directly on TransformerChapter 4. Pretrain once, use everywhere
  20. 2019RoBERTabuilt directly on BERTChapter 4. Pretrain once, use everywhere
  21. 2019Sentence-BERTbuilt directly on BERTChapter 10. Reading more than you remember
  22. 2019Megatron-LMbuilt directly on Transformer
    was influenced by BERT    Chapter 4. Pretrain once, use everywhere
  23. 2019RLHF for LMsbuilt directly on GPT-2Chapter 6. Teaching models what people want
  24. 2019T5 Turning pointbuilt directly on Transformer
    was influenced by BERT    Chapter 4. Pretrain once, use everywhere
  25. 2019Multi-query attentionbuilt directly on TransformerChapter 9. Making it run anywhere
  26. 2020Scaling laws Turning pointbuilt directly on TransformerChapter 5. Scale becomes a science
  27. 2020Longformerbuilt directly on TransformerChapter 10. Reading more than you remember
  28. 2020Dense retrievalbuilt directly on BERTChapter 10. Reading more than you remember
  29. 2020RAGbuilt directly on Dense retrievalChapter 10. Reading more than you remember
  30. 2020GPT-3 Turning pointwas influenced by Scaling laws
    built directly on GPT-2
    built directly on Sparse Transformer    Chapter 5. Scale becomes a science
  31. 2020GShard Turning pointbuilt directly on Sparse MoEChapter 5. Scale becomes a science
  32. 2020BigBirdbuilt directly on TransformerChapter 10. Reading more than you remember
  33. 2020RLHF for summarisation Turning pointbuilt directly on RLHF for LMsChapter 6. Teaching models what people want
  34. 2020Vision Transformerbuilt directly on Transformer
    was influenced by BERT    Chapter 11. Seeing and hearing
  35. 2020The Pilewas influenced by GPT-3Chapter 5. Scale becomes a science
  36. 2021Switch Transformerbuilt directly on Sparse MoEChapter 5. Scale becomes a science
  37. 2021CLIPbuilt directly on GPT-2
    built directly on Vision Transformer    Chapter 11. Seeing and hearing
  38. 2021RoPE Turning pointbuilt directly on TransformerChapter 10. Reading more than you remember
  39. 2021LoRAbuilt directly on TransformerChapter 9. Making it run anywhere
  40. 2021Copilotwas influenced by Codex / HumanEvalChapter 7. The assistant goes public
  41. 2021Codex / HumanEvalbuilt directly on GPT-3Chapter 13. Models that act
  42. 2021ALiBibuilt directly on TransformerChapter 10. Reading more than you remember
  43. 2021FLANwas influenced by T5Chapter 6. Teaching models what people want
  44. 2021T0built directly on T5Chapter 6. Teaching models what people want
  45. 2021HHH assistantwas influenced by RLHF for summarisationChapter 6. Teaching models what people want
  46. 2021RETROwas influenced by Dense retrievalChapter 10. Reading more than you remember
  47. 2021WebGPTbuilt directly on GPT-3Chapter 13. Models that act
  48. 2022MT-NLGwas influenced by Scaling laws
    built directly on Megatron-LM    Chapter 5. Scale becomes a science
  49. 2022Chain of thought Turning pointwas influenced by GPT-3Chapter 12. Thinking longer
  50. 2022InstructGPT Turning pointbuilt directly on GPT-3
    was influenced by FLAN    Chapter 6. Teaching models what people want
  51. 2022Self-consistencybuilt directly on Chain of thoughtChapter 12. Thinking longer
  52. 2022STaRwas influenced by Chain of thoughtChapter 12. Thinking longer
  53. 2022Chinchilla Turning pointbuilt directly on Scaling lawsChapter 5. Scale becomes a science
  54. 2022PaLMwas influenced by GShardChapter 5. Scale becomes a science
  55. 2022HH-RLHFbuilt directly on RLHF for summarisationChapter 6. Teaching models what people want
  56. 2022Flamingobuilt directly on Vision TransformerChapter 11. Seeing and hearing
  57. 2022OPT-175Bwas influenced by Megatron-LMChapter 8. The weights get out
  58. 2022Zero-shot CoTwas influenced by GPT-3Chapter 12. Thinking longer
  59. 2022FlashAttentionbuilt directly on TransformerChapter 9. Making it run anywhere
  60. 2022Emergent abilitiesbuilt directly on Scaling laws
    was influenced by PaLM    Chapter 5. Scale becomes a science
  61. 2022BLOOMbuilt directly on ALiBiChapter 8. The weights get out
  62. 2022LLM.int8()was influenced by OPT-175BChapter 9. Making it run anywhere
  63. 2022Whisperbuilt directly on TransformerChapter 11. Seeing and hearing
  64. 2022ReActbuilt directly on Chain of thought
    was influenced by WebGPT    Chapter 13. Models that act
  65. 2022Flan-T5 / Flan-PaLMbuilt directly on T5
    built directly on PaLM    Chapter 6. Teaching models what people want
  66. 2022GPTQwas influenced by LLM.int8()Chapter 9. Making it run anywhere
  67. 2022Speculative decodingwas influenced by TransformerChapter 9. Making it run anywhere
  68. 2022GPT-3.5 Turning pointbuilt directly on GPT-3Chapter 7. The assistant goes public
  69. 2022ChatGPTbuilt directly on InstructGPT
    built directly on GPT-3.5    Chapter 7. The assistant goes public
  70. 2022Constitutional AIwas influenced by Chain of thoughtChapter 6. Teaching models what people want
  71. 2022Self-Instructbuilt directly on GPT-3
    was influenced by FLAN    Chapter 6. Teaching models what people want
  72. 2023Toolformerbuilt directly on GPT-3Chapter 13. Models that act
  73. 2023LLaMA Turning pointbuilt directly on RoPEChapter 8. The weights get out
  74. 2023llama.cppwas influenced by LLaMAChapter 9. Making it run anywhere
  75. 2023Alpacabuilt directly on LLaMAChapter 8. The weights get out
  76. 2023Claudebuilt directly on Constitutional AI
    was influenced by HH-RLHF    Chapter 7. The assistant goes public
  77. 2023GPT-4 Turning pointbuilt directly on Scaling lawsChapter 7. The assistant goes public
  78. 2023Reflexionbuilt directly on ReActChapter 13. Models that act
  79. 2023AutoGPTwas influenced by GPT-4Chapter 13. Models that act
  80. 2023Pythiawas influenced by Scaling lawsChapter 8. The weights get out
  81. 2023LLaVAbuilt directly on GPT-4
    built directly on LLaMA
    built directly on CLIP    Chapter 11. Seeing and hearing
  82. 2023Emergence miragesucceeded Emergent abilitiesChapter 5. Scale becomes a science
  83. 2023100K contextbuilt directly on ClaudeChapter 10. Reading more than you remember
  84. 2023Tree of thoughtsbuilt directly on Chain of thoughtChapter 12. Thinking longer
  85. 2023Grouped-query attention Turning pointbuilt directly on Multi-query attentionChapter 9. Making it run anywhere
  86. 2023QLoRAbuilt directly on LoRAChapter 9. Making it run anywhere
  87. 2023DPObuilt directly on RLHF for summarisation
    built directly on InstructGPT    Chapter 6. Teaching models what people want
  88. 2023Process supervisionbuilt directly on Chain of thoughtChapter 12. Thinking longer
  89. 2023phi-1was influenced by GPT-3.5Chapter 8. The weights get out
  90. 2023Position interpolationbuilt directly on RoPEChapter 10. Reading more than you remember
  91. 2023Lost in the middlewas influenced by RAGChapter 10. Reading more than you remember
  92. 2023Llama 2 Turning pointbuilt directly on LLaMA
    built directly on Grouped-query attention    Chapter 8. The weights get out
  93. 2023vLLMwas influenced by Multi-query attentionChapter 9. Making it run anywhere
  94. 2023GPT-4Vbuilt directly on GPT-4Chapter 11. Seeing and hearing
  95. 2023Mistral 7Bwas influenced by Sparse Transformer
    was influenced by Longformer
    was influenced by FlashAttention    Chapter 8. The weights get out
  96. 2023SWE-benchwas influenced by GPT-4Chapter 13. Models that act
  97. 2023Mambawas influenced by FlashAttentionChapter 9. Making it run anywhere
  98. 2023Gemini 1.0was influenced by GShardChapter 11. Seeing and hearing
  99. 2023Mixtral 8x7Bwas influenced by Sparse MoEChapter 8. The weights get out
  100. 2024DeepSeek LLMbuilt directly on Scaling lawsChapter 8. The weights get out
  101. 2024DeepSeekMoEwas influenced by Sparse MoEChapter 8. The weights get out
  102. 2024Dolmawas influenced by The PileChapter 8. The weights get out
  103. 2024OLMowas influenced by PythiaChapter 8. The weights get out
  104. 2024GRPObuilt directly on DeepSeek LLMChapter 12. Thinking longer
  105. 2024Gemini 1.5was influenced by GShardChapter 10. Reading more than you remember
  106. 2024Gemmabuilt directly on Gemini 1.0Chapter 8. The weights get out
  107. 2024Claude 3was influenced by GPT-4Chapter 14. Where the threads meet now
  108. 2024Chatbot Arenawas influenced by ChatGPTChapter 14. Where the threads meet now
  109. 2024Devinwas influenced by GPT-4Chapter 13. Models that act
  110. 2024Llama 3built directly on Grouped-query attention
    was influenced by Chinchilla    Chapter 8. The weights get out
  111. 2024Phi-3built directly on phi-1Chapter 8. The weights get out
  112. 2024DeepSeek-V2 / MLAbuilt directly on DeepSeekMoEChapter 9. Making it run anywhere
  113. 2024SWE-agentbuilt directly on SWE-benchChapter 13. Models that act
  114. 2024GPT-4obuilt directly on GPT-4Chapter 11. Seeing and hearing
  115. 2024Claude 3.5 Sonnetbuilt directly on Claude 3Chapter 14. Where the threads meet now
  116. 2024FineWebwas influenced by The PileChapter 8. The weights get out
  117. 2024FlashAttention-3built directly on FlashAttentionChapter 9. Making it run anywhere
  118. 2024Qwen2built directly on RoPEChapter 8. The weights get out
  119. 2024Llama 3.1 405Bwas influenced by Chinchilla
    built directly on Grouped-query attention
    was influenced by Position interpolation    Chapter 8. The weights get out
  120. 2024Test-time computewas influenced by Process supervision
    built directly on Self-consistency    Chapter 12. Thinking longer
  121. 2024o1 Turning pointwas influenced by Chain of thoughtChapter 12. Thinking longer
  122. 2024Computer usewas influenced by GPT-4VChapter 13. Models that act
  123. 2024Gemini 2.0built directly on Gemini 1.5Chapter 14. Where the threads meet now
  124. 2024o3 announcedbuilt directly on o1Chapter 12. Thinking longer
  125. 2024DeepSeek-V3built directly on DeepSeekMoEChapter 8. The weights get out
  126. 2025DeepSeek-R1built directly on DeepSeek-V3Chapter 12. Thinking longer
  127. 2025Deep researchwas influenced by o3 releasedChapter 13. Models that act
  128. 2025Claude 3.7was influenced by o1
    built directly on Claude 3.5 Sonnet    Chapter 12. Thinking longer
  129. 2025Claude Codewas influenced by Computer useChapter 13. Models that act
  130. 2025GPT-4.5was influenced by Scaling lawsChapter 14. Where the threads meet now
  131. 2025Gemini 2.5was influenced by Gemini 1.5Chapter 12. Thinking longer
  132. 2025Llama 4was influenced by Switch TransformerChapter 8. The weights get out
  133. 2025o3 releasedbuilt directly on o1Chapter 12. Thinking longer
  134. 2025Qwen3built directly on Qwen2
    was influenced by DeepSeek-R1    Chapter 8. The weights get out
  135. 2025Claude 4succeeded Claude 3
    was influenced by SWE-bench    Chapter 13. Models that act
  136. 2025Kimi K2was influenced by DeepSeekMoEChapter 8. The weights get out
  137. 2025IMO goldbuilt directly on Gemini 2.5Chapter 12. Thinking longer
  138. 2025gpt-ossbuilt directly on o3 releasedChapter 8. The weights get out
  139. 2025GPT-5succeeded GPT-4oChapter 14. Where the threads meet now
  140. 2025DeepSeek-V3.2was influenced by Sparse Transformer
    was influenced by DeepSeek-V2 / MLA
    built directly on DeepSeek-V3    Chapter 10. Reading more than you remember
  141. 2025Claude Sonnet 4.5built directly on Claude 4Chapter 13. Models that act
  142. 2025IMO-ProofBenchwas influenced by IMO goldChapter 12. Thinking longer
  143. 2025GPT-5.1built directly on GPT-5Chapter 14. Where the threads meet now
  144. 2025Gemini 3was influenced by Gemini 1.5Chapter 14. Where the threads meet now
  145. 2025Claude Opus 4.5was influenced by SWE-benchChapter 13. Models that act
  146. 2025GPT-5.2built directly on GPT-5.1Chapter 14. Where the threads meet now
  147. 2026Claude Opus 4.6built directly on Claude Opus 4.5Chapter 14. Where the threads meet now
  148. 2026GPT-5.3-Codexbuilt directly on GPT-5.2Chapter 13. Models that act
  149. 2026Gemini 3 Deep Thinkbuilt directly on Gemini 3Chapter 12. Thinking longer
  150. 2026Qwen3.5built directly on Qwen3Chapter 8. The weights get out
  151. 2026Gemini 3.1built directly on Gemini 3Chapter 14. Where the threads meet now
  152. 2026GPT-5.4built directly on GPT-5.3-CodexChapter 14. Where the threads meet now
  153. 2026Muse Sparksucceeded Llama 4Chapter 8. The weights get out
  154. 2026Claude Opus 4.7built directly on Claude Opus 4.6Chapter 14. Where the threads meet now
  155. 2026GPT-5.5built directly on GPT-5.4Chapter 14. Where the threads meet now
  156. 2026DeepSeek-V4was influenced by DeepSeek-V3Chapter 8. The weights get out
  157. 2026Claude Opus 4.8built directly on Claude Opus 4.7Chapter 14. Where the threads meet now
  158. 2026Claude Sonnet 5succeeded Claude Opus 4.8Chapter 14. Where the threads meet now
  159. 2026GPT-5.6 Solbuilt directly on GPT-5.5Chapter 14. Where the threads meet now
  160. 2026Kimi K3built directly on Kimi K2Chapter 8. The weights get out
  161. 2026Claude Opus 5succeeded Claude Opus 4.8Chapter 14. Where the threads meet now
  162. 2026Qwen3.8built directly on Qwen3.5Chapter 8. The weights get out
  163. 2026Claude Opus 5.5built directly on Claude Opus 5Chapter 14. Where the threads meet now
  1. 1990Recurrent networksThis milestone.
  2. 1997LSTM Turning pointwas influenced by Recurrent networksChapter 1. Language as probability
  3. 2010RNN LMwas influenced by LSTMChapter 1. Language as probability
  4. 2014GRU / encoder-decoderbuilt directly on LSTMChapter 2. Words become vectors
  5. 2014Attention Turning pointbuilt directly on Seq2seq
    built directly on GRU / encoder-decoder    Chapter 3. Attention, then only attention
  6. 2014Seq2seq Turning pointbuilt directly on LSTMChapter 2. Words become vectors
  7. 2015Luong attentionbuilt directly on AttentionChapter 3. Attention, then only attention
  8. 2016GNMTbuilt directly on Seq2seqChapter 2. Words become vectors
  9. 2017Sparse MoE Turning pointbuilt directly on LSTMChapter 3. Attention, then only attention
  10. 2017ConvS2Sbuilt directly on AttentionChapter 3. Attention, then only attention
  11. 2017Transformer Turning pointbuilt directly on Seq2seqChapter 3. Attention, then only attention
  12. 2017CoVebuilt directly on LSTMChapter 3. Attention, then only attention
  13. 2018ELMobuilt directly on LSTMChapter 4. Pretrain once, use everywhere
  14. 2018GPT-1was influenced by ELMo (indirect influence)Chapter 4. Pretrain once, use everywhere
  15. 2018BERT Turning pointwas influenced by ELMoChapter 4. Pretrain once, use everywhere
  16. 2019Transformer-XLbuilt directly on TransformerChapter 4. Pretrain once, use everywhere
  17. 2019GPT-2 Turning pointbuilt directly on Transformer
    built directly on GPT-1    Chapter 4. Pretrain once, use everywhere
  18. 2019WebTextwas influenced by GPT-1Chapter 4. Pretrain once, use everywhere
  19. 2019Sparse Transformer Turning pointbuilt directly on TransformerChapter 4. Pretrain once, use everywhere
  20. 2019RoBERTabuilt directly on BERTChapter 4. Pretrain once, use everywhere
  21. 2019Sentence-BERTbuilt directly on BERTChapter 10. Reading more than you remember
  22. 2019Megatron-LMbuilt directly on Transformer
    was influenced by BERT    Chapter 4. Pretrain once, use everywhere
  23. 2019RLHF for LMsbuilt directly on GPT-2Chapter 6. Teaching models what people want
  24. 2019T5 Turning pointbuilt directly on Transformer
    was influenced by BERT    Chapter 4. Pretrain once, use everywhere
  25. 2019C4 Turning pointwas influenced by WebText (indirect influence)Chapter 4. Pretrain once, use everywhere
  26. 2019Multi-query attentionbuilt directly on TransformerChapter 9. Making it run anywhere
  27. 2020Scaling laws Turning pointbuilt directly on TransformerChapter 5. Scale becomes a science
  28. 2020Longformerbuilt directly on TransformerChapter 10. Reading more than you remember
  29. 2020Dense retrievalbuilt directly on BERTChapter 10. Reading more than you remember
  30. 2020RAGbuilt directly on Dense retrievalChapter 10. Reading more than you remember
  31. 2020GPT-3 Turning pointwas influenced by Scaling laws
    built directly on GPT-2
    built directly on Sparse Transformer    Chapter 5. Scale becomes a science
  32. 2020GShard Turning pointbuilt directly on Sparse MoEChapter 5. Scale becomes a science
  33. 2020BigBirdbuilt directly on TransformerChapter 10. Reading more than you remember
  34. 2020RLHF for summarisation Turning pointbuilt directly on RLHF for LMsChapter 6. Teaching models what people want
  35. 2020Vision Transformerbuilt directly on Transformer
    was influenced by BERT    Chapter 11. Seeing and hearing
  36. 2020The Pilewas influenced by GPT-3
    was influenced by C4    Chapter 5. Scale becomes a science
  37. 2021Switch Transformerbuilt directly on Sparse MoEChapter 5. Scale becomes a science
  38. 2021CLIPbuilt directly on GPT-2
    built directly on Vision Transformer    Chapter 11. Seeing and hearing
  39. 2021RoPE Turning pointbuilt directly on TransformerChapter 10. Reading more than you remember
  40. 2021LoRAbuilt directly on TransformerChapter 9. Making it run anywhere
  41. 2021Copilotwas influenced by Codex / HumanEvalChapter 7. The assistant goes public
  42. 2021Codex / HumanEvalbuilt directly on GPT-3Chapter 13. Models that act
  43. 2021ALiBibuilt directly on TransformerChapter 10. Reading more than you remember
  44. 2021FLANwas influenced by T5Chapter 6. Teaching models what people want
  45. 2021T0built directly on T5Chapter 6. Teaching models what people want
  46. 2021HHH assistantwas influenced by RLHF for summarisationChapter 6. Teaching models what people want
  47. 2021RETROwas influenced by Dense retrievalChapter 10. Reading more than you remember
  48. 2021WebGPTbuilt directly on GPT-3Chapter 13. Models that act
  49. 2022MT-NLGwas influenced by Scaling laws
    built directly on Megatron-LM    Chapter 5. Scale becomes a science
  50. 2022Chain of thought Turning pointwas influenced by GPT-3Chapter 12. Thinking longer
  51. 2022InstructGPT Turning pointbuilt directly on GPT-3
    was influenced by FLAN    Chapter 6. Teaching models what people want
  52. 2022Self-consistencybuilt directly on Chain of thoughtChapter 12. Thinking longer
  53. 2022STaRwas influenced by Chain of thoughtChapter 12. Thinking longer
  54. 2022Chinchilla Turning pointbuilt directly on Scaling lawsChapter 5. Scale becomes a science
  55. 2022PaLMwas influenced by GShardChapter 5. Scale becomes a science
  56. 2022HH-RLHFbuilt directly on RLHF for summarisationChapter 6. Teaching models what people want
  57. 2022Flamingobuilt directly on Vision TransformerChapter 11. Seeing and hearing
  58. 2022OPT-175Bwas influenced by Megatron-LMChapter 8. The weights get out
  59. 2022Zero-shot CoTwas influenced by GPT-3Chapter 12. Thinking longer
  60. 2022FlashAttentionbuilt directly on TransformerChapter 9. Making it run anywhere
  61. 2022Emergent abilitiesbuilt directly on Scaling laws
    was influenced by PaLM    Chapter 5. Scale becomes a science
  62. 2022BLOOMbuilt directly on ALiBiChapter 8. The weights get out
  63. 2022LLM.int8()was influenced by OPT-175BChapter 9. Making it run anywhere
  64. 2022Whisperbuilt directly on TransformerChapter 11. Seeing and hearing
  65. 2022ReActbuilt directly on Chain of thought
    was influenced by WebGPT    Chapter 13. Models that act
  66. 2022Flan-T5 / Flan-PaLMbuilt directly on T5
    built directly on PaLM    Chapter 6. Teaching models what people want
  67. 2022GPTQwas influenced by LLM.int8()Chapter 9. Making it run anywhere
  68. 2022Speculative decodingwas influenced by TransformerChapter 9. Making it run anywhere
  69. 2022GPT-3.5 Turning pointbuilt directly on GPT-3Chapter 7. The assistant goes public
  70. 2022ChatGPTbuilt directly on InstructGPT
    built directly on GPT-3.5    Chapter 7. The assistant goes public
  71. 2022Constitutional AIwas influenced by Chain of thoughtChapter 6. Teaching models what people want
  72. 2022Self-Instructbuilt directly on GPT-3
    was influenced by FLAN    Chapter 6. Teaching models what people want
  73. 2023Toolformerbuilt directly on GPT-3Chapter 13. Models that act
  74. 2023LLaMA Turning pointbuilt directly on RoPEChapter 8. The weights get out
  75. 2023llama.cppwas influenced by LLaMAChapter 9. Making it run anywhere
  76. 2023Alpacabuilt directly on LLaMAChapter 8. The weights get out
  77. 2023Claudebuilt directly on Constitutional AI
    was influenced by HH-RLHF    Chapter 7. The assistant goes public
  78. 2023GPT-4 Turning pointbuilt directly on Scaling lawsChapter 7. The assistant goes public
  79. 2023Reflexionbuilt directly on ReActChapter 13. Models that act
  80. 2023AutoGPTwas influenced by GPT-4Chapter 13. Models that act
  81. 2023Pythiawas influenced by Scaling lawsChapter 8. The weights get out
  82. 2023LLaVAbuilt directly on GPT-4
    built directly on LLaMA
    built directly on CLIP    Chapter 11. Seeing and hearing
  83. 2023Emergence miragesucceeded Emergent abilitiesChapter 5. Scale becomes a science
  84. 2023100K contextbuilt directly on ClaudeChapter 10. Reading more than you remember
  85. 2023Tree of thoughtsbuilt directly on Chain of thoughtChapter 12. Thinking longer
  86. 2023Grouped-query attention Turning pointbuilt directly on Multi-query attentionChapter 9. Making it run anywhere
  87. 2023QLoRAbuilt directly on LoRAChapter 9. Making it run anywhere
  88. 2023Falconbuilt directly on RefinedWebChapter 8. The weights get out
  89. 2023DPObuilt directly on RLHF for summarisation
    built directly on InstructGPT    Chapter 6. Teaching models what people want
  90. 2023Process supervisionbuilt directly on Chain of thoughtChapter 12. Thinking longer
  91. 2023RefinedWebwas influenced by C4Chapter 8. The weights get out
  92. 2023Function callingwas influenced by Toolformer (indirect influence)Chapter 13. Models that act
  93. 2023phi-1was influenced by GPT-3.5
    was influenced by Self-Instruct (indirect influence)    Chapter 8. The weights get out
  94. 2023Position interpolationbuilt directly on RoPEChapter 10. Reading more than you remember
  95. 2023Lost in the middlewas influenced by RAGChapter 10. Reading more than you remember
  96. 2023Llama 2 Turning pointbuilt directly on LLaMA
    built directly on Grouped-query attention    Chapter 8. The weights get out
  97. 2023vLLMwas influenced by Multi-query attentionChapter 9. Making it run anywhere
  98. 2023GPT-4Vbuilt directly on GPT-4
    was influenced by CLIP (indirect influence)    Chapter 11. Seeing and hearing
  99. 2023Mistral 7Bwas influenced by Sparse Transformer
    was influenced by Longformer
    was influenced by FlashAttention    Chapter 8. The weights get out
  100. 2023SWE-benchwas influenced by GPT-4Chapter 13. Models that act
  101. 2023Mambawas influenced by FlashAttentionChapter 9. Making it run anywhere
  102. 2023Gemini 1.0was influenced by GShardChapter 11. Seeing and hearing
  103. 2023Mixtral 8x7Bwas influenced by Sparse MoEChapter 8. The weights get out
  104. 2024DeepSeek LLMbuilt directly on Scaling lawsChapter 8. The weights get out
  105. 2024DeepSeekMoEwas influenced by Sparse MoEChapter 8. The weights get out
  106. 2024Dolmawas influenced by C4Chapter 8. The weights get out
  107. 2024OLMowas influenced by PythiaChapter 8. The weights get out
  108. 2024GRPObuilt directly on DeepSeek LLMChapter 12. Thinking longer
  109. 2024Gemini 1.5was influenced by GShardChapter 10. Reading more than you remember
  110. 2024Gemmabuilt directly on Gemini 1.0Chapter 8. The weights get out
  111. 2024Claude 3was influenced by GPT-4Chapter 14. Where the threads meet now
  112. 2024Chatbot Arenawas influenced by ChatGPTChapter 14. Where the threads meet now
  113. 2024Devinwas influenced by GPT-4Chapter 13. Models that act
  114. 2024Llama 3built directly on Grouped-query attention
    was influenced by Chinchilla    Chapter 8. The weights get out
  115. 2024Phi-3built directly on phi-1Chapter 8. The weights get out
  116. 2024DeepSeek-V2 / MLAbuilt directly on DeepSeekMoEChapter 9. Making it run anywhere
  117. 2024SWE-agentbuilt directly on SWE-benchChapter 13. Models that act
  118. 2024GPT-4owas influenced by Whisper (indirect influence)Chapter 11. Seeing and hearing
  119. 2024Claude 3.5 Sonnetbuilt directly on Claude 3Chapter 14. Where the threads meet now
  120. 2024FineWebbuilt directly on RefinedWeb
    was influenced by The Pile    Chapter 8. The weights get out
  121. 2024FlashAttention-3built directly on FlashAttentionChapter 9. Making it run anywhere
  122. 2024Qwen2built directly on RoPEChapter 8. The weights get out
  123. 2024Llama 3.1 405Bwas influenced by Chinchilla
    built directly on Grouped-query attention
    was influenced by Position interpolation    Chapter 8. The weights get out
  124. 2024Test-time computewas influenced by Process supervision
    was influenced by Tree of thoughts (indirect influence)
    was influenced by STaR (indirect influence)    Chapter 12. Thinking longer
  125. 2024o1 Turning pointwas influenced by Chain of thoughtChapter 12. Thinking longer
  126. 2024Computer usewas influenced by GPT-4VChapter 13. Models that act
  127. 2024MCPwas influenced by Toolformer (indirect influence)Chapter 13. Models that act
  128. 2024Gemini 2.0built directly on Gemini 1.5Chapter 14. Where the threads meet now
  129. 2024o3 announcedbuilt directly on o1Chapter 12. Thinking longer
  130. 2024DeepSeek-V3built directly on DeepSeekMoEChapter 8. The weights get out
  131. 2025DeepSeek-R1built directly on DeepSeek-V3Chapter 12. Thinking longer
  132. 2025Deep researchwas influenced by WebGPT (indirect influence)Chapter 13. Models that act
  133. 2025Claude 3.7was influenced by o1
    built directly on Claude 3.5 Sonnet    Chapter 12. Thinking longer
  134. 2025Claude Codewas influenced by Computer use
    was influenced by SWE-agent (indirect influence)
    was influenced by MCP    Chapter 13. Models that act
  135. 2025GPT-4.5was influenced by Scaling lawsChapter 14. Where the threads meet now
  136. 2025Gemini 2.5was influenced by Gemini 1.5Chapter 12. Thinking longer
  137. 2025Llama 4was influenced by Switch Transformer
    was influenced by Mixtral 8x7B (indirect influence)    Chapter 8. The weights get out
  138. 2025o3 releasedbuilt directly on o1
    was influenced by Function calling    Chapter 12. Thinking longer
  139. 2025Qwen3built directly on Qwen2
    was influenced by DeepSeek-R1    Chapter 8. The weights get out
  140. 2025Claude 4succeeded Claude 3
    was influenced by SWE-bench    Chapter 13. Models that act
  141. 2025Kimi K2was influenced by DeepSeekMoEChapter 8. The weights get out
  142. 2025IMO goldbuilt directly on Gemini 2.5Chapter 12. Thinking longer
  143. 2025gpt-osswas influenced by DeepSeek-R1 (indirect influence)Chapter 8. The weights get out
  144. 2025GPT-5succeeded GPT-4oChapter 14. Where the threads meet now
  145. 2025GLM-4.5was influenced by DeepSeek-R1 (indirect influence)Chapter 8. The weights get out
  146. 2025DeepSeek-V3.2was influenced by Sparse Transformer
    was influenced by DeepSeek-V2 / MLA
    built directly on DeepSeek-V3    Chapter 10. Reading more than you remember
  147. 2025Claude Sonnet 4.5built directly on Claude 4Chapter 13. Models that act
  148. 2025IMO-ProofBenchwas influenced by IMO goldChapter 12. Thinking longer
  149. 2025GPT-5.1built directly on GPT-5Chapter 14. Where the threads meet now
  150. 2025Gemini 3was influenced by Gemini 1.5Chapter 14. Where the threads meet now
  151. 2025Claude Opus 4.5was influenced by SWE-benchChapter 13. Models that act
  152. 2025GPT-5.2built directly on GPT-5.1Chapter 14. Where the threads meet now
  153. 2026GPT-5.3-Codexwas influenced by Computer use (indirect influence)Chapter 13. Models that act
  154. 2026Claude Opus 4.6built directly on Claude Opus 4.5Chapter 14. Where the threads meet now
  155. 2026Gemini 3 Deep Thinkbuilt directly on Gemini 3Chapter 12. Thinking longer
  156. 2026Qwen3.5built directly on Qwen3Chapter 8. The weights get out
  157. 2026Gemini 3.1built directly on Gemini 3Chapter 14. Where the threads meet now
  158. 2026GPT-5.4built directly on GPT-5.3-CodexChapter 14. Where the threads meet now
  159. 2026Project Glasswingwas influenced by Computer use (indirect influence)Chapter 14. Where the threads meet now
  160. 2026Muse Sparksucceeded Llama 4Chapter 8. The weights get out
  161. 2026Claude Opus 4.7built directly on Claude Opus 4.6Chapter 14. Where the threads meet now
  162. 2026GPT-5.5built directly on GPT-5.4Chapter 14. Where the threads meet now
  163. 2026DeepSeek-V4was influenced by DeepSeek-V3Chapter 8. The weights get out
  164. 2026Claude Opus 4.8built directly on Claude Opus 4.7Chapter 14. Where the threads meet now
  165. 2026Claude Sonnet 5succeeded Claude Opus 4.8Chapter 14. Where the threads meet now
  166. 2026Fable 5 / Mythos 5was influenced by Project GlasswingChapter 14. Where the threads meet now
  167. 2026GLM-5.2built directly on GLM-4.5Chapter 8. The weights get out
  168. 2026GPT-5.6 Solbuilt directly on GPT-5.5Chapter 14. Where the threads meet now
  169. 2026Kimi K3built directly on Kimi K2Chapter 8. The weights get out
  170. 2026Claude Opus 5succeeded Claude Opus 4.8Chapter 14. Where the threads meet now
  171. 2026Qwen3.8built directly on Qwen3.5Chapter 8. The weights get out
  172. 2026Claude Opus 5.5built directly on Claude Opus 5Chapter 14. Where the threads meet now

Direct links and their sources

Where it came from

None recorded in the sources used.

What it led to

Influenced