NLP Lineage

3 June 2014 · Language modelling

GRU / encoder-decoder

Learning Phrase Representations using RNN Encoder-Decoder

K. Cho, B. van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, Y. Bengio

Date
3 June 2014
What the date marks
arXiv submission
Kind
Research paper
Organisation
Université de Montréal
Access
Not applicable
Other dates
EMNLP 2014
Threads
In the guided tour
Chapter 2. Words become vectors
Learning phases
1. Transformer architecture

A milestone: 1 later milestone descends directly from it, and its descendants reach 10 of the 10 threads. How levels are decided

Read the primary source (arxiv.org)

Full summary in progress. Until then, this page shows the milestone's record from the research data.

What it changed

Before it

Neural nets scored translations produced by statistical systems.

The problem

No neural architecture mapped a whole variable-length sentence to another.

What it introduced

An encoder RNN compressing a sentence into a vector and a decoder RNN generating from it, with a simplified gated unit.

Why it matters

It set up the encoder-decoder frame, and the bottleneck it exposed is what attention was invented to fix.

What modern models inherit

The encoder-decoder shape still used for translation and, in modified form, T5.

What came next

Bahdanau attention, by an author of this paper, removed the bottleneck a few months later.

Before and after

The milestones it descends from, and the milestones that descend from it. Work developed alongside it is not counted. Weak links are left out unless you include them.

Before: where it came from

GRU / encoder-decoder has 2 documented ancestors across 1 chapter, reaching back to Recurrent networks in 1990. Showing its direct parents and the 1 turning point among them.

GRU / encoder-decoder has 2 documented ancestors across 1 chapter, reaching back to Recurrent networks in 1990. Showing all of them.

GRU / encoder-decoder has 2 recorded ancestors, including indirect ones, across 1 chapter, reaching back to Recurrent networks in 1990. Showing its direct parents and the 1 turning point among them.

GRU / encoder-decoder has 2 recorded ancestors, including indirect ones, across 1 chapter, reaching back to Recurrent networks in 1990. Showing all of them.

  1. 2014GRU / encoder-decoderThis milestone.
  2. 1997LSTM Turning pointGRU / encoder-decoder built directly on thisChapter 1. Language as probability
  3. 1990Recurrent networksLSTM was influenced by thisChapter 1. Language as probability
  1. 2014GRU / encoder-decoderThis milestone.
  2. 1997LSTM Turning pointGRU / encoder-decoder built directly on thisChapter 1. Language as probability
  3. 1990Recurrent networksLSTM was influenced by thisChapter 1. Language as probability

After: what it led to

GRU / encoder-decoder has 154 documented descendants across 13 chapters, reaching forward to Claude Opus 5.5 in 2026. Showing its direct descendants and the 19 turning points among them.

GRU / encoder-decoder has 154 documented descendants across 13 chapters, reaching forward to Claude Opus 5.5 in 2026. Showing all of them.

GRU / encoder-decoder has 163 recorded descendants, including indirect ones, across 13 chapters, reaching forward to Claude Opus 5.5 in 2026. Showing its direct descendants and the 20 turning points among them.

GRU / encoder-decoder has 163 recorded descendants, including indirect ones, across 13 chapters, reaching forward to Claude Opus 5.5 in 2026. Showing all of them.

  1. 2014GRU / encoder-decoderThis milestone.
  2. 2014Attention Turning pointbuilt directly on GRU / encoder-decoderChapter 3. Attention, then only attention
  3. 2015Luong attentionbuilt directly on AttentionChapter 3. Attention, then only attention
  4. 2016GNMTbuilt directly on AttentionChapter 2. Words become vectors
  5. 2017ConvS2Sbuilt directly on AttentionChapter 3. Attention, then only attention
  6. 2017Transformer Turning pointbuilt directly on AttentionChapter 3. Attention, then only attention
  7. 2018GPT-1built directly on TransformerChapter 4. Pretrain once, use everywhere
  8. 2018BERT Turning pointbuilt directly on TransformerChapter 4. Pretrain once, use everywhere
  9. 2019Transformer-XLbuilt directly on TransformerChapter 4. Pretrain once, use everywhere
  10. 2019GPT-2 Turning pointbuilt directly on TransformerChapter 4. Pretrain once, use everywhere
  11. 2019WebTextwas influenced by GPT-1Chapter 4. Pretrain once, use everywhere
  12. 2019Sparse Transformer Turning pointbuilt directly on TransformerChapter 4. Pretrain once, use everywhere
  13. 2019RoBERTabuilt directly on BERTChapter 4. Pretrain once, use everywhere
  14. 2019Sentence-BERTbuilt directly on BERTChapter 10. Reading more than you remember
  15. 2019Megatron-LMbuilt directly on TransformerChapter 4. Pretrain once, use everywhere
  16. 2019RLHF for LMsbuilt directly on GPT-2Chapter 6. Teaching models what people want
  17. 2019T5 Turning pointbuilt directly on TransformerChapter 4. Pretrain once, use everywhere
  18. 2019Multi-query attentionbuilt directly on TransformerChapter 9. Making it run anywhere
  19. 2020Scaling laws Turning pointbuilt directly on TransformerChapter 5. Scale becomes a science
  20. 2020Longformerbuilt directly on TransformerChapter 10. Reading more than you remember
  21. 2020Dense retrievalbuilt directly on BERTChapter 10. Reading more than you remember
  22. 2020RAGbuilt directly on Dense retrievalChapter 10. Reading more than you remember
  23. 2020GPT-3 Turning pointwas influenced by Scaling laws
    built directly on GPT-2
    built directly on Sparse Transformer    Chapter 5. Scale becomes a science
  24. 2020BigBirdbuilt directly on TransformerChapter 10. Reading more than you remember
  25. 2020RLHF for summarisation Turning pointbuilt directly on RLHF for LMsChapter 6. Teaching models what people want
  26. 2020Vision Transformerbuilt directly on TransformerChapter 11. Seeing and hearing
  27. 2020The Pilewas influenced by GPT-3Chapter 5. Scale becomes a science
  28. 2021Switch Transformerbuilt directly on T5Chapter 5. Scale becomes a science
  29. 2021CLIPbuilt directly on GPT-2
    built directly on Vision Transformer    Chapter 11. Seeing and hearing
  30. 2021RoPE Turning pointbuilt directly on TransformerChapter 10. Reading more than you remember
  31. 2021LoRAbuilt directly on TransformerChapter 9. Making it run anywhere
  32. 2021Copilotwas influenced by Codex / HumanEvalChapter 7. The assistant goes public
  33. 2021Codex / HumanEvalbuilt directly on GPT-3Chapter 13. Models that act
  34. 2021ALiBibuilt directly on TransformerChapter 10. Reading more than you remember
  35. 2021FLANwas influenced by T5Chapter 6. Teaching models what people want
  36. 2021T0built directly on T5Chapter 6. Teaching models what people want
  37. 2021HHH assistantwas influenced by RLHF for summarisationChapter 6. Teaching models what people want
  38. 2021RETROwas influenced by Dense retrievalChapter 10. Reading more than you remember
  39. 2021WebGPTbuilt directly on GPT-3Chapter 13. Models that act
  40. 2022MT-NLGwas influenced by Scaling laws
    built directly on Megatron-LM    Chapter 5. Scale becomes a science
  41. 2022Chain of thought Turning pointwas influenced by GPT-3Chapter 12. Thinking longer
  42. 2022InstructGPT Turning pointbuilt directly on GPT-3
    was influenced by FLAN    Chapter 6. Teaching models what people want
  43. 2022Self-consistencybuilt directly on Chain of thoughtChapter 12. Thinking longer
  44. 2022STaRwas influenced by Chain of thoughtChapter 12. Thinking longer
  45. 2022Chinchilla Turning pointbuilt directly on Scaling lawsChapter 5. Scale becomes a science
  46. 2022PaLMwas influenced by Chain of thoughtChapter 5. Scale becomes a science
  47. 2022HH-RLHFbuilt directly on RLHF for summarisationChapter 6. Teaching models what people want
  48. 2022Flamingobuilt directly on Vision TransformerChapter 11. Seeing and hearing
  49. 2022OPT-175Bwas influenced by Megatron-LMChapter 8. The weights get out
  50. 2022Zero-shot CoTwas influenced by GPT-3Chapter 12. Thinking longer
  51. 2022FlashAttentionbuilt directly on TransformerChapter 9. Making it run anywhere
  52. 2022Emergent abilitiesbuilt directly on Scaling lawsChapter 5. Scale becomes a science
  53. 2022BLOOMbuilt directly on ALiBiChapter 8. The weights get out
  54. 2022LLM.int8()was influenced by OPT-175BChapter 9. Making it run anywhere
  55. 2022Whisperbuilt directly on TransformerChapter 11. Seeing and hearing
  56. 2022ReActbuilt directly on Chain of thought
    was influenced by WebGPT    Chapter 13. Models that act
  57. 2022Flan-T5 / Flan-PaLMbuilt directly on T5Chapter 6. Teaching models what people want
  58. 2022GPTQwas influenced by LLM.int8()Chapter 9. Making it run anywhere
  59. 2022Speculative decodingwas influenced by TransformerChapter 9. Making it run anywhere
  60. 2022GPT-3.5 Turning pointbuilt directly on GPT-3Chapter 7. The assistant goes public
  61. 2022ChatGPTbuilt directly on InstructGPT
    built directly on GPT-3.5    Chapter 7. The assistant goes public
  62. 2022Constitutional AIwas influenced by Chain of thoughtChapter 6. Teaching models what people want
  63. 2022Self-Instructbuilt directly on GPT-3
    was influenced by FLAN    Chapter 6. Teaching models what people want
  64. 2023Toolformerbuilt directly on GPT-3Chapter 13. Models that act
  65. 2023LLaMA Turning pointbuilt directly on RoPEChapter 8. The weights get out
  66. 2023llama.cppwas influenced by LLaMAChapter 9. Making it run anywhere
  67. 2023Alpacabuilt directly on LLaMAChapter 8. The weights get out
  68. 2023Claudebuilt directly on Constitutional AI
    was influenced by HH-RLHF    Chapter 7. The assistant goes public
  69. 2023GPT-4 Turning pointbuilt directly on Scaling lawsChapter 7. The assistant goes public
  70. 2023Reflexionbuilt directly on ReActChapter 13. Models that act
  71. 2023AutoGPTwas influenced by GPT-4Chapter 13. Models that act
  72. 2023Pythiawas influenced by Scaling lawsChapter 8. The weights get out
  73. 2023LLaVAbuilt directly on GPT-4
    built directly on LLaMA
    built directly on CLIP    Chapter 11. Seeing and hearing
  74. 2023Emergence miragesucceeded Emergent abilitiesChapter 5. Scale becomes a science
  75. 2023100K contextbuilt directly on ClaudeChapter 10. Reading more than you remember
  76. 2023Tree of thoughtsbuilt directly on Chain of thoughtChapter 12. Thinking longer
  77. 2023Grouped-query attention Turning pointbuilt directly on Multi-query attentionChapter 9. Making it run anywhere
  78. 2023QLoRAbuilt directly on LoRAChapter 9. Making it run anywhere
  79. 2023DPObuilt directly on RLHF for summarisation
    built directly on InstructGPT    Chapter 6. Teaching models what people want
  80. 2023Process supervisionbuilt directly on Chain of thoughtChapter 12. Thinking longer
  81. 2023phi-1was influenced by GPT-3.5Chapter 8. The weights get out
  82. 2023Position interpolationbuilt directly on RoPEChapter 10. Reading more than you remember
  83. 2023Lost in the middlewas influenced by RAGChapter 10. Reading more than you remember
  84. 2023Llama 2 Turning pointbuilt directly on LLaMA
    built directly on Grouped-query attention    Chapter 8. The weights get out
  85. 2023vLLMwas influenced by Multi-query attentionChapter 9. Making it run anywhere
  86. 2023GPT-4Vbuilt directly on GPT-4Chapter 11. Seeing and hearing
  87. 2023Mistral 7Bwas influenced by Sparse Transformer
    was influenced by Longformer
    was influenced by FlashAttention    Chapter 8. The weights get out
  88. 2023SWE-benchwas influenced by GPT-4Chapter 13. Models that act
  89. 2023Mambawas influenced by FlashAttentionChapter 9. Making it run anywhere
  90. 2023Gemini 1.0was influenced by Vision TransformerChapter 11. Seeing and hearing
  91. 2023Mixtral 8x7Bbuilt directly on Switch Transformer
    built directly on Mistral 7B    Chapter 8. The weights get out
  92. 2024DeepSeek LLMbuilt directly on Scaling lawsChapter 8. The weights get out
  93. 2024DeepSeekMoEwas influenced by Switch TransformerChapter 8. The weights get out
  94. 2024Dolmawas influenced by The PileChapter 8. The weights get out
  95. 2024OLMowas influenced by PythiaChapter 8. The weights get out
  96. 2024GRPObuilt directly on DeepSeek LLMChapter 12. Thinking longer
  97. 2024Gemini 1.5built directly on Gemini 1.0Chapter 10. Reading more than you remember
  98. 2024Gemmabuilt directly on Gemini 1.0
    was influenced by Mistral 7B    Chapter 8. The weights get out
  99. 2024Claude 3was influenced by GPT-4Chapter 14. Where the threads meet now
  100. 2024Chatbot Arenawas influenced by ChatGPTChapter 14. Where the threads meet now
  101. 2024Devinwas influenced by GPT-4Chapter 13. Models that act
  102. 2024Llama 3built directly on Grouped-query attention
    was influenced by Chinchilla    Chapter 8. The weights get out
  103. 2024Phi-3built directly on phi-1Chapter 8. The weights get out
  104. 2024DeepSeek-V2 / MLAbuilt directly on RoPE
    was influenced by Multi-query attention    Chapter 9. Making it run anywhere
  105. 2024SWE-agentbuilt directly on SWE-benchChapter 13. Models that act
  106. 2024GPT-4obuilt directly on GPT-4Chapter 11. Seeing and hearing
  107. 2024Claude 3.5 Sonnetbuilt directly on Claude 3Chapter 14. Where the threads meet now
  108. 2024FineWebwas influenced by The PileChapter 8. The weights get out
  109. 2024FlashAttention-3built directly on FlashAttentionChapter 9. Making it run anywhere
  110. 2024Qwen2built directly on RoPEChapter 8. The weights get out
  111. 2024Llama 3.1 405Bwas influenced by Chinchilla
    built directly on Grouped-query attention
    was influenced by Position interpolation    Chapter 8. The weights get out
  112. 2024Test-time computewas influenced by Process supervision
    built directly on Self-consistency    Chapter 12. Thinking longer
  113. 2024o1 Turning pointwas influenced by Chain of thoughtChapter 12. Thinking longer
  114. 2024Computer usewas influenced by GPT-4VChapter 13. Models that act
  115. 2024Gemini 2.0built directly on Gemini 1.5Chapter 14. Where the threads meet now
  116. 2024o3 announcedbuilt directly on o1Chapter 12. Thinking longer
  117. 2024DeepSeek-V3built directly on DeepSeek-V2 / MLAChapter 8. The weights get out
  118. 2025DeepSeek-R1was influenced by Chain of thought
    built directly on GRPO
    built directly on DeepSeek-V3    Chapter 12. Thinking longer
  119. 2025Deep researchwas influenced by o3 releasedChapter 13. Models that act
  120. 2025Claude 3.7was influenced by o1
    built directly on Claude 3.5 Sonnet    Chapter 12. Thinking longer
  121. 2025Claude Codewas influenced by Computer useChapter 13. Models that act
  122. 2025GPT-4.5was influenced by Scaling lawsChapter 14. Where the threads meet now
  123. 2025Gemini 2.5was influenced by Gemini 1.5Chapter 12. Thinking longer
  124. 2025Llama 4was influenced by Switch TransformerChapter 8. The weights get out
  125. 2025o3 releasedbuilt directly on o1Chapter 12. Thinking longer
  126. 2025Qwen3built directly on Qwen2Chapter 8. The weights get out
  127. 2025Claude 4succeeded Claude 3
    was influenced by SWE-bench    Chapter 13. Models that act
  128. 2025Kimi K2was influenced by DeepSeekMoE
    was influenced by GRPO    Chapter 8. The weights get out
  129. 2025IMO goldbuilt directly on Gemini 2.5Chapter 12. Thinking longer
  130. 2025gpt-ossbuilt directly on o3 releasedChapter 8. The weights get out
  131. 2025GPT-5succeeded GPT-4oChapter 14. Where the threads meet now
  132. 2025DeepSeek-V3.2was influenced by Sparse TransformerChapter 10. Reading more than you remember
  133. 2025Claude Sonnet 4.5built directly on Claude 4Chapter 13. Models that act
  134. 2025IMO-ProofBenchwas influenced by IMO goldChapter 12. Thinking longer
  135. 2025GPT-5.1built directly on GPT-5Chapter 14. Where the threads meet now
  136. 2025Gemini 3was influenced by Gemini 1.5Chapter 14. Where the threads meet now
  137. 2025Claude Opus 4.5was influenced by SWE-benchChapter 13. Models that act
  138. 2025GPT-5.2built directly on GPT-5.1Chapter 14. Where the threads meet now
  139. 2026Claude Opus 4.6built directly on Claude Opus 4.5Chapter 14. Where the threads meet now
  140. 2026GPT-5.3-Codexbuilt directly on GPT-5.2Chapter 13. Models that act
  141. 2026Gemini 3 Deep Thinkbuilt directly on Gemini 3Chapter 12. Thinking longer
  142. 2026Qwen3.5built directly on Qwen3Chapter 8. The weights get out
  143. 2026Gemini 3.1built directly on Gemini 3Chapter 14. Where the threads meet now
  144. 2026GPT-5.4built directly on GPT-5.3-CodexChapter 14. Where the threads meet now
  145. 2026Muse Sparksucceeded Llama 4Chapter 8. The weights get out
  146. 2026Claude Opus 4.7built directly on Claude Opus 4.6Chapter 14. Where the threads meet now
  147. 2026GPT-5.5built directly on GPT-5.4Chapter 14. Where the threads meet now
  148. 2026DeepSeek-V4built directly on DeepSeek-V3.2Chapter 8. The weights get out
  149. 2026Claude Opus 4.8built directly on Claude Opus 4.7Chapter 14. Where the threads meet now
  150. 2026Claude Sonnet 5succeeded Claude Opus 4.8Chapter 14. Where the threads meet now
  151. 2026GPT-5.6 Solbuilt directly on GPT-5.5Chapter 14. Where the threads meet now
  152. 2026Kimi K3built directly on Kimi K2Chapter 8. The weights get out
  153. 2026Claude Opus 5succeeded Claude Opus 4.8Chapter 14. Where the threads meet now
  154. 2026Qwen3.8built directly on Qwen3.5Chapter 8. The weights get out
  155. 2026Claude Opus 5.5built directly on Claude Opus 5Chapter 14. Where the threads meet now
  1. 2014GRU / encoder-decoderThis milestone.
  2. 2014Attention Turning pointbuilt directly on GRU / encoder-decoderChapter 3. Attention, then only attention
  3. 2015Luong attentionbuilt directly on AttentionChapter 3. Attention, then only attention
  4. 2016GNMTbuilt directly on AttentionChapter 2. Words become vectors
  5. 2017ConvS2Sbuilt directly on AttentionChapter 3. Attention, then only attention
  6. 2017Transformer Turning pointbuilt directly on AttentionChapter 3. Attention, then only attention
  7. 2018GPT-1built directly on TransformerChapter 4. Pretrain once, use everywhere
  8. 2018BERT Turning pointbuilt directly on TransformerChapter 4. Pretrain once, use everywhere
  9. 2019Transformer-XLbuilt directly on TransformerChapter 4. Pretrain once, use everywhere
  10. 2019GPT-2 Turning pointbuilt directly on TransformerChapter 4. Pretrain once, use everywhere
  11. 2019WebTextwas influenced by GPT-1Chapter 4. Pretrain once, use everywhere
  12. 2019Sparse Transformer Turning pointbuilt directly on TransformerChapter 4. Pretrain once, use everywhere
  13. 2019RoBERTabuilt directly on BERTChapter 4. Pretrain once, use everywhere
  14. 2019Sentence-BERTbuilt directly on BERTChapter 10. Reading more than you remember
  15. 2019Megatron-LMbuilt directly on TransformerChapter 4. Pretrain once, use everywhere
  16. 2019RLHF for LMsbuilt directly on GPT-2Chapter 6. Teaching models what people want
  17. 2019T5 Turning pointbuilt directly on TransformerChapter 4. Pretrain once, use everywhere
  18. 2019C4 Turning pointwas influenced by WebText (indirect influence)Chapter 4. Pretrain once, use everywhere
  19. 2019Multi-query attentionbuilt directly on TransformerChapter 9. Making it run anywhere
  20. 2020Scaling laws Turning pointbuilt directly on TransformerChapter 5. Scale becomes a science
  21. 2020Longformerbuilt directly on TransformerChapter 10. Reading more than you remember
  22. 2020Dense retrievalbuilt directly on BERTChapter 10. Reading more than you remember
  23. 2020RAGbuilt directly on Dense retrievalChapter 10. Reading more than you remember
  24. 2020GPT-3 Turning pointwas influenced by Scaling laws
    built directly on GPT-2
    built directly on Sparse Transformer    Chapter 5. Scale becomes a science
  25. 2020BigBirdbuilt directly on TransformerChapter 10. Reading more than you remember
  26. 2020RLHF for summarisation Turning pointbuilt directly on RLHF for LMsChapter 6. Teaching models what people want
  27. 2020Vision Transformerbuilt directly on TransformerChapter 11. Seeing and hearing
  28. 2020The Pilewas influenced by GPT-3Chapter 5. Scale becomes a science
  29. 2021Switch Transformerbuilt directly on T5Chapter 5. Scale becomes a science
  30. 2021CLIPbuilt directly on GPT-2
    built directly on Vision Transformer    Chapter 11. Seeing and hearing
  31. 2021RoPE Turning pointbuilt directly on TransformerChapter 10. Reading more than you remember
  32. 2021LoRAbuilt directly on TransformerChapter 9. Making it run anywhere
  33. 2021Copilotwas influenced by Codex / HumanEvalChapter 7. The assistant goes public
  34. 2021Codex / HumanEvalbuilt directly on GPT-3Chapter 13. Models that act
  35. 2021ALiBibuilt directly on TransformerChapter 10. Reading more than you remember
  36. 2021FLANwas influenced by T5Chapter 6. Teaching models what people want
  37. 2021T0built directly on T5Chapter 6. Teaching models what people want
  38. 2021HHH assistantwas influenced by RLHF for summarisationChapter 6. Teaching models what people want
  39. 2021RETROwas influenced by Dense retrievalChapter 10. Reading more than you remember
  40. 2021WebGPTbuilt directly on GPT-3Chapter 13. Models that act
  41. 2022MT-NLGwas influenced by Scaling laws
    built directly on Megatron-LM    Chapter 5. Scale becomes a science
  42. 2022Chain of thought Turning pointwas influenced by GPT-3Chapter 12. Thinking longer
  43. 2022InstructGPT Turning pointbuilt directly on GPT-3
    was influenced by FLAN    Chapter 6. Teaching models what people want
  44. 2022Self-consistencybuilt directly on Chain of thoughtChapter 12. Thinking longer
  45. 2022STaRwas influenced by Chain of thoughtChapter 12. Thinking longer
  46. 2022Chinchilla Turning pointbuilt directly on Scaling lawsChapter 5. Scale becomes a science
  47. 2022PaLMwas influenced by Chain of thoughtChapter 5. Scale becomes a science
  48. 2022HH-RLHFbuilt directly on RLHF for summarisationChapter 6. Teaching models what people want
  49. 2022Flamingobuilt directly on Vision TransformerChapter 11. Seeing and hearing
  50. 2022OPT-175Bwas influenced by Megatron-LMChapter 8. The weights get out
  51. 2022Zero-shot CoTwas influenced by GPT-3Chapter 12. Thinking longer
  52. 2022FlashAttentionbuilt directly on TransformerChapter 9. Making it run anywhere
  53. 2022Emergent abilitiesbuilt directly on Scaling lawsChapter 5. Scale becomes a science
  54. 2022BLOOMbuilt directly on ALiBiChapter 8. The weights get out
  55. 2022LLM.int8()was influenced by OPT-175BChapter 9. Making it run anywhere
  56. 2022Whisperbuilt directly on TransformerChapter 11. Seeing and hearing
  57. 2022ReActbuilt directly on Chain of thought
    was influenced by WebGPT    Chapter 13. Models that act
  58. 2022Flan-T5 / Flan-PaLMbuilt directly on T5Chapter 6. Teaching models what people want
  59. 2022GPTQwas influenced by LLM.int8()Chapter 9. Making it run anywhere
  60. 2022Speculative decodingwas influenced by TransformerChapter 9. Making it run anywhere
  61. 2022GPT-3.5 Turning pointbuilt directly on GPT-3Chapter 7. The assistant goes public
  62. 2022ChatGPTbuilt directly on InstructGPT
    built directly on GPT-3.5    Chapter 7. The assistant goes public
  63. 2022Constitutional AIwas influenced by Chain of thoughtChapter 6. Teaching models what people want
  64. 2022Self-Instructbuilt directly on GPT-3
    was influenced by FLAN    Chapter 6. Teaching models what people want
  65. 2023Toolformerbuilt directly on GPT-3Chapter 13. Models that act
  66. 2023LLaMA Turning pointbuilt directly on RoPEChapter 8. The weights get out
  67. 2023llama.cppwas influenced by LLaMAChapter 9. Making it run anywhere
  68. 2023Alpacabuilt directly on LLaMAChapter 8. The weights get out
  69. 2023Claudebuilt directly on Constitutional AI
    was influenced by HH-RLHF    Chapter 7. The assistant goes public
  70. 2023GPT-4 Turning pointbuilt directly on Scaling lawsChapter 7. The assistant goes public
  71. 2023Reflexionbuilt directly on ReActChapter 13. Models that act
  72. 2023AutoGPTwas influenced by GPT-4Chapter 13. Models that act
  73. 2023Pythiawas influenced by Scaling lawsChapter 8. The weights get out
  74. 2023LLaVAbuilt directly on GPT-4
    built directly on LLaMA
    built directly on CLIP    Chapter 11. Seeing and hearing
  75. 2023Emergence miragesucceeded Emergent abilitiesChapter 5. Scale becomes a science
  76. 2023100K contextbuilt directly on ClaudeChapter 10. Reading more than you remember
  77. 2023Tree of thoughtsbuilt directly on Chain of thoughtChapter 12. Thinking longer
  78. 2023Grouped-query attention Turning pointbuilt directly on Multi-query attentionChapter 9. Making it run anywhere
  79. 2023QLoRAbuilt directly on LoRAChapter 9. Making it run anywhere
  80. 2023Falconbuilt directly on RefinedWebChapter 8. The weights get out
  81. 2023DPObuilt directly on RLHF for summarisation
    built directly on InstructGPT    Chapter 6. Teaching models what people want
  82. 2023Process supervisionbuilt directly on Chain of thoughtChapter 12. Thinking longer
  83. 2023RefinedWebwas influenced by C4Chapter 8. The weights get out
  84. 2023Function callingwas influenced by Toolformer (indirect influence)Chapter 13. Models that act
  85. 2023phi-1was influenced by GPT-3.5
    was influenced by Self-Instruct (indirect influence)    Chapter 8. The weights get out
  86. 2023Position interpolationbuilt directly on RoPEChapter 10. Reading more than you remember
  87. 2023Lost in the middlewas influenced by RAGChapter 10. Reading more than you remember
  88. 2023Llama 2 Turning pointbuilt directly on LLaMA
    built directly on Grouped-query attention    Chapter 8. The weights get out
  89. 2023vLLMwas influenced by Multi-query attentionChapter 9. Making it run anywhere
  90. 2023GPT-4Vbuilt directly on GPT-4
    was influenced by CLIP (indirect influence)    Chapter 11. Seeing and hearing
  91. 2023Mistral 7Bwas influenced by Sparse Transformer
    was influenced by Longformer
    was influenced by FlashAttention    Chapter 8. The weights get out
  92. 2023SWE-benchwas influenced by GPT-4Chapter 13. Models that act
  93. 2023Mambawas influenced by FlashAttentionChapter 9. Making it run anywhere
  94. 2023Gemini 1.0was influenced by Vision TransformerChapter 11. Seeing and hearing
  95. 2023Mixtral 8x7Bbuilt directly on Switch Transformer
    built directly on Mistral 7B    Chapter 8. The weights get out
  96. 2024DeepSeek LLMbuilt directly on Scaling lawsChapter 8. The weights get out
  97. 2024DeepSeekMoEwas influenced by Switch TransformerChapter 8. The weights get out
  98. 2024Dolmawas influenced by C4
    was influenced by The Pile    Chapter 8. The weights get out
  99. 2024OLMowas influenced by PythiaChapter 8. The weights get out
  100. 2024GRPObuilt directly on DeepSeek LLMChapter 12. Thinking longer
  101. 2024Gemini 1.5built directly on Gemini 1.0Chapter 10. Reading more than you remember
  102. 2024Gemmabuilt directly on Gemini 1.0
    was influenced by Mistral 7B    Chapter 8. The weights get out
  103. 2024Claude 3was influenced by GPT-4Chapter 14. Where the threads meet now
  104. 2024Chatbot Arenawas influenced by ChatGPTChapter 14. Where the threads meet now
  105. 2024Devinwas influenced by GPT-4Chapter 13. Models that act
  106. 2024Llama 3built directly on Grouped-query attention
    was influenced by Chinchilla    Chapter 8. The weights get out
  107. 2024Phi-3built directly on phi-1Chapter 8. The weights get out
  108. 2024DeepSeek-V2 / MLAbuilt directly on RoPE
    was influenced by Multi-query attention    Chapter 9. Making it run anywhere
  109. 2024SWE-agentbuilt directly on SWE-benchChapter 13. Models that act
  110. 2024GPT-4owas influenced by Whisper (indirect influence)Chapter 11. Seeing and hearing
  111. 2024Claude 3.5 Sonnetbuilt directly on Claude 3Chapter 14. Where the threads meet now
  112. 2024FineWebwas influenced by The PileChapter 8. The weights get out
  113. 2024FlashAttention-3built directly on FlashAttentionChapter 9. Making it run anywhere
  114. 2024Qwen2built directly on RoPEChapter 8. The weights get out
  115. 2024Llama 3.1 405Bwas influenced by Chinchilla
    built directly on Grouped-query attention
    was influenced by Position interpolation    Chapter 8. The weights get out
  116. 2024Test-time computewas influenced by Process supervision
    was influenced by Tree of thoughts (indirect influence)
    was influenced by STaR (indirect influence)    Chapter 12. Thinking longer
  117. 2024o1 Turning pointwas influenced by Chain of thoughtChapter 12. Thinking longer
  118. 2024Computer usewas influenced by GPT-4VChapter 13. Models that act
  119. 2024MCPwas influenced by Toolformer (indirect influence)Chapter 13. Models that act
  120. 2024Gemini 2.0built directly on Gemini 1.5Chapter 14. Where the threads meet now
  121. 2024o3 announcedbuilt directly on o1Chapter 12. Thinking longer
  122. 2024DeepSeek-V3built directly on DeepSeek-V2 / MLA
    was influenced by GPT-4o    Chapter 8. The weights get out
  123. 2025DeepSeek-R1was influenced by Chain of thought
    built directly on GRPO
    built directly on DeepSeek-V3    Chapter 12. Thinking longer
  124. 2025Deep researchwas influenced by WebGPT (indirect influence)Chapter 13. Models that act
  125. 2025Claude 3.7was influenced by o1
    built directly on Claude 3.5 Sonnet    Chapter 12. Thinking longer
  126. 2025Claude Codewas influenced by Computer use
    was influenced by SWE-agent (indirect influence)
    was influenced by MCP    Chapter 13. Models that act
  127. 2025GPT-4.5was influenced by Scaling lawsChapter 14. Where the threads meet now
  128. 2025Gemini 2.5was influenced by Gemini 1.5Chapter 12. Thinking longer
  129. 2025Llama 4was influenced by Switch TransformerChapter 8. The weights get out
  130. 2025o3 releasedbuilt directly on o1
    was influenced by Function calling    Chapter 12. Thinking longer
  131. 2025Qwen3built directly on Qwen2Chapter 8. The weights get out
  132. 2025Claude 4succeeded Claude 3
    was influenced by SWE-bench    Chapter 13. Models that act
  133. 2025Kimi K2was influenced by DeepSeekMoE
    was influenced by DeepSeek-V3 (indirect influence)
    was influenced by GRPO    Chapter 8. The weights get out
  134. 2025IMO goldbuilt directly on Gemini 2.5Chapter 12. Thinking longer
  135. 2025gpt-osswas influenced by DeepSeek-R1 (indirect influence)Chapter 8. The weights get out
  136. 2025GPT-5succeeded GPT-4oChapter 14. Where the threads meet now
  137. 2025GLM-4.5was influenced by DeepSeek-R1 (indirect influence)Chapter 8. The weights get out
  138. 2025DeepSeek-V3.2was influenced by Sparse TransformerChapter 10. Reading more than you remember
  139. 2025Claude Sonnet 4.5built directly on Claude 4Chapter 13. Models that act
  140. 2025IMO-ProofBenchwas influenced by IMO goldChapter 12. Thinking longer
  141. 2025GPT-5.1built directly on GPT-5Chapter 14. Where the threads meet now
  142. 2025Gemini 3was influenced by Gemini 1.5Chapter 14. Where the threads meet now
  143. 2025Claude Opus 4.5was influenced by SWE-benchChapter 13. Models that act
  144. 2025GPT-5.2built directly on GPT-5.1Chapter 14. Where the threads meet now
  145. 2026GPT-5.3-Codexwas influenced by Computer use (indirect influence)Chapter 13. Models that act
  146. 2026Claude Opus 4.6built directly on Claude Opus 4.5Chapter 14. Where the threads meet now
  147. 2026Gemini 3 Deep Thinkbuilt directly on Gemini 3Chapter 12. Thinking longer
  148. 2026Qwen3.5built directly on Qwen3Chapter 8. The weights get out
  149. 2026Gemini 3.1built directly on Gemini 3Chapter 14. Where the threads meet now
  150. 2026GPT-5.4built directly on GPT-5.3-CodexChapter 14. Where the threads meet now
  151. 2026Project Glasswingwas influenced by Computer use (indirect influence)Chapter 14. Where the threads meet now
  152. 2026Muse Sparksucceeded Llama 4Chapter 8. The weights get out
  153. 2026Claude Opus 4.7built directly on Claude Opus 4.6Chapter 14. Where the threads meet now
  154. 2026GPT-5.5built directly on GPT-5.4Chapter 14. Where the threads meet now
  155. 2026DeepSeek-V4built directly on DeepSeek-V3.2Chapter 8. The weights get out
  156. 2026Claude Opus 4.8built directly on Claude Opus 4.7Chapter 14. Where the threads meet now
  157. 2026Claude Sonnet 5succeeded Claude Opus 4.8Chapter 14. Where the threads meet now
  158. 2026Fable 5 / Mythos 5was influenced by Project GlasswingChapter 14. Where the threads meet now
  159. 2026GLM-5.2built directly on GLM-4.5Chapter 8. The weights get out
  160. 2026GPT-5.6 Solbuilt directly on GPT-5.5Chapter 14. Where the threads meet now
  161. 2026Kimi K3built directly on Kimi K2Chapter 8. The weights get out
  162. 2026Claude Opus 5succeeded Claude Opus 4.8Chapter 14. Where the threads meet now
  163. 2026Qwen3.8built directly on Qwen3.5Chapter 8. The weights get out
  164. 2026Claude Opus 5.5built directly on Claude Opus 5Chapter 14. Where the threads meet now

Direct links and their sources

Where it came from

Built directly on

What it led to

Built on directly by

Developed alongside

Terms