NLP Lineage

December 2015 · Efficiency and infrastructure

Residual connections

Deep Residual Learning for Image Recognition

K. He, X. Zhang, S. Ren, J. Sun

Date
December 2015
What the date marks
arXiv submission
Kind
Architectural idea
Organisation
Microsoft Research
Access
Not applicable
Other dates
CVPR 2016
Threads
In the guided tour
Chapter 3. Attention, then only attention
Learning phases
1. Transformer architecture

A milestone: 2 later milestones descend directly from it, and its descendants reach 10 of the 10 threads. How levels are decided

Read the primary source (arxiv.org)

Full summary in progress. Until then, this page shows the milestone's record from the research data.

What it changed

Before it

Adding layers past a certain depth made networks worse, not better.

The problem

Gradients degrade through long stacks of transformations.

What it introduced

Add the input of a block to its output, so the block only has to learn a correction.

Why it matters

A vision paper, but every transformer block is wrapped in residual connections. Without them, depth does not train.

What modern models inherit

The skip connection around every attention and feed-forward sublayer.

What came next

Adopted wholesale by the Transformer and everything after it.

Before and after

The milestones it descends from, and the milestones that descend from it. Work developed alongside it is not counted. Weak links are left out unless you include them.

Before: where it came from

No documented ancestors are recorded for Residual connections without weak links.

No documented ancestors are recorded for Residual connections without weak links.

No recorded ancestors are recorded for Residual connections.

No recorded ancestors are recorded for Residual connections.

After: what it led to

Residual connections has 151 documented descendants across 12 chapters, reaching forward to Claude Opus 5.5 in 2026. Showing its direct descendants and the 18 turning points among them.

Residual connections has 151 documented descendants across 12 chapters, reaching forward to Claude Opus 5.5 in 2026. Showing all of them.

Residual connections has 160 recorded descendants, including indirect ones, across 12 chapters, reaching forward to Claude Opus 5.5 in 2026. Showing its direct descendants and the 19 turning points among them.

Residual connections has 160 recorded descendants, including indirect ones, across 12 chapters, reaching forward to Claude Opus 5.5 in 2026. Showing all of them.

  1. 2015Residual connectionsThis milestone.
  2. 2017ConvS2Sbuilt directly on Residual connectionsChapter 3. Attention, then only attention
  3. 2017Transformer Turning pointwas influenced by Residual connectionsChapter 3. Attention, then only attention
  4. 2018GPT-1built directly on TransformerChapter 4. Pretrain once, use everywhere
  5. 2018BERT Turning pointbuilt directly on TransformerChapter 4. Pretrain once, use everywhere
  6. 2019Transformer-XLbuilt directly on TransformerChapter 4. Pretrain once, use everywhere
  7. 2019GPT-2 Turning pointbuilt directly on TransformerChapter 4. Pretrain once, use everywhere
  8. 2019WebTextwas influenced by GPT-1Chapter 4. Pretrain once, use everywhere
  9. 2019Sparse Transformer Turning pointbuilt directly on TransformerChapter 4. Pretrain once, use everywhere
  10. 2019RoBERTabuilt directly on BERTChapter 4. Pretrain once, use everywhere
  11. 2019Sentence-BERTbuilt directly on BERTChapter 10. Reading more than you remember
  12. 2019Megatron-LMbuilt directly on TransformerChapter 4. Pretrain once, use everywhere
  13. 2019RLHF for LMsbuilt directly on GPT-2Chapter 6. Teaching models what people want
  14. 2019T5 Turning pointbuilt directly on TransformerChapter 4. Pretrain once, use everywhere
  15. 2019Multi-query attentionbuilt directly on TransformerChapter 9. Making it run anywhere
  16. 2020Scaling laws Turning pointbuilt directly on TransformerChapter 5. Scale becomes a science
  17. 2020Longformerbuilt directly on TransformerChapter 10. Reading more than you remember
  18. 2020Dense retrievalbuilt directly on BERTChapter 10. Reading more than you remember
  19. 2020RAGbuilt directly on Dense retrievalChapter 10. Reading more than you remember
  20. 2020GPT-3 Turning pointwas influenced by Scaling laws
    built directly on GPT-2
    built directly on Sparse Transformer    Chapter 5. Scale becomes a science
  21. 2020BigBirdbuilt directly on TransformerChapter 10. Reading more than you remember
  22. 2020RLHF for summarisation Turning pointbuilt directly on RLHF for LMsChapter 6. Teaching models what people want
  23. 2020Vision Transformerbuilt directly on TransformerChapter 11. Seeing and hearing
  24. 2020The Pilewas influenced by GPT-3Chapter 5. Scale becomes a science
  25. 2021Switch Transformerbuilt directly on T5Chapter 5. Scale becomes a science
  26. 2021CLIPbuilt directly on GPT-2
    built directly on Vision Transformer    Chapter 11. Seeing and hearing
  27. 2021RoPE Turning pointbuilt directly on TransformerChapter 10. Reading more than you remember
  28. 2021LoRAbuilt directly on TransformerChapter 9. Making it run anywhere
  29. 2021Copilotwas influenced by Codex / HumanEvalChapter 7. The assistant goes public
  30. 2021Codex / HumanEvalbuilt directly on GPT-3Chapter 13. Models that act
  31. 2021ALiBibuilt directly on TransformerChapter 10. Reading more than you remember
  32. 2021FLANwas influenced by T5Chapter 6. Teaching models what people want
  33. 2021T0built directly on T5Chapter 6. Teaching models what people want
  34. 2021HHH assistantwas influenced by RLHF for summarisationChapter 6. Teaching models what people want
  35. 2021RETROwas influenced by Dense retrievalChapter 10. Reading more than you remember
  36. 2021WebGPTbuilt directly on GPT-3Chapter 13. Models that act
  37. 2022MT-NLGwas influenced by Scaling laws
    built directly on Megatron-LM    Chapter 5. Scale becomes a science
  38. 2022Chain of thought Turning pointwas influenced by GPT-3Chapter 12. Thinking longer
  39. 2022InstructGPT Turning pointbuilt directly on GPT-3
    was influenced by FLAN    Chapter 6. Teaching models what people want
  40. 2022Self-consistencybuilt directly on Chain of thoughtChapter 12. Thinking longer
  41. 2022STaRwas influenced by Chain of thoughtChapter 12. Thinking longer
  42. 2022Chinchilla Turning pointbuilt directly on Scaling lawsChapter 5. Scale becomes a science
  43. 2022PaLMwas influenced by Chain of thoughtChapter 5. Scale becomes a science
  44. 2022HH-RLHFbuilt directly on RLHF for summarisationChapter 6. Teaching models what people want
  45. 2022Flamingobuilt directly on Vision TransformerChapter 11. Seeing and hearing
  46. 2022OPT-175Bwas influenced by Megatron-LMChapter 8. The weights get out
  47. 2022Zero-shot CoTwas influenced by GPT-3Chapter 12. Thinking longer
  48. 2022FlashAttentionbuilt directly on TransformerChapter 9. Making it run anywhere
  49. 2022Emergent abilitiesbuilt directly on Scaling lawsChapter 5. Scale becomes a science
  50. 2022BLOOMbuilt directly on ALiBiChapter 8. The weights get out
  51. 2022LLM.int8()was influenced by OPT-175BChapter 9. Making it run anywhere
  52. 2022Whisperbuilt directly on TransformerChapter 11. Seeing and hearing
  53. 2022ReActbuilt directly on Chain of thought
    was influenced by WebGPT    Chapter 13. Models that act
  54. 2022Flan-T5 / Flan-PaLMbuilt directly on T5Chapter 6. Teaching models what people want
  55. 2022GPTQwas influenced by LLM.int8()Chapter 9. Making it run anywhere
  56. 2022Speculative decodingwas influenced by TransformerChapter 9. Making it run anywhere
  57. 2022GPT-3.5 Turning pointbuilt directly on GPT-3Chapter 7. The assistant goes public
  58. 2022ChatGPTbuilt directly on InstructGPT
    built directly on GPT-3.5    Chapter 7. The assistant goes public
  59. 2022Constitutional AIwas influenced by Chain of thoughtChapter 6. Teaching models what people want
  60. 2022Self-Instructbuilt directly on GPT-3
    was influenced by FLAN    Chapter 6. Teaching models what people want
  61. 2023Toolformerbuilt directly on GPT-3Chapter 13. Models that act
  62. 2023LLaMA Turning pointbuilt directly on RoPEChapter 8. The weights get out
  63. 2023llama.cppwas influenced by LLaMAChapter 9. Making it run anywhere
  64. 2023Alpacabuilt directly on LLaMAChapter 8. The weights get out
  65. 2023Claudebuilt directly on Constitutional AI
    was influenced by HH-RLHF    Chapter 7. The assistant goes public
  66. 2023GPT-4 Turning pointbuilt directly on Scaling lawsChapter 7. The assistant goes public
  67. 2023Reflexionbuilt directly on ReActChapter 13. Models that act
  68. 2023AutoGPTwas influenced by GPT-4Chapter 13. Models that act
  69. 2023Pythiawas influenced by Scaling lawsChapter 8. The weights get out
  70. 2023LLaVAbuilt directly on GPT-4
    built directly on LLaMA
    built directly on CLIP    Chapter 11. Seeing and hearing
  71. 2023Emergence miragesucceeded Emergent abilitiesChapter 5. Scale becomes a science
  72. 2023100K contextbuilt directly on ClaudeChapter 10. Reading more than you remember
  73. 2023Tree of thoughtsbuilt directly on Chain of thoughtChapter 12. Thinking longer
  74. 2023Grouped-query attention Turning pointbuilt directly on Multi-query attentionChapter 9. Making it run anywhere
  75. 2023QLoRAbuilt directly on LoRAChapter 9. Making it run anywhere
  76. 2023DPObuilt directly on RLHF for summarisation
    built directly on InstructGPT    Chapter 6. Teaching models what people want
  77. 2023Process supervisionbuilt directly on Chain of thoughtChapter 12. Thinking longer
  78. 2023phi-1was influenced by GPT-3.5Chapter 8. The weights get out
  79. 2023Position interpolationbuilt directly on RoPEChapter 10. Reading more than you remember
  80. 2023Lost in the middlewas influenced by RAGChapter 10. Reading more than you remember
  81. 2023Llama 2 Turning pointbuilt directly on LLaMA
    built directly on Grouped-query attention    Chapter 8. The weights get out
  82. 2023vLLMwas influenced by Multi-query attentionChapter 9. Making it run anywhere
  83. 2023GPT-4Vbuilt directly on GPT-4Chapter 11. Seeing and hearing
  84. 2023Mistral 7Bwas influenced by Sparse Transformer
    was influenced by Longformer
    was influenced by FlashAttention    Chapter 8. The weights get out
  85. 2023SWE-benchwas influenced by GPT-4Chapter 13. Models that act
  86. 2023Mambawas influenced by FlashAttentionChapter 9. Making it run anywhere
  87. 2023Gemini 1.0was influenced by Vision TransformerChapter 11. Seeing and hearing
  88. 2023Mixtral 8x7Bbuilt directly on Switch Transformer
    built directly on Mistral 7B    Chapter 8. The weights get out
  89. 2024DeepSeek LLMbuilt directly on Scaling lawsChapter 8. The weights get out
  90. 2024DeepSeekMoEwas influenced by Switch TransformerChapter 8. The weights get out
  91. 2024Dolmawas influenced by The PileChapter 8. The weights get out
  92. 2024OLMowas influenced by PythiaChapter 8. The weights get out
  93. 2024GRPObuilt directly on DeepSeek LLMChapter 12. Thinking longer
  94. 2024Gemini 1.5built directly on Gemini 1.0Chapter 10. Reading more than you remember
  95. 2024Gemmabuilt directly on Gemini 1.0
    was influenced by Mistral 7B    Chapter 8. The weights get out
  96. 2024Claude 3was influenced by GPT-4Chapter 14. Where the threads meet now
  97. 2024Chatbot Arenawas influenced by ChatGPTChapter 14. Where the threads meet now
  98. 2024Devinwas influenced by GPT-4Chapter 13. Models that act
  99. 2024Llama 3built directly on Grouped-query attention
    was influenced by Chinchilla    Chapter 8. The weights get out
  100. 2024Phi-3built directly on phi-1Chapter 8. The weights get out
  101. 2024DeepSeek-V2 / MLAbuilt directly on RoPE
    was influenced by Multi-query attention    Chapter 9. Making it run anywhere
  102. 2024SWE-agentbuilt directly on SWE-benchChapter 13. Models that act
  103. 2024GPT-4obuilt directly on GPT-4Chapter 11. Seeing and hearing
  104. 2024Claude 3.5 Sonnetbuilt directly on Claude 3Chapter 14. Where the threads meet now
  105. 2024FineWebwas influenced by The PileChapter 8. The weights get out
  106. 2024FlashAttention-3built directly on FlashAttentionChapter 9. Making it run anywhere
  107. 2024Qwen2built directly on RoPEChapter 8. The weights get out
  108. 2024Llama 3.1 405Bwas influenced by Chinchilla
    built directly on Grouped-query attention
    was influenced by Position interpolation    Chapter 8. The weights get out
  109. 2024Test-time computewas influenced by Process supervision
    built directly on Self-consistency    Chapter 12. Thinking longer
  110. 2024o1 Turning pointwas influenced by Chain of thoughtChapter 12. Thinking longer
  111. 2024Computer usewas influenced by GPT-4VChapter 13. Models that act
  112. 2024Gemini 2.0built directly on Gemini 1.5Chapter 14. Where the threads meet now
  113. 2024o3 announcedbuilt directly on o1Chapter 12. Thinking longer
  114. 2024DeepSeek-V3built directly on DeepSeek-V2 / MLAChapter 8. The weights get out
  115. 2025DeepSeek-R1was influenced by Chain of thought
    built directly on GRPO
    built directly on DeepSeek-V3    Chapter 12. Thinking longer
  116. 2025Deep researchwas influenced by o3 releasedChapter 13. Models that act
  117. 2025Claude 3.7was influenced by o1
    built directly on Claude 3.5 Sonnet    Chapter 12. Thinking longer
  118. 2025Claude Codewas influenced by Computer useChapter 13. Models that act
  119. 2025GPT-4.5was influenced by Scaling lawsChapter 14. Where the threads meet now
  120. 2025Gemini 2.5was influenced by Gemini 1.5Chapter 12. Thinking longer
  121. 2025Llama 4was influenced by Switch TransformerChapter 8. The weights get out
  122. 2025o3 releasedbuilt directly on o1Chapter 12. Thinking longer
  123. 2025Qwen3built directly on Qwen2Chapter 8. The weights get out
  124. 2025Claude 4succeeded Claude 3
    was influenced by SWE-bench    Chapter 13. Models that act
  125. 2025Kimi K2was influenced by DeepSeekMoE
    was influenced by GRPO    Chapter 8. The weights get out
  126. 2025IMO goldbuilt directly on Gemini 2.5Chapter 12. Thinking longer
  127. 2025gpt-ossbuilt directly on o3 releasedChapter 8. The weights get out
  128. 2025GPT-5succeeded GPT-4oChapter 14. Where the threads meet now
  129. 2025DeepSeek-V3.2was influenced by Sparse TransformerChapter 10. Reading more than you remember
  130. 2025Claude Sonnet 4.5built directly on Claude 4Chapter 13. Models that act
  131. 2025IMO-ProofBenchwas influenced by IMO goldChapter 12. Thinking longer
  132. 2025GPT-5.1built directly on GPT-5Chapter 14. Where the threads meet now
  133. 2025Gemini 3was influenced by Gemini 1.5Chapter 14. Where the threads meet now
  134. 2025Claude Opus 4.5was influenced by SWE-benchChapter 13. Models that act
  135. 2025GPT-5.2built directly on GPT-5.1Chapter 14. Where the threads meet now
  136. 2026Claude Opus 4.6built directly on Claude Opus 4.5Chapter 14. Where the threads meet now
  137. 2026GPT-5.3-Codexbuilt directly on GPT-5.2Chapter 13. Models that act
  138. 2026Gemini 3 Deep Thinkbuilt directly on Gemini 3Chapter 12. Thinking longer
  139. 2026Qwen3.5built directly on Qwen3Chapter 8. The weights get out
  140. 2026Gemini 3.1built directly on Gemini 3Chapter 14. Where the threads meet now
  141. 2026GPT-5.4built directly on GPT-5.3-CodexChapter 14. Where the threads meet now
  142. 2026Muse Sparksucceeded Llama 4Chapter 8. The weights get out
  143. 2026Claude Opus 4.7built directly on Claude Opus 4.6Chapter 14. Where the threads meet now
  144. 2026GPT-5.5built directly on GPT-5.4Chapter 14. Where the threads meet now
  145. 2026DeepSeek-V4built directly on DeepSeek-V3.2Chapter 8. The weights get out
  146. 2026Claude Opus 4.8built directly on Claude Opus 4.7Chapter 14. Where the threads meet now
  147. 2026Claude Sonnet 5succeeded Claude Opus 4.8Chapter 14. Where the threads meet now
  148. 2026GPT-5.6 Solbuilt directly on GPT-5.5Chapter 14. Where the threads meet now
  149. 2026Kimi K3built directly on Kimi K2Chapter 8. The weights get out
  150. 2026Claude Opus 5succeeded Claude Opus 4.8Chapter 14. Where the threads meet now
  151. 2026Qwen3.8built directly on Qwen3.5Chapter 8. The weights get out
  152. 2026Claude Opus 5.5built directly on Claude Opus 5Chapter 14. Where the threads meet now
  1. 2015Residual connectionsThis milestone.
  2. 2017ConvS2Sbuilt directly on Residual connectionsChapter 3. Attention, then only attention
  3. 2017Transformer Turning pointwas influenced by Residual connectionsChapter 3. Attention, then only attention
  4. 2018GPT-1built directly on TransformerChapter 4. Pretrain once, use everywhere
  5. 2018BERT Turning pointbuilt directly on TransformerChapter 4. Pretrain once, use everywhere
  6. 2019Transformer-XLbuilt directly on TransformerChapter 4. Pretrain once, use everywhere
  7. 2019GPT-2 Turning pointbuilt directly on TransformerChapter 4. Pretrain once, use everywhere
  8. 2019WebTextwas influenced by GPT-1Chapter 4. Pretrain once, use everywhere
  9. 2019Sparse Transformer Turning pointbuilt directly on TransformerChapter 4. Pretrain once, use everywhere
  10. 2019RoBERTabuilt directly on BERTChapter 4. Pretrain once, use everywhere
  11. 2019Sentence-BERTbuilt directly on BERTChapter 10. Reading more than you remember
  12. 2019Megatron-LMbuilt directly on TransformerChapter 4. Pretrain once, use everywhere
  13. 2019RLHF for LMsbuilt directly on GPT-2Chapter 6. Teaching models what people want
  14. 2019T5 Turning pointbuilt directly on TransformerChapter 4. Pretrain once, use everywhere
  15. 2019C4 Turning pointwas influenced by WebText (indirect influence)Chapter 4. Pretrain once, use everywhere
  16. 2019Multi-query attentionbuilt directly on TransformerChapter 9. Making it run anywhere
  17. 2020Scaling laws Turning pointbuilt directly on TransformerChapter 5. Scale becomes a science
  18. 2020Longformerbuilt directly on TransformerChapter 10. Reading more than you remember
  19. 2020Dense retrievalbuilt directly on BERTChapter 10. Reading more than you remember
  20. 2020RAGbuilt directly on Dense retrievalChapter 10. Reading more than you remember
  21. 2020GPT-3 Turning pointwas influenced by Scaling laws
    built directly on GPT-2
    built directly on Sparse Transformer    Chapter 5. Scale becomes a science
  22. 2020BigBirdbuilt directly on TransformerChapter 10. Reading more than you remember
  23. 2020RLHF for summarisation Turning pointbuilt directly on RLHF for LMsChapter 6. Teaching models what people want
  24. 2020Vision Transformerbuilt directly on TransformerChapter 11. Seeing and hearing
  25. 2020The Pilewas influenced by GPT-3Chapter 5. Scale becomes a science
  26. 2021Switch Transformerbuilt directly on T5Chapter 5. Scale becomes a science
  27. 2021CLIPbuilt directly on GPT-2
    built directly on Vision Transformer    Chapter 11. Seeing and hearing
  28. 2021RoPE Turning pointbuilt directly on TransformerChapter 10. Reading more than you remember
  29. 2021LoRAbuilt directly on TransformerChapter 9. Making it run anywhere
  30. 2021Copilotwas influenced by Codex / HumanEvalChapter 7. The assistant goes public
  31. 2021Codex / HumanEvalbuilt directly on GPT-3Chapter 13. Models that act
  32. 2021ALiBibuilt directly on TransformerChapter 10. Reading more than you remember
  33. 2021FLANwas influenced by T5Chapter 6. Teaching models what people want
  34. 2021T0built directly on T5Chapter 6. Teaching models what people want
  35. 2021HHH assistantwas influenced by RLHF for summarisationChapter 6. Teaching models what people want
  36. 2021RETROwas influenced by Dense retrievalChapter 10. Reading more than you remember
  37. 2021WebGPTbuilt directly on GPT-3Chapter 13. Models that act
  38. 2022MT-NLGwas influenced by Scaling laws
    built directly on Megatron-LM    Chapter 5. Scale becomes a science
  39. 2022Chain of thought Turning pointwas influenced by GPT-3Chapter 12. Thinking longer
  40. 2022InstructGPT Turning pointbuilt directly on GPT-3
    was influenced by FLAN    Chapter 6. Teaching models what people want
  41. 2022Self-consistencybuilt directly on Chain of thoughtChapter 12. Thinking longer
  42. 2022STaRwas influenced by Chain of thoughtChapter 12. Thinking longer
  43. 2022Chinchilla Turning pointbuilt directly on Scaling lawsChapter 5. Scale becomes a science
  44. 2022PaLMwas influenced by Chain of thoughtChapter 5. Scale becomes a science
  45. 2022HH-RLHFbuilt directly on RLHF for summarisationChapter 6. Teaching models what people want
  46. 2022Flamingobuilt directly on Vision TransformerChapter 11. Seeing and hearing
  47. 2022OPT-175Bwas influenced by Megatron-LMChapter 8. The weights get out
  48. 2022Zero-shot CoTwas influenced by GPT-3Chapter 12. Thinking longer
  49. 2022FlashAttentionbuilt directly on TransformerChapter 9. Making it run anywhere
  50. 2022Emergent abilitiesbuilt directly on Scaling lawsChapter 5. Scale becomes a science
  51. 2022BLOOMbuilt directly on ALiBiChapter 8. The weights get out
  52. 2022LLM.int8()was influenced by OPT-175BChapter 9. Making it run anywhere
  53. 2022Whisperbuilt directly on TransformerChapter 11. Seeing and hearing
  54. 2022ReActbuilt directly on Chain of thought
    was influenced by WebGPT    Chapter 13. Models that act
  55. 2022Flan-T5 / Flan-PaLMbuilt directly on T5Chapter 6. Teaching models what people want
  56. 2022GPTQwas influenced by LLM.int8()Chapter 9. Making it run anywhere
  57. 2022Speculative decodingwas influenced by TransformerChapter 9. Making it run anywhere
  58. 2022GPT-3.5 Turning pointbuilt directly on GPT-3Chapter 7. The assistant goes public
  59. 2022ChatGPTbuilt directly on InstructGPT
    built directly on GPT-3.5    Chapter 7. The assistant goes public
  60. 2022Constitutional AIwas influenced by Chain of thoughtChapter 6. Teaching models what people want
  61. 2022Self-Instructbuilt directly on GPT-3
    was influenced by FLAN    Chapter 6. Teaching models what people want
  62. 2023Toolformerbuilt directly on GPT-3Chapter 13. Models that act
  63. 2023LLaMA Turning pointbuilt directly on RoPEChapter 8. The weights get out
  64. 2023llama.cppwas influenced by LLaMAChapter 9. Making it run anywhere
  65. 2023Alpacabuilt directly on LLaMAChapter 8. The weights get out
  66. 2023Claudebuilt directly on Constitutional AI
    was influenced by HH-RLHF    Chapter 7. The assistant goes public
  67. 2023GPT-4 Turning pointbuilt directly on Scaling lawsChapter 7. The assistant goes public
  68. 2023Reflexionbuilt directly on ReActChapter 13. Models that act
  69. 2023AutoGPTwas influenced by GPT-4Chapter 13. Models that act
  70. 2023Pythiawas influenced by Scaling lawsChapter 8. The weights get out
  71. 2023LLaVAbuilt directly on GPT-4
    built directly on LLaMA
    built directly on CLIP    Chapter 11. Seeing and hearing
  72. 2023Emergence miragesucceeded Emergent abilitiesChapter 5. Scale becomes a science
  73. 2023100K contextbuilt directly on ClaudeChapter 10. Reading more than you remember
  74. 2023Tree of thoughtsbuilt directly on Chain of thoughtChapter 12. Thinking longer
  75. 2023Grouped-query attention Turning pointbuilt directly on Multi-query attentionChapter 9. Making it run anywhere
  76. 2023QLoRAbuilt directly on LoRAChapter 9. Making it run anywhere
  77. 2023Falconbuilt directly on RefinedWebChapter 8. The weights get out
  78. 2023DPObuilt directly on RLHF for summarisation
    built directly on InstructGPT    Chapter 6. Teaching models what people want
  79. 2023Process supervisionbuilt directly on Chain of thoughtChapter 12. Thinking longer
  80. 2023RefinedWebwas influenced by C4Chapter 8. The weights get out
  81. 2023Function callingwas influenced by Toolformer (indirect influence)Chapter 13. Models that act
  82. 2023phi-1was influenced by GPT-3.5
    was influenced by Self-Instruct (indirect influence)    Chapter 8. The weights get out
  83. 2023Position interpolationbuilt directly on RoPEChapter 10. Reading more than you remember
  84. 2023Lost in the middlewas influenced by RAGChapter 10. Reading more than you remember
  85. 2023Llama 2 Turning pointbuilt directly on LLaMA
    built directly on Grouped-query attention    Chapter 8. The weights get out
  86. 2023vLLMwas influenced by Multi-query attentionChapter 9. Making it run anywhere
  87. 2023GPT-4Vbuilt directly on GPT-4
    was influenced by CLIP (indirect influence)    Chapter 11. Seeing and hearing
  88. 2023Mistral 7Bwas influenced by Sparse Transformer
    was influenced by Longformer
    was influenced by FlashAttention    Chapter 8. The weights get out
  89. 2023SWE-benchwas influenced by GPT-4Chapter 13. Models that act
  90. 2023Mambawas influenced by FlashAttentionChapter 9. Making it run anywhere
  91. 2023Gemini 1.0was influenced by Vision TransformerChapter 11. Seeing and hearing
  92. 2023Mixtral 8x7Bbuilt directly on Switch Transformer
    built directly on Mistral 7B    Chapter 8. The weights get out
  93. 2024DeepSeek LLMbuilt directly on Scaling lawsChapter 8. The weights get out
  94. 2024DeepSeekMoEwas influenced by Switch TransformerChapter 8. The weights get out
  95. 2024Dolmawas influenced by C4
    was influenced by The Pile    Chapter 8. The weights get out
  96. 2024OLMowas influenced by PythiaChapter 8. The weights get out
  97. 2024GRPObuilt directly on DeepSeek LLMChapter 12. Thinking longer
  98. 2024Gemini 1.5built directly on Gemini 1.0Chapter 10. Reading more than you remember
  99. 2024Gemmabuilt directly on Gemini 1.0
    was influenced by Mistral 7B    Chapter 8. The weights get out
  100. 2024Claude 3was influenced by GPT-4Chapter 14. Where the threads meet now
  101. 2024Chatbot Arenawas influenced by ChatGPTChapter 14. Where the threads meet now
  102. 2024Devinwas influenced by GPT-4Chapter 13. Models that act
  103. 2024Llama 3built directly on Grouped-query attention
    was influenced by Chinchilla    Chapter 8. The weights get out
  104. 2024Phi-3built directly on phi-1Chapter 8. The weights get out
  105. 2024DeepSeek-V2 / MLAbuilt directly on RoPE
    was influenced by Multi-query attention    Chapter 9. Making it run anywhere
  106. 2024SWE-agentbuilt directly on SWE-benchChapter 13. Models that act
  107. 2024GPT-4owas influenced by Whisper (indirect influence)Chapter 11. Seeing and hearing
  108. 2024Claude 3.5 Sonnetbuilt directly on Claude 3Chapter 14. Where the threads meet now
  109. 2024FineWebwas influenced by The PileChapter 8. The weights get out
  110. 2024FlashAttention-3built directly on FlashAttentionChapter 9. Making it run anywhere
  111. 2024Qwen2built directly on RoPEChapter 8. The weights get out
  112. 2024Llama 3.1 405Bwas influenced by Chinchilla
    built directly on Grouped-query attention
    was influenced by Position interpolation    Chapter 8. The weights get out
  113. 2024Test-time computewas influenced by Process supervision
    was influenced by Tree of thoughts (indirect influence)
    was influenced by STaR (indirect influence)    Chapter 12. Thinking longer
  114. 2024o1 Turning pointwas influenced by Chain of thoughtChapter 12. Thinking longer
  115. 2024Computer usewas influenced by GPT-4VChapter 13. Models that act
  116. 2024MCPwas influenced by Toolformer (indirect influence)Chapter 13. Models that act
  117. 2024Gemini 2.0built directly on Gemini 1.5Chapter 14. Where the threads meet now
  118. 2024o3 announcedbuilt directly on o1Chapter 12. Thinking longer
  119. 2024DeepSeek-V3built directly on DeepSeek-V2 / MLA
    was influenced by GPT-4o    Chapter 8. The weights get out
  120. 2025DeepSeek-R1was influenced by Chain of thought
    built directly on GRPO
    built directly on DeepSeek-V3    Chapter 12. Thinking longer
  121. 2025Deep researchwas influenced by WebGPT (indirect influence)Chapter 13. Models that act
  122. 2025Claude 3.7was influenced by o1
    built directly on Claude 3.5 Sonnet    Chapter 12. Thinking longer
  123. 2025Claude Codewas influenced by Computer use
    was influenced by SWE-agent (indirect influence)
    was influenced by MCP    Chapter 13. Models that act
  124. 2025GPT-4.5was influenced by Scaling lawsChapter 14. Where the threads meet now
  125. 2025Gemini 2.5was influenced by Gemini 1.5Chapter 12. Thinking longer
  126. 2025Llama 4was influenced by Switch TransformerChapter 8. The weights get out
  127. 2025o3 releasedbuilt directly on o1
    was influenced by Function calling    Chapter 12. Thinking longer
  128. 2025Qwen3built directly on Qwen2Chapter 8. The weights get out
  129. 2025Claude 4succeeded Claude 3
    was influenced by SWE-bench    Chapter 13. Models that act
  130. 2025Kimi K2was influenced by DeepSeekMoE
    was influenced by DeepSeek-V3 (indirect influence)
    was influenced by GRPO    Chapter 8. The weights get out
  131. 2025IMO goldbuilt directly on Gemini 2.5Chapter 12. Thinking longer
  132. 2025gpt-osswas influenced by DeepSeek-R1 (indirect influence)Chapter 8. The weights get out
  133. 2025GPT-5succeeded GPT-4oChapter 14. Where the threads meet now
  134. 2025GLM-4.5was influenced by DeepSeek-R1 (indirect influence)Chapter 8. The weights get out
  135. 2025DeepSeek-V3.2was influenced by Sparse TransformerChapter 10. Reading more than you remember
  136. 2025Claude Sonnet 4.5built directly on Claude 4Chapter 13. Models that act
  137. 2025IMO-ProofBenchwas influenced by IMO goldChapter 12. Thinking longer
  138. 2025GPT-5.1built directly on GPT-5Chapter 14. Where the threads meet now
  139. 2025Gemini 3was influenced by Gemini 1.5Chapter 14. Where the threads meet now
  140. 2025Claude Opus 4.5was influenced by SWE-benchChapter 13. Models that act
  141. 2025GPT-5.2built directly on GPT-5.1Chapter 14. Where the threads meet now
  142. 2026GPT-5.3-Codexwas influenced by Computer use (indirect influence)Chapter 13. Models that act
  143. 2026Claude Opus 4.6built directly on Claude Opus 4.5Chapter 14. Where the threads meet now
  144. 2026Gemini 3 Deep Thinkbuilt directly on Gemini 3Chapter 12. Thinking longer
  145. 2026Qwen3.5built directly on Qwen3Chapter 8. The weights get out
  146. 2026Gemini 3.1built directly on Gemini 3Chapter 14. Where the threads meet now
  147. 2026GPT-5.4built directly on GPT-5.3-CodexChapter 14. Where the threads meet now
  148. 2026Project Glasswingwas influenced by Computer use (indirect influence)Chapter 14. Where the threads meet now
  149. 2026Muse Sparksucceeded Llama 4Chapter 8. The weights get out
  150. 2026Claude Opus 4.7built directly on Claude Opus 4.6Chapter 14. Where the threads meet now
  151. 2026GPT-5.5built directly on GPT-5.4Chapter 14. Where the threads meet now
  152. 2026DeepSeek-V4built directly on DeepSeek-V3.2Chapter 8. The weights get out
  153. 2026Claude Opus 4.8built directly on Claude Opus 4.7Chapter 14. Where the threads meet now
  154. 2026Claude Sonnet 5succeeded Claude Opus 4.8Chapter 14. Where the threads meet now
  155. 2026Fable 5 / Mythos 5was influenced by Project GlasswingChapter 14. Where the threads meet now
  156. 2026GLM-5.2built directly on GLM-4.5Chapter 8. The weights get out
  157. 2026GPT-5.6 Solbuilt directly on GPT-5.5Chapter 14. Where the threads meet now
  158. 2026Kimi K3built directly on Kimi K2Chapter 8. The weights get out
  159. 2026Claude Opus 5succeeded Claude Opus 4.8Chapter 14. Where the threads meet now
  160. 2026Qwen3.8built directly on Qwen3.5Chapter 8. The weights get out
  161. 2026Claude Opus 5.5built directly on Claude Opus 5Chapter 14. Where the threads meet now

Direct links and their sources

Where it came from

None recorded in the sources used.

What it led to

Built on directly by

Influenced

Terms