NLP Lineage

21 July 2016 · Language modelling

Layer norm

Layer Normalization

J. Ba, J. Kiros, G. Hinton

Date
21 July 2016
What the date marks
arXiv submission
Kind
Technique
Organisation
University of Toronto
Access
Not applicable
Threads
In the guided tour
Chapter 3. Attention, then only attention
Learning phases
1. Transformer architecture

A milestone: 2 later milestones descend directly from it, and its descendants reach 10 of the 10 threads. How levels are decided

Read the primary source (arxiv.org)

Full summary in progress. Until then, this page shows the milestone's record from the research data.

What it changed

Before it

Batch normalisation stabilised training but depended on batch statistics.

The problem

Batch statistics are awkward for variable-length sequences and small batches.

What it introduced

Normalise across the features of each example independently of the batch.

Why it matters

The other half of the transformer block's plumbing. Pre-norm placement is what makes deep stacks trainable.

What modern models inherit

The normalisation layer inside every transformer block.

What came next

Used in the Transformer, then moved before each sublayer in GPT-2.

Before and after

The milestones it descends from, and the milestones that descend from it. Work developed alongside it is not counted. Weak links are left out unless you include them.

Before: where it came from

No documented ancestors are recorded for Layer norm without weak links.

No documented ancestors are recorded for Layer norm without weak links.

No recorded ancestors are recorded for Layer norm.

No recorded ancestors are recorded for Layer norm.

After: what it led to

Layer norm has 150 documented descendants across 12 chapters, reaching forward to Claude Opus 5.5 in 2026. Showing its direct descendants and the 18 turning points among them.

Layer norm has 150 documented descendants across 12 chapters, reaching forward to Claude Opus 5.5 in 2026. Showing all of them.

Layer norm has 159 recorded descendants, including indirect ones, across 12 chapters, reaching forward to Claude Opus 5.5 in 2026. Showing its direct descendants and the 19 turning points among them.

Layer norm has 159 recorded descendants, including indirect ones, across 12 chapters, reaching forward to Claude Opus 5.5 in 2026. Showing all of them.

  1. 2016Layer normThis milestone.
  2. 2017Transformer Turning pointbuilt directly on Layer normChapter 3. Attention, then only attention
  3. 2018GPT-1built directly on TransformerChapter 4. Pretrain once, use everywhere
  4. 2018BERT Turning pointbuilt directly on TransformerChapter 4. Pretrain once, use everywhere
  5. 2019Transformer-XLbuilt directly on TransformerChapter 4. Pretrain once, use everywhere
  6. 2019GPT-2 Turning pointbuilt directly on Layer normChapter 4. Pretrain once, use everywhere
  7. 2019WebTextwas influenced by GPT-1Chapter 4. Pretrain once, use everywhere
  8. 2019Sparse Transformer Turning pointbuilt directly on TransformerChapter 4. Pretrain once, use everywhere
  9. 2019RoBERTabuilt directly on BERTChapter 4. Pretrain once, use everywhere
  10. 2019Sentence-BERTbuilt directly on BERTChapter 10. Reading more than you remember
  11. 2019Megatron-LMbuilt directly on Transformer
    was influenced by GPT-2    Chapter 4. Pretrain once, use everywhere
  12. 2019RLHF for LMsbuilt directly on GPT-2Chapter 6. Teaching models what people want
  13. 2019T5 Turning pointbuilt directly on Transformer
    was influenced by GPT-2    Chapter 4. Pretrain once, use everywhere
  14. 2019Multi-query attentionbuilt directly on TransformerChapter 9. Making it run anywhere
  15. 2020Scaling laws Turning pointbuilt directly on Transformer
    was influenced by GPT-2    Chapter 5. Scale becomes a science
  16. 2020Longformerbuilt directly on TransformerChapter 10. Reading more than you remember
  17. 2020Dense retrievalbuilt directly on BERTChapter 10. Reading more than you remember
  18. 2020RAGbuilt directly on Dense retrievalChapter 10. Reading more than you remember
  19. 2020GPT-3 Turning pointbuilt directly on GPT-2Chapter 5. Scale becomes a science
  20. 2020BigBirdbuilt directly on TransformerChapter 10. Reading more than you remember
  21. 2020RLHF for summarisation Turning pointbuilt directly on RLHF for LMsChapter 6. Teaching models what people want
  22. 2020Vision Transformerbuilt directly on TransformerChapter 11. Seeing and hearing
  23. 2020The Pilewas influenced by GPT-3Chapter 5. Scale becomes a science
  24. 2021Switch Transformerbuilt directly on T5Chapter 5. Scale becomes a science
  25. 2021CLIPbuilt directly on GPT-2Chapter 11. Seeing and hearing
  26. 2021RoPE Turning pointbuilt directly on TransformerChapter 10. Reading more than you remember
  27. 2021LoRAbuilt directly on TransformerChapter 9. Making it run anywhere
  28. 2021Copilotwas influenced by Codex / HumanEvalChapter 7. The assistant goes public
  29. 2021Codex / HumanEvalbuilt directly on GPT-3Chapter 13. Models that act
  30. 2021ALiBibuilt directly on TransformerChapter 10. Reading more than you remember
  31. 2021FLANwas influenced by GPT-3
    was influenced by T5    Chapter 6. Teaching models what people want
  32. 2021T0built directly on T5Chapter 6. Teaching models what people want
  33. 2021HHH assistantwas influenced by RLHF for summarisationChapter 6. Teaching models what people want
  34. 2021RETROwas influenced by Dense retrievalChapter 10. Reading more than you remember
  35. 2021WebGPTbuilt directly on GPT-3Chapter 13. Models that act
  36. 2022MT-NLGwas influenced by Scaling laws
    built directly on Megatron-LM    Chapter 5. Scale becomes a science
  37. 2022Chain of thought Turning pointwas influenced by GPT-3Chapter 12. Thinking longer
  38. 2022InstructGPT Turning pointbuilt directly on GPT-3Chapter 6. Teaching models what people want
  39. 2022Self-consistencybuilt directly on Chain of thoughtChapter 12. Thinking longer
  40. 2022STaRwas influenced by Chain of thoughtChapter 12. Thinking longer
  41. 2022Chinchilla Turning pointbuilt directly on Scaling lawsChapter 5. Scale becomes a science
  42. 2022PaLMwas influenced by Chain of thoughtChapter 5. Scale becomes a science
  43. 2022HH-RLHFbuilt directly on RLHF for summarisationChapter 6. Teaching models what people want
  44. 2022Flamingowas influenced by GPT-3
    built directly on CLIP
    built directly on Vision Transformer    Chapter 11. Seeing and hearing
  45. 2022OPT-175Bbuilt directly on GPT-3
    was influenced by Megatron-LM    Chapter 8. The weights get out
  46. 2022Zero-shot CoTwas influenced by GPT-3Chapter 12. Thinking longer
  47. 2022FlashAttentionbuilt directly on TransformerChapter 9. Making it run anywhere
  48. 2022Emergent abilitiesbuilt directly on Scaling lawsChapter 5. Scale becomes a science
  49. 2022BLOOMbuilt directly on ALiBiChapter 8. The weights get out
  50. 2022LLM.int8()was influenced by OPT-175BChapter 9. Making it run anywhere
  51. 2022Whisperbuilt directly on TransformerChapter 11. Seeing and hearing
  52. 2022ReActbuilt directly on Chain of thought
    was influenced by WebGPT    Chapter 13. Models that act
  53. 2022Flan-T5 / Flan-PaLMbuilt directly on T5Chapter 6. Teaching models what people want
  54. 2022GPTQwas influenced by LLM.int8()Chapter 9. Making it run anywhere
  55. 2022Speculative decodingwas influenced by TransformerChapter 9. Making it run anywhere
  56. 2022GPT-3.5 Turning pointbuilt directly on GPT-3Chapter 7. The assistant goes public
  57. 2022ChatGPTbuilt directly on InstructGPT
    built directly on GPT-3.5    Chapter 7. The assistant goes public
  58. 2022Constitutional AIwas influenced by Chain of thoughtChapter 6. Teaching models what people want
  59. 2022Self-Instructbuilt directly on GPT-3Chapter 6. Teaching models what people want
  60. 2023Toolformerbuilt directly on GPT-3Chapter 13. Models that act
  61. 2023LLaMA Turning pointbuilt directly on RoPEChapter 8. The weights get out
  62. 2023llama.cppwas influenced by LLaMAChapter 9. Making it run anywhere
  63. 2023Alpacabuilt directly on LLaMA
    was influenced by GPT-3.5
    built directly on Self-Instruct    Chapter 8. The weights get out
  64. 2023Claudebuilt directly on Constitutional AI
    was influenced by HH-RLHF    Chapter 7. The assistant goes public
  65. 2023GPT-4 Turning pointbuilt directly on Scaling lawsChapter 7. The assistant goes public
  66. 2023Reflexionbuilt directly on ReActChapter 13. Models that act
  67. 2023AutoGPTwas influenced by GPT-4Chapter 13. Models that act
  68. 2023Pythiawas influenced by Scaling lawsChapter 8. The weights get out
  69. 2023LLaVAbuilt directly on CLIPChapter 11. Seeing and hearing
  70. 2023Emergence miragesucceeded Emergent abilitiesChapter 5. Scale becomes a science
  71. 2023100K contextbuilt directly on ClaudeChapter 10. Reading more than you remember
  72. 2023Tree of thoughtsbuilt directly on Chain of thoughtChapter 12. Thinking longer
  73. 2023Grouped-query attention Turning pointbuilt directly on Multi-query attentionChapter 9. Making it run anywhere
  74. 2023QLoRAbuilt directly on LoRAChapter 9. Making it run anywhere
  75. 2023DPObuilt directly on RLHF for summarisation
    built directly on InstructGPT    Chapter 6. Teaching models what people want
  76. 2023Process supervisionbuilt directly on Chain of thoughtChapter 12. Thinking longer
  77. 2023phi-1was influenced by GPT-3.5Chapter 8. The weights get out
  78. 2023Position interpolationbuilt directly on RoPEChapter 10. Reading more than you remember
  79. 2023Lost in the middlewas influenced by RAGChapter 10. Reading more than you remember
  80. 2023Llama 2 Turning pointbuilt directly on LLaMA
    built directly on Grouped-query attention
    was influenced by InstructGPT    Chapter 8. The weights get out
  81. 2023vLLMwas influenced by Multi-query attentionChapter 9. Making it run anywhere
  82. 2023GPT-4Vbuilt directly on GPT-4Chapter 11. Seeing and hearing
  83. 2023Mistral 7Bwas influenced by Sparse Transformer
    was influenced by Longformer
    was influenced by FlashAttention    Chapter 8. The weights get out
  84. 2023SWE-benchwas influenced by GPT-4
    built directly on Codex / HumanEval    Chapter 13. Models that act
  85. 2023Mambawas influenced by FlashAttentionChapter 9. Making it run anywhere
  86. 2023Gemini 1.0was influenced by Vision TransformerChapter 11. Seeing and hearing
  87. 2023Mixtral 8x7Bbuilt directly on Switch Transformer
    built directly on Mistral 7B    Chapter 8. The weights get out
  88. 2024DeepSeek LLMbuilt directly on Scaling lawsChapter 8. The weights get out
  89. 2024DeepSeekMoEwas influenced by Switch TransformerChapter 8. The weights get out
  90. 2024Dolmawas influenced by The PileChapter 8. The weights get out
  91. 2024OLMowas influenced by PythiaChapter 8. The weights get out
  92. 2024GRPOwas influenced by InstructGPT
    built directly on DeepSeek LLM    Chapter 12. Thinking longer
  93. 2024Gemini 1.5built directly on Gemini 1.0Chapter 10. Reading more than you remember
  94. 2024Gemmabuilt directly on Gemini 1.0
    was influenced by Mistral 7B    Chapter 8. The weights get out
  95. 2024Claude 3was influenced by GPT-4Chapter 14. Where the threads meet now
  96. 2024Chatbot Arenawas influenced by ChatGPTChapter 14. Where the threads meet now
  97. 2024Devinwas influenced by GPT-4Chapter 13. Models that act
  98. 2024Llama 3built directly on Grouped-query attention
    was influenced by Chinchilla    Chapter 8. The weights get out
  99. 2024Phi-3built directly on phi-1Chapter 8. The weights get out
  100. 2024DeepSeek-V2 / MLAbuilt directly on RoPE
    was influenced by Multi-query attention    Chapter 9. Making it run anywhere
  101. 2024SWE-agentbuilt directly on SWE-bench
    was influenced by ReAct    Chapter 13. Models that act
  102. 2024GPT-4obuilt directly on GPT-4Chapter 11. Seeing and hearing
  103. 2024Claude 3.5 Sonnetbuilt directly on Claude 3Chapter 14. Where the threads meet now
  104. 2024FineWebwas influenced by The PileChapter 8. The weights get out
  105. 2024FlashAttention-3built directly on FlashAttentionChapter 9. Making it run anywhere
  106. 2024Qwen2built directly on RoPEChapter 8. The weights get out
  107. 2024Llama 3.1 405Bwas influenced by Chinchilla
    built directly on Grouped-query attention
    was influenced by Position interpolation    Chapter 8. The weights get out
  108. 2024Test-time computewas influenced by Process supervision
    built directly on Self-consistency    Chapter 12. Thinking longer
  109. 2024o1 Turning pointwas influenced by Chain of thoughtChapter 12. Thinking longer
  110. 2024Computer usewas influenced by GPT-4VChapter 13. Models that act
  111. 2024Gemini 2.0built directly on Gemini 1.5Chapter 14. Where the threads meet now
  112. 2024o3 announcedbuilt directly on o1Chapter 12. Thinking longer
  113. 2024DeepSeek-V3built directly on DeepSeek-V2 / MLAChapter 8. The weights get out
  114. 2025DeepSeek-R1was influenced by Chain of thoughtChapter 12. Thinking longer
  115. 2025Deep researchwas influenced by o3 releasedChapter 13. Models that act
  116. 2025Claude 3.7was influenced by o1Chapter 12. Thinking longer
  117. 2025Claude Codewas influenced by Computer use
    was influenced by Claude 3.7    Chapter 13. Models that act
  118. 2025GPT-4.5was influenced by Scaling lawsChapter 14. Where the threads meet now
  119. 2025Gemini 2.5was influenced by o1
    was influenced by Gemini 1.5    Chapter 12. Thinking longer
  120. 2025Llama 4was influenced by Switch TransformerChapter 8. The weights get out
  121. 2025o3 releasedbuilt directly on o1Chapter 12. Thinking longer
  122. 2025Qwen3built directly on Qwen2Chapter 8. The weights get out
  123. 2025Claude 4succeeded Claude 3
    was influenced by SWE-bench    Chapter 13. Models that act
  124. 2025Kimi K2was influenced by DeepSeekMoE
    was influenced by GRPO    Chapter 8. The weights get out
  125. 2025IMO goldbuilt directly on Gemini 2.5
    was influenced by Test-time compute    Chapter 12. Thinking longer
  126. 2025gpt-ossbuilt directly on o3 releasedChapter 8. The weights get out
  127. 2025GPT-5succeeded GPT-4oChapter 14. Where the threads meet now
  128. 2025DeepSeek-V3.2was influenced by Sparse TransformerChapter 10. Reading more than you remember
  129. 2025Claude Sonnet 4.5built directly on Claude 4Chapter 13. Models that act
  130. 2025IMO-ProofBenchwas influenced by IMO goldChapter 12. Thinking longer
  131. 2025GPT-5.1built directly on GPT-5Chapter 14. Where the threads meet now
  132. 2025Gemini 3was influenced by Gemini 1.5Chapter 14. Where the threads meet now
  133. 2025Claude Opus 4.5was influenced by SWE-benchChapter 13. Models that act
  134. 2025GPT-5.2built directly on GPT-5.1Chapter 14. Where the threads meet now
  135. 2026Claude Opus 4.6built directly on Claude Opus 4.5Chapter 14. Where the threads meet now
  136. 2026GPT-5.3-Codexbuilt directly on GPT-5.2Chapter 13. Models that act
  137. 2026Gemini 3 Deep Thinkbuilt directly on Gemini 3Chapter 12. Thinking longer
  138. 2026Qwen3.5built directly on Qwen3Chapter 8. The weights get out
  139. 2026Gemini 3.1built directly on Gemini 3Chapter 14. Where the threads meet now
  140. 2026GPT-5.4built directly on GPT-5.3-CodexChapter 14. Where the threads meet now
  141. 2026Muse Sparksucceeded Llama 4Chapter 8. The weights get out
  142. 2026Claude Opus 4.7built directly on Claude Opus 4.6Chapter 14. Where the threads meet now
  143. 2026GPT-5.5built directly on GPT-5.4Chapter 14. Where the threads meet now
  144. 2026DeepSeek-V4built directly on DeepSeek-V3.2Chapter 8. The weights get out
  145. 2026Claude Opus 4.8built directly on Claude Opus 4.7Chapter 14. Where the threads meet now
  146. 2026Claude Sonnet 5succeeded Claude Opus 4.8Chapter 14. Where the threads meet now
  147. 2026GPT-5.6 Solbuilt directly on GPT-5.5Chapter 14. Where the threads meet now
  148. 2026Kimi K3built directly on Kimi K2Chapter 8. The weights get out
  149. 2026Claude Opus 5succeeded Claude Opus 4.8Chapter 14. Where the threads meet now
  150. 2026Qwen3.8built directly on Qwen3.5Chapter 8. The weights get out
  151. 2026Claude Opus 5.5built directly on Claude Opus 5Chapter 14. Where the threads meet now
  1. 2016Layer normThis milestone.
  2. 2017Transformer Turning pointbuilt directly on Layer normChapter 3. Attention, then only attention
  3. 2018GPT-1built directly on TransformerChapter 4. Pretrain once, use everywhere
  4. 2018BERT Turning pointbuilt directly on TransformerChapter 4. Pretrain once, use everywhere
  5. 2019Transformer-XLbuilt directly on TransformerChapter 4. Pretrain once, use everywhere
  6. 2019GPT-2 Turning pointbuilt directly on Layer normChapter 4. Pretrain once, use everywhere
  7. 2019WebTextwas influenced by GPT-1Chapter 4. Pretrain once, use everywhere
  8. 2019Sparse Transformer Turning pointbuilt directly on TransformerChapter 4. Pretrain once, use everywhere
  9. 2019RoBERTabuilt directly on BERTChapter 4. Pretrain once, use everywhere
  10. 2019Sentence-BERTbuilt directly on BERTChapter 10. Reading more than you remember
  11. 2019Megatron-LMbuilt directly on Transformer
    was influenced by GPT-2    Chapter 4. Pretrain once, use everywhere
  12. 2019RLHF for LMsbuilt directly on GPT-2Chapter 6. Teaching models what people want
  13. 2019T5 Turning pointbuilt directly on Transformer
    was influenced by GPT-2    Chapter 4. Pretrain once, use everywhere
  14. 2019C4 Turning pointwas influenced by WebText (indirect influence)Chapter 4. Pretrain once, use everywhere
  15. 2019Multi-query attentionbuilt directly on TransformerChapter 9. Making it run anywhere
  16. 2020Scaling laws Turning pointbuilt directly on Transformer
    was influenced by GPT-2    Chapter 5. Scale becomes a science
  17. 2020Longformerbuilt directly on TransformerChapter 10. Reading more than you remember
  18. 2020Dense retrievalbuilt directly on BERTChapter 10. Reading more than you remember
  19. 2020RAGbuilt directly on Dense retrievalChapter 10. Reading more than you remember
  20. 2020GPT-3 Turning pointbuilt directly on GPT-2Chapter 5. Scale becomes a science
  21. 2020BigBirdbuilt directly on TransformerChapter 10. Reading more than you remember
  22. 2020RLHF for summarisation Turning pointbuilt directly on RLHF for LMsChapter 6. Teaching models what people want
  23. 2020Vision Transformerbuilt directly on TransformerChapter 11. Seeing and hearing
  24. 2020The Pilewas influenced by GPT-3Chapter 5. Scale becomes a science
  25. 2021Switch Transformerbuilt directly on T5Chapter 5. Scale becomes a science
  26. 2021CLIPbuilt directly on GPT-2Chapter 11. Seeing and hearing
  27. 2021RoPE Turning pointbuilt directly on TransformerChapter 10. Reading more than you remember
  28. 2021LoRAbuilt directly on TransformerChapter 9. Making it run anywhere
  29. 2021Copilotwas influenced by Codex / HumanEvalChapter 7. The assistant goes public
  30. 2021Codex / HumanEvalbuilt directly on GPT-3Chapter 13. Models that act
  31. 2021ALiBibuilt directly on TransformerChapter 10. Reading more than you remember
  32. 2021FLANwas influenced by GPT-3
    was influenced by T5    Chapter 6. Teaching models what people want
  33. 2021T0built directly on T5Chapter 6. Teaching models what people want
  34. 2021HHH assistantwas influenced by RLHF for summarisationChapter 6. Teaching models what people want
  35. 2021RETROwas influenced by Dense retrievalChapter 10. Reading more than you remember
  36. 2021WebGPTbuilt directly on GPT-3Chapter 13. Models that act
  37. 2022MT-NLGwas influenced by Scaling laws
    built directly on Megatron-LM    Chapter 5. Scale becomes a science
  38. 2022Chain of thought Turning pointwas influenced by GPT-3Chapter 12. Thinking longer
  39. 2022InstructGPT Turning pointbuilt directly on GPT-3Chapter 6. Teaching models what people want
  40. 2022Self-consistencybuilt directly on Chain of thoughtChapter 12. Thinking longer
  41. 2022STaRwas influenced by Chain of thoughtChapter 12. Thinking longer
  42. 2022Chinchilla Turning pointbuilt directly on Scaling lawsChapter 5. Scale becomes a science
  43. 2022PaLMwas influenced by Chain of thoughtChapter 5. Scale becomes a science
  44. 2022HH-RLHFbuilt directly on RLHF for summarisationChapter 6. Teaching models what people want
  45. 2022Flamingowas influenced by GPT-3
    built directly on CLIP
    built directly on Vision Transformer    Chapter 11. Seeing and hearing
  46. 2022OPT-175Bbuilt directly on GPT-3
    was influenced by Megatron-LM    Chapter 8. The weights get out
  47. 2022Zero-shot CoTwas influenced by GPT-3Chapter 12. Thinking longer
  48. 2022FlashAttentionbuilt directly on TransformerChapter 9. Making it run anywhere
  49. 2022Emergent abilitiesbuilt directly on Scaling lawsChapter 5. Scale becomes a science
  50. 2022BLOOMbuilt directly on ALiBiChapter 8. The weights get out
  51. 2022LLM.int8()was influenced by OPT-175BChapter 9. Making it run anywhere
  52. 2022Whisperbuilt directly on TransformerChapter 11. Seeing and hearing
  53. 2022ReActbuilt directly on Chain of thought
    was influenced by WebGPT    Chapter 13. Models that act
  54. 2022Flan-T5 / Flan-PaLMbuilt directly on T5Chapter 6. Teaching models what people want
  55. 2022GPTQwas influenced by LLM.int8()Chapter 9. Making it run anywhere
  56. 2022Speculative decodingwas influenced by TransformerChapter 9. Making it run anywhere
  57. 2022GPT-3.5 Turning pointbuilt directly on GPT-3Chapter 7. The assistant goes public
  58. 2022ChatGPTbuilt directly on InstructGPT
    built directly on GPT-3.5    Chapter 7. The assistant goes public
  59. 2022Constitutional AIwas influenced by Chain of thoughtChapter 6. Teaching models what people want
  60. 2022Self-Instructbuilt directly on GPT-3Chapter 6. Teaching models what people want
  61. 2023Toolformerbuilt directly on GPT-3Chapter 13. Models that act
  62. 2023LLaMA Turning pointbuilt directly on RoPEChapter 8. The weights get out
  63. 2023llama.cppwas influenced by LLaMAChapter 9. Making it run anywhere
  64. 2023Alpacabuilt directly on LLaMA
    was influenced by GPT-3.5
    built directly on Self-Instruct    Chapter 8. The weights get out
  65. 2023Claudebuilt directly on Constitutional AI
    was influenced by HH-RLHF    Chapter 7. The assistant goes public
  66. 2023GPT-4 Turning pointbuilt directly on Scaling lawsChapter 7. The assistant goes public
  67. 2023Reflexionbuilt directly on ReActChapter 13. Models that act
  68. 2023AutoGPTwas influenced by GPT-4Chapter 13. Models that act
  69. 2023Pythiawas influenced by Scaling lawsChapter 8. The weights get out
  70. 2023LLaVAbuilt directly on CLIPChapter 11. Seeing and hearing
  71. 2023Emergence miragesucceeded Emergent abilitiesChapter 5. Scale becomes a science
  72. 2023100K contextbuilt directly on ClaudeChapter 10. Reading more than you remember
  73. 2023Tree of thoughtsbuilt directly on Chain of thoughtChapter 12. Thinking longer
  74. 2023Grouped-query attention Turning pointbuilt directly on Multi-query attentionChapter 9. Making it run anywhere
  75. 2023QLoRAbuilt directly on LoRAChapter 9. Making it run anywhere
  76. 2023Falconbuilt directly on RefinedWebChapter 8. The weights get out
  77. 2023DPObuilt directly on RLHF for summarisation
    built directly on InstructGPT    Chapter 6. Teaching models what people want
  78. 2023Process supervisionbuilt directly on Chain of thoughtChapter 12. Thinking longer
  79. 2023RefinedWebwas influenced by C4Chapter 8. The weights get out
  80. 2023Function callingwas influenced by Toolformer (indirect influence)Chapter 13. Models that act
  81. 2023phi-1was influenced by GPT-3.5
    was influenced by Self-Instruct (indirect influence)    Chapter 8. The weights get out
  82. 2023Position interpolationbuilt directly on RoPEChapter 10. Reading more than you remember
  83. 2023Lost in the middlewas influenced by RAGChapter 10. Reading more than you remember
  84. 2023Llama 2 Turning pointbuilt directly on LLaMA
    built directly on Grouped-query attention
    was influenced by InstructGPT    Chapter 8. The weights get out
  85. 2023vLLMwas influenced by Multi-query attentionChapter 9. Making it run anywhere
  86. 2023GPT-4Vwas influenced by CLIP (indirect influence)Chapter 11. Seeing and hearing
  87. 2023Mistral 7Bwas influenced by Sparse Transformer
    was influenced by Longformer
    was influenced by FlashAttention    Chapter 8. The weights get out
  88. 2023SWE-benchwas influenced by GPT-4
    built directly on Codex / HumanEval    Chapter 13. Models that act
  89. 2023Mambawas influenced by FlashAttentionChapter 9. Making it run anywhere
  90. 2023Gemini 1.0was influenced by Vision TransformerChapter 11. Seeing and hearing
  91. 2023Mixtral 8x7Bbuilt directly on Switch Transformer
    built directly on Mistral 7B    Chapter 8. The weights get out
  92. 2024DeepSeek LLMbuilt directly on Scaling lawsChapter 8. The weights get out
  93. 2024DeepSeekMoEwas influenced by Switch TransformerChapter 8. The weights get out
  94. 2024Dolmawas influenced by The PileChapter 8. The weights get out
  95. 2024OLMowas influenced by PythiaChapter 8. The weights get out
  96. 2024GRPOwas influenced by InstructGPT
    built directly on DeepSeek LLM    Chapter 12. Thinking longer
  97. 2024Gemini 1.5built directly on Gemini 1.0Chapter 10. Reading more than you remember
  98. 2024Gemmabuilt directly on Gemini 1.0
    was influenced by Mistral 7B    Chapter 8. The weights get out
  99. 2024Claude 3was influenced by GPT-4Chapter 14. Where the threads meet now
  100. 2024Chatbot Arenawas influenced by ChatGPTChapter 14. Where the threads meet now
  101. 2024Devinwas influenced by GPT-4Chapter 13. Models that act
  102. 2024Llama 3built directly on Grouped-query attention
    was influenced by Chinchilla    Chapter 8. The weights get out
  103. 2024Phi-3built directly on phi-1Chapter 8. The weights get out
  104. 2024DeepSeek-V2 / MLAbuilt directly on RoPE
    was influenced by Multi-query attention    Chapter 9. Making it run anywhere
  105. 2024SWE-agentbuilt directly on SWE-bench
    was influenced by ReAct    Chapter 13. Models that act
  106. 2024GPT-4owas influenced by Whisper (indirect influence)Chapter 11. Seeing and hearing
  107. 2024Claude 3.5 Sonnetbuilt directly on Claude 3Chapter 14. Where the threads meet now
  108. 2024FineWebwas influenced by The PileChapter 8. The weights get out
  109. 2024FlashAttention-3built directly on FlashAttentionChapter 9. Making it run anywhere
  110. 2024Qwen2built directly on RoPEChapter 8. The weights get out
  111. 2024Llama 3.1 405Bwas influenced by Chinchilla
    built directly on Grouped-query attention
    was influenced by Position interpolation    Chapter 8. The weights get out
  112. 2024Test-time computewas influenced by Process supervision
    was influenced by Tree of thoughts (indirect influence)
    was influenced by STaR (indirect influence)    Chapter 12. Thinking longer
  113. 2024o1 Turning pointwas influenced by Chain of thoughtChapter 12. Thinking longer
  114. 2024Computer usewas influenced by GPT-4VChapter 13. Models that act
  115. 2024MCPwas influenced by Toolformer (indirect influence)Chapter 13. Models that act
  116. 2024Gemini 2.0built directly on Gemini 1.5
    was influenced by Computer use (indirect influence)    Chapter 14. Where the threads meet now
  117. 2024o3 announcedbuilt directly on o1Chapter 12. Thinking longer
  118. 2024DeepSeek-V3built directly on DeepSeek-V2 / MLA
    was influenced by GPT-4o    Chapter 8. The weights get out
  119. 2025DeepSeek-R1was influenced by Chain of thoughtChapter 12. Thinking longer
  120. 2025Deep researchwas influenced by WebGPT (indirect influence)Chapter 13. Models that act
  121. 2025Claude 3.7was influenced by o1Chapter 12. Thinking longer
  122. 2025Claude Codewas influenced by Computer use
    was influenced by MCP    Chapter 13. Models that act
  123. 2025GPT-4.5was influenced by Scaling lawsChapter 14. Where the threads meet now
  124. 2025Gemini 2.5was influenced by o1
    was influenced by Gemini 1.5    Chapter 12. Thinking longer
  125. 2025Llama 4was influenced by Switch TransformerChapter 8. The weights get out
  126. 2025o3 releasedbuilt directly on o1
    was influenced by Function calling    Chapter 12. Thinking longer
  127. 2025Qwen3built directly on Qwen2Chapter 8. The weights get out
  128. 2025Claude 4succeeded Claude 3
    was influenced by SWE-bench    Chapter 13. Models that act
  129. 2025Kimi K2was influenced by DeepSeekMoE
    was influenced by DeepSeek-V3 (indirect influence)
    was influenced by GRPO    Chapter 8. The weights get out
  130. 2025IMO goldbuilt directly on Gemini 2.5
    was influenced by Test-time compute    Chapter 12. Thinking longer
  131. 2025gpt-osswas influenced by DeepSeek-R1 (indirect influence)Chapter 8. The weights get out
  132. 2025GPT-5succeeded GPT-4oChapter 14. Where the threads meet now
  133. 2025GLM-4.5was influenced by DeepSeek-R1 (indirect influence)Chapter 8. The weights get out
  134. 2025DeepSeek-V3.2was influenced by Sparse TransformerChapter 10. Reading more than you remember
  135. 2025Claude Sonnet 4.5built directly on Claude 4Chapter 13. Models that act
  136. 2025IMO-ProofBenchwas influenced by IMO goldChapter 12. Thinking longer
  137. 2025GPT-5.1built directly on GPT-5Chapter 14. Where the threads meet now
  138. 2025Gemini 3was influenced by Gemini 1.5Chapter 14. Where the threads meet now
  139. 2025Claude Opus 4.5was influenced by SWE-benchChapter 13. Models that act
  140. 2025GPT-5.2built directly on GPT-5.1Chapter 14. Where the threads meet now
  141. 2026GPT-5.3-Codexwas influenced by Computer use (indirect influence)Chapter 13. Models that act
  142. 2026Claude Opus 4.6built directly on Claude Opus 4.5Chapter 14. Where the threads meet now
  143. 2026Gemini 3 Deep Thinkbuilt directly on Gemini 3Chapter 12. Thinking longer
  144. 2026Qwen3.5built directly on Qwen3Chapter 8. The weights get out
  145. 2026Gemini 3.1built directly on Gemini 3Chapter 14. Where the threads meet now
  146. 2026GPT-5.4built directly on GPT-5.3-CodexChapter 14. Where the threads meet now
  147. 2026Project Glasswingwas influenced by Computer use (indirect influence)Chapter 14. Where the threads meet now
  148. 2026Muse Sparksucceeded Llama 4Chapter 8. The weights get out
  149. 2026Claude Opus 4.7built directly on Claude Opus 4.6Chapter 14. Where the threads meet now
  150. 2026GPT-5.5built directly on GPT-5.4Chapter 14. Where the threads meet now
  151. 2026DeepSeek-V4built directly on DeepSeek-V3.2Chapter 8. The weights get out
  152. 2026Claude Opus 4.8built directly on Claude Opus 4.7Chapter 14. Where the threads meet now
  153. 2026Claude Sonnet 5succeeded Claude Opus 4.8Chapter 14. Where the threads meet now
  154. 2026Fable 5 / Mythos 5was influenced by Project GlasswingChapter 14. Where the threads meet now
  155. 2026GLM-5.2built directly on GLM-4.5Chapter 8. The weights get out
  156. 2026GPT-5.6 Solbuilt directly on GPT-5.5Chapter 14. Where the threads meet now
  157. 2026Kimi K3built directly on Kimi K2Chapter 8. The weights get out
  158. 2026Claude Opus 5succeeded Claude Opus 4.8Chapter 14. Where the threads meet now
  159. 2026Qwen3.8built directly on Qwen3.5Chapter 8. The weights get out
  160. 2026Claude Opus 5.5built directly on Claude Opus 5Chapter 14. Where the threads meet now

Direct links and their sources

Where it came from

None recorded in the sources used.

What it led to

Built on directly by

Terms