NLP Lineage

9 January 2019 · Language modelling

Transformer-XL

Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context

Z. Dai, Z. Yang, Y. Yang, J. Carbonell, Q. Le, R. Salakhutdinov

Date
9 January 2019
What the date marks
arXiv submission
Kind
Architectural idea
Organisation
CMU / Google Brain
Access
Apache 2.0
Other dates
ACL 2019
Threads
In the guided tour
Chapter 4. Pretrain once, use everywhere
Learning phases
1. Transformer architecture

A milestone: 1 later milestone descends directly from it, and its descendants reach 9 of the 10 threads. How levels are decided

Read the primary source (arxiv.org)

Full summary in progress. Until then, this page shows the milestone's record from the research data.

What it changed

Before it

Transformers processed fixed segments with no memory between them.

The problem

Context was capped by the segment, and text was chopped arbitrarily at segment boundaries.

What it introduced

Reuse hidden states from the previous segment and encode position relatively rather than absolutely.

Why it matters

The first serious attack on the context-length limit, and the origin of relative position encoding.

What modern models inherit

Relative position encoding and the idea that context length is an engineering target.

What came next

Sparse attention, then RoPE and ALiBi, then the long-context race.

Before and after

The milestones it descends from, and the milestones that descend from it. Work developed alongside it is not counted. Weak links are left out unless you include them.

Before: where it came from

Transformer-XL has 10 documented ancestors across 3 chapters, reaching back to Recurrent networks in 1990. Showing its direct parents and the 4 turning points among them.

Transformer-XL has 10 documented ancestors across 3 chapters, reaching back to Recurrent networks in 1990. Showing all of them.

Transformer-XL has 11 recorded ancestors, including indirect ones, across 3 chapters, reaching back to Recurrent networks in 1990. Showing its direct parents and the 4 turning points among them.

Transformer-XL has 11 recorded ancestors, including indirect ones, across 3 chapters, reaching back to Recurrent networks in 1990. Showing all of them.

  1. 2019Transformer-XLThis milestone.
  2. 2017Transformer Turning pointTransformer-XL built directly on thisChapter 3. Attention, then only attention
  3. 2016Layer normTransformer built directly on thisChapter 3. Attention, then only attention
  4. 2015Residual connectionsTransformer was influenced by thisChapter 3. Attention, then only attention
  5. 2015Luong attentionTransformer was influenced by thisChapter 3. Attention, then only attention
  6. 2014AdamTransformer was influenced by thisChapter 2. Words become vectors
  7. 2014Seq2seq Turning pointTransformer built directly on thisChapter 2. Words become vectors
  8. 2014Attention Turning pointTransformer built directly on thisChapter 3. Attention, then only attention
  9. 2014GRU / encoder-decoderAttention built directly on thisChapter 2. Words become vectors
  10. 1997LSTM Turning pointSeq2seq built directly on thisChapter 1. Language as probability
  11. 1990Recurrent networksLSTM was influenced by thisChapter 1. Language as probability
  1. 2019Transformer-XLThis milestone.
  2. 2017Transformer Turning pointTransformer-XL built directly on thisChapter 3. Attention, then only attention
  3. 2017ConvS2STransformer was influenced by this (indirect influence)Chapter 3. Attention, then only attention
  4. 2016Layer normTransformer built directly on thisChapter 3. Attention, then only attention
  5. 2015Residual connectionsTransformer was influenced by thisChapter 3. Attention, then only attention
  6. 2015Luong attentionTransformer was influenced by thisChapter 3. Attention, then only attention
  7. 2014AdamTransformer was influenced by thisChapter 2. Words become vectors
  8. 2014Seq2seq Turning pointTransformer built directly on thisChapter 2. Words become vectors
  9. 2014Attention Turning pointTransformer built directly on thisChapter 3. Attention, then only attention
  10. 2014GRU / encoder-decoderAttention built directly on thisChapter 2. Words become vectors
  11. 1997LSTM Turning pointSeq2seq built directly on thisChapter 1. Language as probability
  12. 1990Recurrent networksLSTM was influenced by thisChapter 1. Language as probability

After: what it led to

Transformer-XL has 27 documented descendants across 5 chapters, reaching forward to Qwen3.8 in 2026. Showing its direct descendants and the 3 turning points among them.

Transformer-XL has 27 documented descendants across 5 chapters, reaching forward to Qwen3.8 in 2026. Showing all of them.

Transformer-XL has 124 recorded descendants, including indirect ones, across 11 chapters, reaching forward to Claude Opus 5.5 in 2026. Showing its direct descendants and the 10 turning points among them.

Transformer-XL has 124 recorded descendants, including indirect ones, across 11 chapters, reaching forward to Claude Opus 5.5 in 2026. Showing all of them.

  1. 2019Transformer-XLThis milestone.
  2. 2021RoPE Turning pointwas influenced by Transformer-XLChapter 10. Reading more than you remember
  3. 2023LLaMA Turning pointbuilt directly on RoPEChapter 8. The weights get out
  4. 2023llama.cppwas influenced by LLaMAChapter 9. Making it run anywhere
  5. 2023Alpacabuilt directly on LLaMAChapter 8. The weights get out
  6. 2023LLaVAbuilt directly on LLaMAChapter 11. Seeing and hearing
  7. 2023QLoRAbuilt directly on LLaMAChapter 9. Making it run anywhere
  8. 2023Position interpolationbuilt directly on RoPEChapter 10. Reading more than you remember
  9. 2023Llama 2 Turning pointbuilt directly on LLaMAChapter 8. The weights get out
  10. 2024DeepSeek LLMwas influenced by Llama 2Chapter 8. The weights get out
  11. 2024OLMowas influenced by Llama 2Chapter 8. The weights get out
  12. 2024GRPObuilt directly on DeepSeek LLMChapter 12. Thinking longer
  13. 2024Gemmawas influenced by Llama 2Chapter 8. The weights get out
  14. 2024Llama 3built directly on Llama 2Chapter 8. The weights get out
  15. 2024DeepSeek-V2 / MLAbuilt directly on RoPEChapter 9. Making it run anywhere
  16. 2024Qwen2built directly on RoPEChapter 8. The weights get out
  17. 2024Llama 3.1 405Bwas influenced by Position interpolationChapter 8. The weights get out
  18. 2024DeepSeek-V3built directly on DeepSeek-V2 / MLAChapter 8. The weights get out
  19. 2025DeepSeek-R1built directly on DeepSeek-V3Chapter 12. Thinking longer
  20. 2025Llama 4succeeded Llama 3.1 405BChapter 8. The weights get out
  21. 2025Qwen3built directly on Qwen2Chapter 8. The weights get out
  22. 2025Kimi K2was influenced by GRPOChapter 8. The weights get out
  23. 2025DeepSeek-V3.2was influenced by DeepSeek-V2 / MLAChapter 10. Reading more than you remember
  24. 2026Qwen3.5built directly on Qwen3Chapter 8. The weights get out
  25. 2026Muse Sparksucceeded Llama 4Chapter 8. The weights get out
  26. 2026DeepSeek-V4built directly on DeepSeek-V3.2
    was influenced by DeepSeek-V3    Chapter 8. The weights get out
  27. 2026Kimi K3built directly on Kimi K2Chapter 8. The weights get out
  28. 2026Qwen3.8built directly on Qwen3.5Chapter 8. The weights get out
  1. 2019Transformer-XLThis milestone.
  2. 2019Sparse Transformer Turning pointwas influenced by Transformer-XL (indirect influence)Chapter 4. Pretrain once, use everywhere
  3. 2020GPT-3 Turning pointbuilt directly on Sparse TransformerChapter 5. Scale becomes a science
  4. 2020BigBirdwas influenced by Sparse TransformerChapter 10. Reading more than you remember
  5. 2020The Pilewas influenced by GPT-3Chapter 5. Scale becomes a science
  6. 2021RoPE Turning pointwas influenced by Transformer-XLChapter 10. Reading more than you remember
  7. 2021LoRAbuilt directly on GPT-3Chapter 9. Making it run anywhere
  8. 2021Copilotwas influenced by Codex / HumanEvalChapter 7. The assistant goes public
  9. 2021Codex / HumanEvalbuilt directly on GPT-3Chapter 13. Models that act
  10. 2021FLANwas influenced by GPT-3Chapter 6. Teaching models what people want
  11. 2021WebGPTbuilt directly on GPT-3Chapter 13. Models that act
  12. 2022Chain of thought Turning pointwas influenced by GPT-3Chapter 12. Thinking longer
  13. 2022InstructGPT Turning pointbuilt directly on GPT-3Chapter 6. Teaching models what people want
  14. 2022Self-consistencybuilt directly on Chain of thoughtChapter 12. Thinking longer
  15. 2022STaRwas influenced by Chain of thoughtChapter 12. Thinking longer
  16. 2022PaLMwas influenced by Chain of thoughtChapter 5. Scale becomes a science
  17. 2022Flamingowas influenced by GPT-3Chapter 11. Seeing and hearing
  18. 2022OPT-175Bbuilt directly on GPT-3Chapter 8. The weights get out
  19. 2022Zero-shot CoTwas influenced by GPT-3Chapter 12. Thinking longer
  20. 2022FlashAttentionsucceeded BigBirdChapter 9. Making it run anywhere
  21. 2022Emergent abilitiesbuilt directly on Chain of thoughtChapter 5. Scale becomes a science
  22. 2022BLOOMwas influenced by The PileChapter 8. The weights get out
  23. 2022LLM.int8()was influenced by OPT-175BChapter 9. Making it run anywhere
  24. 2022ReActbuilt directly on Chain of thought
    was influenced by WebGPT    Chapter 13. Models that act
  25. 2022Flan-T5 / Flan-PaLMbuilt directly on Chain of thought
    built directly on FLAN    Chapter 6. Teaching models what people want
  26. 2022GPTQwas influenced by LLM.int8()Chapter 9. Making it run anywhere
  27. 2022GPT-3.5 Turning pointbuilt directly on GPT-3Chapter 7. The assistant goes public
  28. 2022ChatGPTbuilt directly on InstructGPT
    built directly on GPT-3.5    Chapter 7. The assistant goes public
  29. 2022Constitutional AIwas influenced by Chain of thoughtChapter 6. Teaching models what people want
  30. 2022Self-Instructbuilt directly on GPT-3Chapter 6. Teaching models what people want
  31. 2023Toolformerbuilt directly on GPT-3Chapter 13. Models that act
  32. 2023LLaMA Turning pointbuilt directly on RoPEChapter 8. The weights get out
  33. 2023llama.cppwas influenced by LLaMAChapter 9. Making it run anywhere
  34. 2023Alpacabuilt directly on LLaMAChapter 8. The weights get out
  35. 2023Claudebuilt directly on Constitutional AIChapter 7. The assistant goes public
  36. 2023GPT-4 Turning pointsucceeded GPT-3.5
    built directly on InstructGPT    Chapter 7. The assistant goes public
  37. 2023Reflexionbuilt directly on ReActChapter 13. Models that act
  38. 2023AutoGPTwas influenced by GPT-4
    was influenced by ReAct (indirect influence)    Chapter 13. Models that act
  39. 2023Pythiabuilt directly on The PileChapter 8. The weights get out
  40. 2023LLaVAbuilt directly on LLaMAChapter 11. Seeing and hearing
  41. 2023Emergence miragesucceeded Emergent abilitiesChapter 5. Scale becomes a science
  42. 2023100K contextbuilt directly on ClaudeChapter 10. Reading more than you remember
  43. 2023Tree of thoughtsbuilt directly on Chain of thoughtChapter 12. Thinking longer
  44. 2023QLoRAbuilt directly on LLaMAChapter 9. Making it run anywhere
  45. 2023DPObuilt directly on InstructGPTChapter 6. Teaching models what people want
  46. 2023Process supervisionbuilt directly on Chain of thoughtChapter 12. Thinking longer
  47. 2023Function callingwas influenced by Toolformer (indirect influence)Chapter 13. Models that act
  48. 2023phi-1was influenced by GPT-3.5
    was influenced by Self-Instruct (indirect influence)    Chapter 8. The weights get out
  49. 2023Position interpolationbuilt directly on RoPEChapter 10. Reading more than you remember
  50. 2023Lost in the middlewas influenced by 100K context (indirect influence)Chapter 10. Reading more than you remember
  51. 2023Llama 2 Turning pointbuilt directly on LLaMAChapter 8. The weights get out
  52. 2023GPT-4Vbuilt directly on GPT-4Chapter 11. Seeing and hearing
  53. 2023Mistral 7Bwas influenced by Sparse TransformerChapter 8. The weights get out
  54. 2023SWE-benchbuilt directly on Codex / HumanEvalChapter 13. Models that act
  55. 2023Mambawas influenced by FlashAttentionChapter 9. Making it run anywhere
  56. 2023Gemini 1.0succeeded PaLMChapter 11. Seeing and hearing
  57. 2023Mixtral 8x7Bbuilt directly on Mistral 7BChapter 8. The weights get out
  58. 2024DeepSeek LLMwas influenced by Llama 2Chapter 8. The weights get out
  59. 2024Dolmawas influenced by The PileChapter 8. The weights get out
  60. 2024OLMowas influenced by Llama 2Chapter 8. The weights get out
  61. 2024GRPOwas influenced by InstructGPTChapter 12. Thinking longer
  62. 2024Gemini 1.5built directly on Gemini 1.0Chapter 10. Reading more than you remember
  63. 2024Gemmawas influenced by Mistral 7BChapter 8. The weights get out
  64. 2024Claude 3was influenced by GPT-4
    built directly on Constitutional AI    Chapter 14. Where the threads meet now
  65. 2024Chatbot Arenawas influenced by ChatGPTChapter 14. Where the threads meet now
  66. 2024Devinwas influenced by GPT-4
    built directly on SWE-bench
    was influenced by ReAct (indirect influence)    Chapter 13. Models that act
  67. 2024Llama 3built directly on Llama 2Chapter 8. The weights get out
  68. 2024Phi-3built directly on phi-1Chapter 8. The weights get out
  69. 2024DeepSeek-V2 / MLAbuilt directly on RoPEChapter 9. Making it run anywhere
  70. 2024SWE-agentbuilt directly on SWE-bench
    was influenced by ReAct    Chapter 13. Models that act
  71. 2024GPT-4obuilt directly on GPT-4Chapter 11. Seeing and hearing
  72. 2024Claude 3.5 Sonnetbuilt directly on Claude 3Chapter 14. Where the threads meet now
  73. 2024FineWebwas influenced by The PileChapter 8. The weights get out
  74. 2024FlashAttention-3built directly on FlashAttentionChapter 9. Making it run anywhere
  75. 2024Qwen2built directly on RoPEChapter 8. The weights get out
  76. 2024Llama 3.1 405Bwas influenced by Position interpolationChapter 8. The weights get out
  77. 2024Test-time computewas influenced by Process supervision
    was influenced by Tree of thoughts (indirect influence)
    was influenced by STaR (indirect influence)    Chapter 12. Thinking longer
  78. 2024o1 Turning pointwas influenced by Chain of thoughtChapter 12. Thinking longer
  79. 2024Computer usewas influenced by Function calling (indirect influence)Chapter 13. Models that act
  80. 2024MCPwas influenced by Toolformer (indirect influence)Chapter 13. Models that act
  81. 2024Gemini 2.0was influenced by Computer use (indirect influence)Chapter 14. Where the threads meet now
  82. 2024o3 announcedbuilt directly on o1Chapter 12. Thinking longer
  83. 2024DeepSeek-V3built directly on DeepSeek-V2 / MLAChapter 8. The weights get out
  84. 2025DeepSeek-R1was influenced by Chain of thought
    built directly on DeepSeek-V3    Chapter 12. Thinking longer
  85. 2025Deep researchwas influenced by WebGPT (indirect influence)Chapter 13. Models that act
  86. 2025Claude 3.7was influenced by o1Chapter 12. Thinking longer
  87. 2025Claude Codewas influenced by MCPChapter 13. Models that act
  88. 2025GPT-4.5built directly on GPT-4oChapter 14. Where the threads meet now
  89. 2025Gemini 2.5was influenced by o1Chapter 12. Thinking longer
  90. 2025Llama 4succeeded Llama 3.1 405B
    was influenced by Mixtral 8x7B (indirect influence)    Chapter 8. The weights get out
  91. 2025o3 releasedbuilt directly on o1
    was influenced by Function calling    Chapter 12. Thinking longer
  92. 2025Qwen3built directly on Qwen2Chapter 8. The weights get out
  93. 2025Claude 4was influenced by SWE-benchChapter 13. Models that act
  94. 2025Kimi K2was influenced by DeepSeek-V3 (indirect influence)Chapter 8. The weights get out
  95. 2025IMO goldbuilt directly on Gemini 2.5
    was influenced by Test-time compute    Chapter 12. Thinking longer
  96. 2025gpt-osswas influenced by DeepSeek-R1 (indirect influence)Chapter 8. The weights get out
  97. 2025GPT-5built directly on o3 released
    succeeded GPT-4o    Chapter 14. Where the threads meet now
  98. 2025GLM-4.5was influenced by DeepSeek-R1 (indirect influence)Chapter 8. The weights get out
  99. 2025DeepSeek-V3.2was influenced by Sparse TransformerChapter 10. Reading more than you remember
  100. 2025Claude Sonnet 4.5built directly on Claude 4Chapter 13. Models that act
  101. 2025IMO-ProofBenchwas influenced by IMO goldChapter 12. Thinking longer
  102. 2025GPT-5.1built directly on GPT-5Chapter 14. Where the threads meet now
  103. 2025Gemini 3built directly on Gemini 2.5Chapter 14. Where the threads meet now
  104. 2025Claude Opus 4.5was influenced by SWE-benchChapter 13. Models that act
  105. 2025GPT-5.2built directly on GPT-5.1Chapter 14. Where the threads meet now
  106. 2026Claude Opus 4.6built directly on Claude Opus 4.5Chapter 14. Where the threads meet now
  107. 2026GPT-5.3-Codexwas influenced by Computer use (indirect influence)Chapter 13. Models that act
  108. 2026Gemini 3 Deep Thinkbuilt directly on Gemini 3
    was influenced by IMO gold    Chapter 12. Thinking longer
  109. 2026Qwen3.5built directly on Qwen3Chapter 8. The weights get out
  110. 2026Gemini 3.1built directly on Gemini 3Chapter 14. Where the threads meet now
  111. 2026GPT-5.4built directly on GPT-5.3-CodexChapter 14. Where the threads meet now
  112. 2026Project Glasswingwas influenced by Computer use (indirect influence)Chapter 14. Where the threads meet now
  113. 2026Muse Sparksucceeded Llama 4Chapter 8. The weights get out
  114. 2026Claude Opus 4.7built directly on Claude Opus 4.6Chapter 14. Where the threads meet now
  115. 2026GPT-5.5built directly on GPT-5.4Chapter 14. Where the threads meet now
  116. 2026DeepSeek-V4built directly on DeepSeek-V3.2Chapter 8. The weights get out
  117. 2026Claude Opus 4.8built directly on Claude Opus 4.7Chapter 14. Where the threads meet now
  118. 2026Claude Sonnet 5succeeded Claude Opus 4.8Chapter 14. Where the threads meet now
  119. 2026Fable 5 / Mythos 5was influenced by Project GlasswingChapter 14. Where the threads meet now
  120. 2026GLM-5.2built directly on GLM-4.5Chapter 8. The weights get out
  121. 2026GPT-5.6 Solbuilt directly on GPT-5.5Chapter 14. Where the threads meet now
  122. 2026Kimi K3built directly on Kimi K2Chapter 8. The weights get out
  123. 2026Claude Opus 5succeeded Claude Opus 4.8Chapter 14. Where the threads meet now
  124. 2026Qwen3.8built directly on Qwen3.5Chapter 8. The weights get out
  125. 2026Claude Opus 5.5built directly on Claude Opus 5Chapter 14. Where the threads meet now

Direct links and their sources

Where it came from

Built directly on

What it led to

Influenced